JP6280679B2 - Frame thinning device, frame interpolation device, video encoding device, video decoding device, and programs thereof - Google Patents

Description

  The present invention relates to a technique for transmitting a high frame rate video, and more particularly to a frame thinning device, a frame interpolation device, a video encoding device, a video for improving transmission efficiency and suppressing judder in high frame rate video transmission. The present invention relates to a decoding device and these programs.

  Video coding represented by HEVC / H.265 and AVC / H.264 requires various arithmetic processes such as motion prediction / motion compensation and entropy coding. In the field of super high-definition and digital cinema, there is a need for technology for encoding ultra-high-definition video with a frame frequency of 120 Hz at maximum, which is twice to four times the height and width of conventional high-definition broadcasting. However, this system is required to have a very high throughput and is difficult to realize in terms of cost and circuit scale. Hereinafter, an image that requires such a high frame rate is referred to as a “high frame rate image”.

  In video distribution for mobile terminals such as mobile terminals with limited bandwidth and processing capacity, it is known that frames are thinned out in advance by a predetermined amount and converted to a low frame rate before encoding and transmission (for example, , See Patent Document 1). However, it may cause judder where the movement is unnatural for an image with a lot of movement such as sports. In order to reduce this judder, a technique for converting to a high frame rate on the receiving side is known (for example, see Non-Patent Document 1). In this technique, the reception side suppresses the occurrence of judder by converting to a high frame rate by interpolating a new frame using only the received low frame rate video.

  On the other hand, as a high frame rate video encoding technique, a technique is known in which existing low frame rate codecs are operated in parallel and the codecs to be processed are allocated in units of frames (for example, Non-Patent Document 2). reference). With this technique, it is possible to easily encode an image having an integer multiple frame rate.

JP 2012-244666 A

B.‐D. Choi, J.‐W. Han, C.-S. Kim, and S.‐J.Ko, “Motion‐ Compensated Frame Interpolation Using Bilateral Motion Estimation and Adaptive Overlapped Block Matching Compensation,” IEEE Trans. on Circuits and Systems for Video Technology, vol.17, no.4, Apr. 2007. S. Sakaida, “Chapter VI: Super Hi-Vision and Its Encoding System,” in M. Mrak, M. Grgic, and M. Kunt, “High-Quality Visual Experience,” Springer, Jul. 2010

  As in the technique of Patent Document 1, judder occurs in a technique in which a high frame rate video is converted into a low frame rate video and then encoded and transmitted. In order to suppress the occurrence of this judder, the technique of Non-Patent Document 1 is configured such that the receiving side converts to a higher frame rate by interpolating a new frame using only the received low frame rate video. However, the interpolation may not be performed correctly and deterioration may be noticeable. For example, in a video with a lot of complicated motion and noise, an occlusion, and a video with a lot of uncovered areas, the interpolation cannot be performed correctly, and the deterioration becomes conspicuous.

  In addition, as in the technique of Non-Patent Document 2, as a high frame rate video encoding technique, when an existing low frame rate codec is operated in parallel and the codec to be processed is distributed in units of frames, In addition, a plurality of codecs are required, and correlation between these codecs is not considered. For this reason, there is a problem that the encoding efficiency is remarkably reduced as compared with the encoding process of a single high frame rate video.

  Therefore, there is a demand for a technique for improving transmission efficiency and suppressing judder without using an existing low frame rate codec in parallel for transmission of high frame rate video.

  An object of the present invention is made in view of the above-described problem, and is a frame thinning device and a frame interpolation device for improving transmission efficiency and suppressing judder with respect to transmission of a predetermined video such as a high frame rate video. It is to provide a video encoding device, a video decoding device, and a program thereof.

  In the present invention, on the transmission side, for example, a frame constituting an input video such as a high frame rate video is divided into a frame group (main video) that is transmitted directly to the reception side and a frame group (sub video) that is not transmitted. Typically, the main video is encoded and transmitted using an existing video encoding process. The sub video restoration target frame is interpolated by bidirectional motion prediction / motion compensation using the decoded main video frame that is temporally continuous to the frame, thereby eliminating the need for transmission of the original sub video. In addition, by performing the same motion prediction on the transmission side and the reception side, it is not necessary to transmit the motion vector itself.

  However, although bidirectional motion prediction using a decoded main video frame has the above-described advantages, it does not refer to a frame to be interpolated, and may cause an erroneous motion vector to be calculated.

  In order to suppress this, in a preferred embodiment according to the present invention, an error map corresponding to each motion within the search range of the bidirectional motion prediction is generated on the transmitting side and the receiving side, Select motion vector candidates. Further, regarding the selected plurality of motion vector candidates, the transmission side performs motion compensation for each of these motion vector candidates to obtain an interpolation result. By comparing the obtained interpolation result with the sub-picture of the original picture, it is determined which motion vector candidate interpolates the sub-video most appropriately, and only the index of the motion vector candidate is used as side information on the receiving side. To transmit.

  On the receiving side, bi-directional motion prediction is performed using the decoded main video, and a plurality of motion vector candidates are calculated in the same manner as on the transmitting side, and the motion vector actually used for interpolation is the index received as side information. Determine by reference. A restored video is obtained by synthesizing the interpolated video based on the determined motion vector and the decoded main video in the order of thinning out on the transmission side. Note that the present invention can also be applied to the case where the main video is transmitted without encoding.

  Accordingly, it is possible to improve transmission efficiency and suppress judder with respect to transmission of high frame rate video.

That is, the frame decimation device of the present invention is a frame decimation device that decimates and transmits a frame group with respect to a predetermined image, and includes a main image composed of a frame group that transmits a predetermined image to a receiving side and a frame group that does not transmit A frame dividing means for dividing the sub video into a plurality of frames of a main video temporally continuous with respect to the sub video, and a motion vector of a prediction target block for restoring the sub video by motion prediction A motion vector predicting unit that performs a plurality of predictions and determines a plurality of motion vector candidates and motion compensation using the plurality of motion vector candidates predicted with the main video generate respective interpolated videos corresponding to the plurality of motion vector candidates. The frame interpolation means, the respective interpolated video and the sub video are compared, and the plurality of motion vectors are compared. Comparing means for determining a motion vector corresponding to an interpolated video having the smallest difference among complements, and output means for outputting the determined motion vector index as side information to the outside together with the main video. And

  Further, in the frame thinning-out apparatus according to the present invention, the motion vector prediction means performs a block prediction with reference to the input block by performing a motion prediction between the input blocks referred to in the continuous main video frame. A plurality of motion vector candidates including a motion vector candidate corresponding to the minimum point as an error and a motion vector candidate corresponding to a minimum point that is minimum with respect to peripheral errors other than the minimum point, for the prediction target block in It is characterized by determining.

  In the frame thinning-out apparatus according to the present invention, the motion vector predicting unit generates an error map related to the prediction target block by performing motion prediction between input blocks respectively referred to in the continuous main video frames. A motion prediction means; a minimum point search means for searching and extracting a minimum point of the entire error map; and determining the extracted minimum point coordinates as a first motion vector candidate; and a threshold value with a predetermined value for the error map A threshold setting / processing means for generating an error map related to the prediction target block after the new threshold processing by performing processing, and the minimum point coordinates of each region formed in the error map divided by the threshold processing. The minimum point is searched and extracted for the region excluding the included region, and the extracted minimum point coordinates are set as the second and subsequent motion vector candidates. The second and subsequent motion vector candidates obtained by operating the minimum point search means for each area to be determined, and the operation of the threshold setting / processing means and the minimum point search means for each area at a specified number of times of one or more; Motion vector storage means for storing a plurality of motion vector candidates including the first motion vector candidate.

Furthermore, the frame interpolating apparatus of the present invention is a frame interpolating apparatus for interpolating a frame from a thinned video obtained by thinning a frame group with respect to a predetermined video and restoring the predetermined video, and intermittently regarding the predetermined video. Input means for inputting side information indicating an index of a motion vector of a prediction target block used to restore a main video from which frames have been thinned out and a sub video corresponding to the thinned frame group; Motion that refers to a plurality of frames of a main video that are temporally continuous with respect to a video, predicts a plurality of motion vectors of a prediction target block for restoring the sub video by motion prediction, and determines a plurality of motion vector candidates A vector prediction means, referring to the side information, and a prediction pair having the plurality of motion vector candidates according to the index Motion vector selection means for selecting one motion vector for the block; frame interpolation means for generating an interpolated video corresponding to the sub video using the main video and the selected motion vector; and Frame synthesis means for performing frame synthesis by interpolating with frames of the interpolated video and restoring the predetermined video.

  Further, in the frame interpolating apparatus according to the present invention, the motion vector predicting means performs block prediction with reference to the input block by performing motion prediction between the input blocks referred to in the continuous main video frames. A plurality of motion vector candidates including a motion vector candidate corresponding to the minimum point as an error and a motion vector candidate corresponding to a minimum point that is minimum with respect to peripheral errors other than the minimum point, for the prediction target block in It is characterized by determining.

  In the frame interpolation apparatus of the present invention, the motion vector predicting unit generates an error map related to the prediction target block by performing motion prediction between input blocks referred to in the continuous main video frames. A motion prediction means; a minimum point search means for searching and extracting a minimum point of the entire error map; and determining the extracted minimum point coordinates as a first motion vector candidate; and a threshold value with a predetermined value for the error map A threshold setting / processing means for generating an error map related to the prediction target block after the new threshold processing by performing processing, and the minimum point coordinates of each region formed in the error map divided by the threshold processing. Search and extract the minimum point for the area excluding the included area, and use the extracted minimum point coordinates as the second and subsequent motion vector candidates Region-specific minimum point search means, and the second and subsequent motion vector candidates obtained by operating the threshold value setting / processing means and the region-specific minimum point search means one or more times. Motion vector storage means for storing a plurality of motion vector candidates including the first motion vector candidate.

Furthermore, the video encoding device of the present invention is a video encoding device that encodes and transmits a predetermined video by thinning out a frame group, and transmits the predetermined video and a main video including a frame group that transmits the predetermined video to the receiving side. A frame dividing means for dividing the main video into sub-pictures, a video encoding means for encoding the main video and outputting to the outside, and a local decoding for generating a locally decoded main video from the encoded main video Means, referring to a plurality of frames of the locally decoded main video that are temporally continuous with respect to the sub video, and predicting a plurality of motion vectors of a prediction target block for restoring the sub video by motion prediction , Using motion vector predicting means for determining a plurality of motion vector candidates, the locally decoded main video, and a plurality of predicted motion vector candidates. Frame interpolation means for generating respective interpolated videos corresponding to a plurality of motion vector candidates by motion compensation, comparing each interpolated video with the sub-video, and determining the most difference among the plurality of motion vector candidates. Comparing means for determining a motion vector corresponding to a small number of interpolated videos, and output means for outputting the determined motion vector index as side information to the outside.

Furthermore, the video decoding apparatus of the present invention is a video decoding apparatus that restores a predetermined video by interpolating a frame from the thinned video obtained by thinning and encoding a frame group with respect to the predetermined video. A video decoding means for inputting a thinned video in which frames are intermittently thinned and encoded as a main video, decoding the main video, and generating a decoded main video, and the thinned frame input means for inputting a side information indicating an index of the motion vector of the prediction target block used to restore the sub-images corresponding to the group, a plurality reference frame of the main image to be temporally continuous to the sub video , Predicting a plurality of motion vectors of a prediction target block for restoring the sub video by motion prediction, and determining a plurality of motion vector candidates Vector prediction means, motion vector selection means for referring to the side information and selecting one motion vector for the prediction target block having the plurality of motion vector candidates according to the index, the decoded main video, and Frame interpolation means for generating an interpolated video corresponding to the sub video using the selected motion vector, and frame synthesis by interpolating the decoded main video with the frame of the interpolated video, thereby restoring the predetermined video Frame synthesizing means.

Furthermore, a program according to an aspect of the present invention includes a step of dividing a predetermined video into a main video composed of a frame group transmitted to a receiving side and a sub video composed of a frame group not transmitted to the computer, Determining a plurality of motion vector candidates by referring to a plurality of temporally continuous frames of a main video, predicting a plurality of motion vectors of a prediction target block for restoring the sub video by motion prediction, A step of generating each interpolated video corresponding to a plurality of motion vector candidates by motion compensation using the video and a plurality of motion vector candidates predicted, and comparing each interpolated video with the sub video, Determining a motion vector corresponding to an interpolated video with the smallest difference among the motion vector candidates of The index of the constant motion vector as side information, a program for executing the steps of: outputting to the outside together with the main image.

In another aspect of the present invention, a program for prediction is used to restore a main video in which frames are intermittently thinned with respect to a predetermined video and a sub video corresponding to the thinned frame group. A step of inputting side information indicating an index of a motion vector of the target block, and a prediction for restoring the sub video by motion prediction with reference to a plurality of temporally continuous main video frames for the sub video Predicting a plurality of motion vectors of the target block, determining a plurality of motion vector candidates, referring to the side information, and one motion vector for the prediction target block having the plurality of motion vector candidates according to the index Selecting the sub-image using the main video and the selected motion vector. And generating an interpolation image corresponding to an image, the main image performs frame synthesis by interpolating a frame of the interpolation image, a program for executing the steps of: restoring the predetermined image.

According to still another aspect of the present invention, there is provided a program for dividing a predetermined video into a main video composed of a frame group transmitted to a receiving side and a sub video composed of a frame group not transmitted to a computer; A step of encoding and outputting to the outside, a step of generating a locally decoded main image from the encoded main image, and a plurality of frames of the locally decoded main image that are temporally continuous with respect to the sub-image A plurality of motion vectors of a prediction target block for restoring the sub-video by motion prediction , determining a plurality of motion vector candidates, and a plurality of motion vectors predicted with the locally decoded main video Generate interpolated videos corresponding to multiple motion vector candidates by motion compensation using candidates. Comparing each interpolated video with the sub video, determining a motion vector corresponding to the interpolated video with the smallest difference among the plurality of motion vector candidates, and an index of the determined motion vector And a step of outputting to the outside as side information.

Further, a program according to another aspect of the present invention inputs a thinned video obtained by intermittently decimating and encoding frames with respect to a predetermined video as a main video to a computer, and performs decoding processing on the main video. A step of generating a decoded main video, a step of inputting side information indicating an index of a motion vector of a prediction target block used for restoring a sub video corresponding to the thinned frame group, and the sub video A plurality of frames of a main video that are temporally continuous, a plurality of motion vectors of a prediction target block for restoring the sub video by motion prediction, and a plurality of motion vector candidates are determined; , Referring to the side information, and a prediction target block having the plurality of motion vector candidates according to the index. Selecting one motion vector for the video, generating an interpolated video corresponding to the sub video using the decoded main video and the selected motion vector, and decoding the main video A program for performing frame synthesis by interpolating with a frame of the interpolated video and restoring the predetermined video.

  According to the present invention, when transmitting a predetermined video such as a high frame rate video by thinning out the frame, the information amount of the bit stream to be transmitted is reduced and the transmission efficiency is improved. Judder can be reduced by motion compensation. In particular, it is not necessary to transmit the motion vector itself by comparing the interpolation results of a plurality of motion vector candidates with the original image in advance on the transmission side and transmitting only the optimal motion vector index together with the main video, The amount of side information can be reduced. Further, according to the present invention, when a predetermined video such as a high frame rate video is thinned and encoded and transmitted, the high frame rate video is encoded by the existing low frame rate codec alone. Thus, the coding efficiency can be improved as compared with the case of using the low frame rate codec in parallel.

It is a block diagram which shows the outline of the flame | frame thinning-out apparatus of one Embodiment by this invention. It is a block diagram which shows the outline of the frame interpolation apparatus of one Embodiment by this invention. It is a block diagram which shows the outline of the bidirectional | two-way motion vector prediction part of the suitable example in the frame thinning-out apparatus of one Embodiment by this invention, and a frame interpolation apparatus. It is a flowchart which shows the suitable example of the motion prediction process in the bidirectional | two-way motion vector estimation part which concerns on this invention. It is a figure which shows an example of the bidirectional | two-way motion prediction by the bidirectional | two-way motion estimation part which concerns on this invention. (A), (B) is a figure explaining the threshold value process of the error map by the bidirectional | two-way motion estimation part which concerns on this invention. (A)-(D) is a figure explaining the extraction method of the minimum point coordinate of each area | region by the threshold value process of the error map by the bidirectional | two-way motion estimation part which concerns on this invention. It is a block diagram which shows the outline of the video coding apparatus of one Embodiment by this invention. It is a block diagram which shows the outline of the video decoding apparatus of one Embodiment by this invention. It is a figure which shows another example of the bidirectional | two-way motion prediction by the bidirectional | two-way motion estimation part which concerns on this invention.

  First, a frame thinning device and a frame interpolation device according to the first embodiment of the present invention will be described.

[First Embodiment]
(Frame thinning device)
FIG. 1 is a block diagram showing an outline of a frame thinning device 1 according to the first embodiment of the present invention. The frame decimation device 1 is a device that decimates and transmits a frame group for a predetermined video, and includes a frame division unit 11, a bidirectional motion vector prediction unit 12, a frame interpolation unit 13, a comparison unit 14, and a side information encoding unit 15. Prepare.

  The frame dividing unit 11 divides an input video such as a high frame rate video into a frame group (main video) transmitted to the receiving side and a frame group (sub video) not transmitted, and the main video is output to the outside as a main stream. The sub video is output to the comparison unit 14. Thereby, the amount of information of the main stream to be transmitted can be reduced. Note that the main video output in the main stream can be temporarily stored in a predetermined memory (not shown), and is temporarily stored in a manner that can be read by the bidirectional motion vector prediction unit 12 and the frame interpolation unit 13. The

  The bidirectional motion vector predicting unit 12 refers to a main video frame that is temporally continuous with respect to a sub video frame to be restored (preferably, a main video frame that is temporally positioned before and after). A plurality of motion vectors of a prediction target block for restoring the sub-video are predicted by motion prediction, and a plurality of motion vector candidates are output to the frame interpolation unit 13.

  More specifically, the bidirectional motion vector prediction unit 12 performs bidirectional motion prediction between input blocks (preferably between two input blocks having the same coordinates) that are referred to in the continuous main video frames. As a result, the motion vector candidate corresponding to the minimum point as the error and the minimum point that is the minimum with respect to the peripheral error other than the minimum point for the prediction target block in the block to be interpolated with reference to these input blocks A plurality of motion vector candidates including the motion vector candidates corresponding to the above are determined. Here, the two input blocks to be referred to in the successive main video frames are set in the same manner as described above at all coordinates for restoring the thinned sub video frames, and these input blocks are referred to. Determining a plurality of motion vector candidates for each prediction target block sufficient to restore the thinned sub-video frame by setting the prediction target block in all regions in the block to be interpolated Can do. For further details on the principle of bidirectional motion prediction using the input blocks b1 and b2, see, for example, Non-Patent Document 1. Therefore, the bidirectional motion vector prediction unit 12 performs motion vector prediction with reference to only the main video without using the sub-video of the original image. Details of a preferred example of the operation of the bidirectional motion vector prediction unit 12 will be described later.

  The frame interpolation unit 13 generates each interpolated video corresponding to the plurality of motion vector candidates by motion compensation using the main video and the predicted plurality of motion vector candidates, and outputs the generated interpolated video to the comparison unit 14.

  The comparison unit 14 compares each of the interpolated videos with the sub-picture of the original picture, and selects an interpolated picture having the smallest difference from the original picture, thereby corresponding to the selected interpolated picture among the plurality of motion vector candidates. A motion vector is determined, and an index of the determined motion vector is output to the side information encoding unit 15 as side information. More specifically, the comparison unit 14 calculates the smallest error between the prediction target block in the interpolated video and the block corresponding to the coordinate position of the prediction target block in the sub-image of the original image divided by the frame division unit 11. The optimal motion vector is set, and the index of the motion vector is output to the side information encoding unit 15 as side information.

  The side information encoding unit 15 encodes side information indicating an optimal motion vector index of each prediction target block using existing lossless encoding, and outputs the encoded bit stream (side stream) to the outside. When the side information is not encoded, the function of the side information encoding unit 15 is unnecessary, and the side information can be directly output as a side stream directly from the comparison unit 14.

  As described above, the frame thinning device 1 transmits a frame group with respect to a predetermined video together with predetermined side information.

(Frame interpolation device)
FIG. 2 is a block diagram showing an outline of the frame interpolation device 5 according to the first embodiment of the present invention. The frame interpolation device 5 is a device that interpolates a frame from a thinned video obtained by thinning a frame group with respect to a predetermined video and restores the predetermined video, and includes a bidirectional motion vector prediction unit 51, a side information decoding unit 52, A motion vector selection unit 53, a frame interpolation unit 54, and a frame synthesis unit 55 are provided.

  The bi-directional motion vector predicting unit 51 is a thinned video (ie, a main video output from the frame thinning device 1) in which frames are intermittently thinned with respect to a predetermined video such as a high frame rate image transmitted as a main stream. ) In order to restore the sub video with reference to the main video frames that are temporally continuous with respect to the sub video frames to be restored (preferably, the main video frames that are temporally positioned before and after). A plurality of motion vectors of the prediction target block are predicted, and a plurality of motion vector candidates are output to the motion vector selection unit 53.

  More specifically, the bi-directional motion vector prediction unit 51, as with the bi-directional motion vector prediction unit 12 on the transmission side, performs bi-directional motion prediction between two input blocks that are respectively referred to in successive main video frames. For the prediction target block in the block to be interpolated with reference to these input blocks, the motion vector candidate corresponding to the minimum point as an error and the minimum that is minimum with respect to peripheral errors other than the minimum point A plurality of motion vector candidates including a motion vector candidate corresponding to a point is determined. A suitable example of the operation of the bidirectional motion vector predicting unit 51 is the same as the example of the bidirectional motion vector predicting unit 12 in the frame thinning-out apparatus 1, and details thereof will be described later.

  The side information decoding unit 52 transmits side information (encoded side information output from the frame decimation device 1) related to the prediction target block for restoring the sub-video, transmitted as a side stream, to the transmission side code. The side information indicating the optimal motion vector index of the prediction target block is acquired by decoding using a decoding process corresponding to the conversion process, and is output to the motion vector selection unit 53. When the side information is not encoded, the function of the side information decoding unit 52 is unnecessary, and the motion vector selection unit 53 can be configured to acquire the side information directly from the side stream.

  The motion vector selection unit 53 refers to side information indicating the index of the optimal motion vector of each prediction target block, and refers to this index from among a plurality of motion vector candidates generated by the bidirectional motion vector prediction unit 51. One motion vector is selected and output to the frame interpolation unit 54.

  The frame interpolation unit 54 restores the interpolated video frame corresponding to the sub video using the received main video and the selected motion vector, and outputs the restored sub video to the frame synthesis unit 55.

  The frame synthesizing unit 55 performs frame synthesis on the received main video and the restored sub-video in the order thinned out on the transmission side, and restores the restored video obtained by restoring the predetermined video such as the high frame rate image to the outside Output to.

(Configuration of preferred example of bidirectional motion vector prediction unit)
FIG. 3 is a block diagram showing an outline of the bidirectional motion vector prediction units 12 and 51 of the preferred example in the frame thinning device 1 and the frame interpolation device 5 of the first embodiment according to the present invention. The bidirectional motion vector prediction units 12 and 51 perform processing common to the transmission side (the frame thinning device 1 side) and the reception side (the frame interpolation device 5 side). Since the main video that is input to the bidirectional motion vector prediction unit 51 on the reception side is common to the transmission side, a plurality of motion vector candidates to be predicted match on the transmission side and the reception side.

  The bi-directional motion vector predicting units 12 and 51 are main video frame groups that are temporally continuous to the frames of the prediction target block for restoring the sub video (preferably, main video frames positioned before and after the temporal video). Group) is a functional unit that performs bidirectional motion prediction between two input blocks b1 and b2 that are referred to in each group, and determines a plurality of motion vector candidates, and includes a bidirectional motion prediction unit 31, a minimum point search unit 32, a threshold value A setting / processing unit 33, a labeling unit 34, a region-specific minimum point searching unit 35, and a motion vector storage unit 36 are provided.

  The bidirectional motion prediction unit 31 performs bidirectional motion prediction between two input blocks (reference blocks) b1 and b2 that are respectively referred to in the continuous main video frames, generates an error map, and generates a minimum point search unit. 32 and the threshold value setting / processing unit 33. Here, the error map, which will be described in detail later with reference to FIG. 5, is an input block b1 set on a prediction target block for restoring a sub video and a main video frame positioned before and after in time. , B2 represents an error distribution between the search blocks in the input blocks b1 and b2, which appears when the search block is moved point-symmetrically with respect to the block to be interpolated (prediction target block).

  The minimum point search unit 32 searches for and extracts the minimum point of the entire error map, outputs this minimum point coordinate to the motion vector accumulation unit 36 as a corresponding motion vector candidate, and also sets information on the minimum value of the error map as a threshold value. The initial value of the threshold value used in the setting / processing unit 33 is output. This minimum point coordinate indicates the position coordinate of the search block in the input blocks b1 and b2, and indicates the coordinate point of the prediction target block with the smallest error between the search blocks in the input blocks b1 and b2.

  The threshold setting / processing unit 33 stores threshold information used to extract the minimum point or other local minimum points in the error map, and sets (updates) the threshold at a threshold level shifted from the threshold by a predetermined value. Then, a threshold value process is performed on the error map with the set (updated) threshold value to generate a new error map after the threshold process, and outputs the error map to the labeling unit 34 together with the stored minimum point coordinate information. In the error map after the threshold processing, a plurality of regions divided by the threshold processing appear.

  Such a threshold update type threshold processing is performed by updating the threshold at a threshold level shifted by a predetermined value from the threshold at the time of extracting the minimum point or other local minimum points in the error map at a specified number of times. Is configured to perform threshold processing.

  The labeling unit 34 acquires a label managed by the motion vector accumulation unit 36, and for each region excluding the region including the stored minimum point coordinate from the regions obtained by the error map after the threshold processing, the labeling unit 34 Is performed, and a labeled error map is output to the region-specific minimum point search unit 35. The label may be in any form as long as it indicates an index assigned to each motion vector candidate.

  The area-specific minimum point search unit 35 searches and extracts the minimum point of each area in the error map for each label (for each area) with respect to the labeled error map, and corresponds the minimum point coordinates of each area. It outputs to the motion vector storage part 36 as a motion vector candidate. Thereafter, the region-specific minimum point search unit 35 notifies the threshold setting / processing unit 33 that a plurality of motion vector candidates for the error map after the new threshold processing has been stored. Upon receiving the notification, the threshold setting / processing unit 33 repeatedly updates the threshold at the specified number of times, executes threshold processing, and controls further motion vector candidate determination operations.

  The motion vector accumulation unit 36 accumulates a plurality of motion vectors obtained by the minimum point search unit 32 and the region-specific minimum point search unit 35, and a block (prediction target block) to be interpolated on a frame corresponding to the sub-picture to be interpolated. Are output to the outside as a plurality of motion vector candidates. The motion vector accumulating unit 36 also outputs a labeled motion vector for each of a plurality of motion vector candidates by managing the motion vector candidates corresponding to the stored minimum point coordinates with labels. Each of the plurality of motion vector candidates can be identified by the index of each label.

(Operation of preferred example of bidirectional motion vector prediction unit)
The operation of the preferred example of the bidirectional motion vector prediction units 12 and 51 will be described in detail with a more specific example. Here, an example in which the main video is an even-numbered frame group of the input video and the sub-video is an odd-numbered frame group of the input video will be mainly described.

  FIG. 4 is a flowchart showing a preferred example of motion prediction processing in the bidirectional motion vector prediction units 12 and 51. First, the bidirectional motion prediction unit 31 performs bidirectional motion prediction between two input blocks b1 and b2 that are respectively referred to in successive main video frames, and generates an error map related to the prediction target block (step S11). .

  An example of bidirectional motion prediction by the bidirectional motion prediction unit 31 is shown in FIG. A search range centered on the same coordinates as the prediction target block on the frame in the sub-picture to be restored is set on the frame of the main video positioned in front of and behind the prediction target block (input block b1, input block). b2) In the input blocks b1 and b2, the search block is moved symmetrically with respect to the prediction target block. By calculating an error (square error, absolute error, etc.) between search blocks in the input blocks b1, b2 at each search block position (i, j), an error map relating to the prediction target block is generated.

  Next, the minimum point search unit 32 searches and extracts the minimum point of the entire error map from the error map generated by the bidirectional motion prediction unit 31, and sets the minimum point coordinate as a corresponding motion vector candidate. (Step S12). The motion vector candidate corresponding to the minimum point coordinate is stored in the motion vector storage unit 36. Further, the threshold value for the error map corresponding to the extracted minimum point is set to the initial value of the threshold value used in the threshold value setting / processing unit 33.

Next, when the set threshold value is updated and a new motion vector candidate is extracted (step S13: Yes), the threshold value setting / processing unit 33 uses the threshold value from the initial value (th ₀ ) to a predetermined value (step width: STEP). The threshold value (th ₁ ) is set (updated) at the threshold level shifted in () (step S14). Subsequently, the threshold setting / processing unit 33 generates a new error map after threshold processing by performing threshold processing on the error map with the set (updated) threshold (step S15).

  Next, the labeling unit 34 obtains a label managed by the motion vector accumulation unit 36, and for each region excluding the region including the stored minimum point coordinate among the regions obtained by the error map after the threshold processing. Labeling is executed by attaching a label to the error map to form a labeled error map (step S16).

  Next, the minimum point search unit 35 for each area searches and extracts the minimum point of each area in the error map for each label (for each area) with respect to the labeled error map, and the minimum point coordinates of each area Is a corresponding motion vector candidate (step S17). The minimum point coordinates are stored in the motion vector storage unit 36. The minimum point coordinates of each region correspond to the minimum point coordinates with respect to the original error map. Here, when it is determined that the motion vector accumulation unit 36 has stored a predetermined number or more of motion vector candidates or there are no more regions (regions of a predetermined size) to be labeled in the error map, It can be configured to process all candidate motion vectors as determined.

Here, a threshold value process and a labeling process performed on the error map will be described in detail as a method of extracting the minimum point coordinates of each area of the error map. 6A and 6B are diagrams for explaining threshold processing of the error map according to the present invention, and a search block position (i, j) in the input block b1 in the first reference frame of the main video. On the basis of the error from the search block in the input block b2 in the second reference frame of the main video, the sub-position located between the search blocks of the first reference frame and the second reference frame in terms of time and space coordinates. An example of an error map in which error amounts corresponding to blocks (prediction target blocks) to be interpolated in a video frame are mapped is shown. When an error map having a plurality of minimum points including the minimum point (see FIG. 6A) is subjected to threshold processing, a plurality of regions are generated in addition to the region including the minimum point (see FIG. 6B). . Each of these areas is labeled, and a minimum point search is performed for each label. As a result, it is possible to search for the minimum point of the error map with a small amount of calculation. The threshold setting information may be set for each block or video and transmitted as side information, or the threshold setting procedure may be determined in advance between the transmission side and the reception side. As an example, a threshold level that is shifted by a predetermined value (step width: STEP) with respect to the threshold initial value (th ₀ ) obtained when searching for the minimum point of the prediction target block is determined in advance between the transmission side and the reception side. To leave. Even when it is configured to change the predetermined value (step width: STEP) according to the threshold value to be updated, a function of the step width according to the threshold value can be used as a function between the transmission side and the reception side. It may be determined in advance.

  Further, by further updating the threshold value, a more accurate minimum point search is possible. That is, in this update type threshold processing, the threshold value is updated at a threshold level shifted by a predetermined value from the threshold value at the time of minimum point extraction by the minimum point search unit 32 by the specified number of times, and the error map is updated with the updated threshold value Threshold processing is performed.

For example, in FIG. 4, when the set threshold value is further updated and further motion vector candidates are extracted (step S13: Yes), the threshold level shifted from the threshold value (th ₁ ) by a predetermined value (step width: STEP) The threshold value (th ₂ ) is updated with (Step S14). Subsequently, the threshold setting / processing unit 33 generates a new error map after threshold processing by performing threshold processing on the error map with the updated threshold (step S15).

Subsequently, the labeling unit 34 acquires a label managed by the motion vector accumulation unit 36, and among the regions obtained by the error map after the threshold processing, the already stored minimum point coordinates (thresholds th ₀ and th ₁ are used). Labeling is performed by attaching a label to each area excluding the area including the saved minimum point coordinates), and a labeled error map is formed (step S16).

  Subsequently, the minimum point search unit 35 for each region searches and extracts the minimum point of each region in the error map for each label (for each region) with respect to the labeled error map, and the minimum point coordinates of each region Is a corresponding motion vector candidate (step S17). The motion vector candidates corresponding to the minimum point coordinates for each region are stored in the motion vector storage unit 36.

  Then, after repeating this update type threshold processing a specified number of times, the motion prediction processing in the bidirectional motion vector prediction units 12 and 51 is terminated (step S13: No).

An example of updating the threshold value twice will be briefly described with reference to FIG. First, the bidirectional motion vector prediction units 12 and 51 search for the minimum value of the entire error map (see FIG. 7B) for the error map (see FIG. 7A), and correspond to this minimum point coordinate. The motion vector accumulation unit 36 stores the motion vector candidates. Further, to set the threshold for the error map corresponding to the minimum point extracted as the threshold value th _0. Then, to update the threshold value _{(th 1 = th 0 + STEP} ) at plus a predetermined value (step width STEP) to the threshold th _0, to form an error map of performing threshold processing and labeling processing (Fig. 7 (C )reference). Of the plurality of areas formed by this error map, an area including the minimum point coordinates already stored in the motion vector storage unit 36 is an excluded area. Subsequently, the bidirectional motion vector prediction units 12 and 51 perform a minimum point search for each region other than the excluded region, and the motion vector storage unit 36 sets the newly acquired minimum point coordinate as a corresponding new motion vector candidate. save.

Further, the threshold value (th ₂ = th ₁ + STEP) is updated by adding a predetermined value (step width STEP) to the threshold value th ₁ , and an error map is formed by performing threshold processing and labeling processing (see FIG. 7D). ). Of the plurality of regions formed by this error map, the region including the minimum point coordinates already stored by the motion vector storage unit 36 is an excluded region, and the bidirectional motion vector predicting units 12 and 51 A minimum point search is performed for the region, and the newly acquired minimum point coordinate is stored in the motion vector storage unit 36 as a corresponding new motion vector candidate. As a result, it is possible to search for the minimum point of the error map with a small amount of calculation. In addition, although other existing techniques may be used for the minimum point search of the error map, other existing techniques can determine a corresponding motion vector candidate for any local minimum point, but according to the above preferred example If it is a method, the threshold value process with respect to an error map can be used to enable a minimum point search that coincides with high accuracy between the transmission side and the reception side, and further, the calculation load of the minimum point search can be reduced.

  As described above, the frame thinning device 1 and the frame interpolation device 5 according to the first embodiment of the present invention reduce the amount of information of the main stream and improve the transmission efficiency by thinning and transmitting the frame group of the high frame rate video. At the same time, judder can be reduced by bidirectional motion prediction and motion compensation. In particular, by comparing the interpolation results of a plurality of vector candidates with the original image in advance on the transmission side and transmitting only the optimal motion vector index together with the main video, it is not necessary to transmit the motion vector itself. The amount of information can be reduced.

  Next, a video encoding device and a video decoding device according to a second embodiment of the present invention will be described.

[Second Embodiment]
(Video encoding device)
FIG. 8 is a block diagram showing an outline of the video encoding device 10 according to the second embodiment of the present invention. The video encoding device 10 is a device that thins out and encodes a group of frames for a predetermined video, and transmits a frame division unit 11, a bidirectional motion vector prediction unit 12, a frame interpolation unit 13, a comparison unit 14, and side information encoding. Unit 15, video encoding unit 16 and local decoding unit 17. In FIG. 8, the same components as those in FIG. 1 are denoted by the same reference numerals.

  The frame dividing unit 11 divides an input video such as a high frame rate video into a frame group (main video) to be transmitted to the receiving side and a frame group (sub video) to be transmitted to the receiving side. The sub video is output to the comparison unit 14.

  The video encoding unit 16 encodes the main video and outputs it as a bit stream (main stream) to the local decoding unit 17 and the outside.

  The local decoding unit 17 generates a main video that is locally decoded from the main stream, and temporarily stores it in a predetermined memory (not shown). The locally decoded main video is temporarily stored in a manner that can be read by the bidirectional motion vector prediction unit 12 and the frame interpolation unit 13.

  The bidirectional motion vector prediction unit 12 refers to a locally decoded main video frame (preferably, a main video frame positioned before and after in time) that is temporally continuous with respect to a sub video frame to be restored. Then, a plurality of motion vectors of a prediction target block for restoring the sub-video are predicted by bidirectional motion prediction, and a plurality of motion vector candidates are output to the frame interpolation unit 13. The configuration and operation of the bidirectional motion vector prediction unit 12 can be the same as in the first embodiment.

  The frame interpolation unit 13 generates each interpolated video corresponding to the plurality of motion vector candidates by motion compensation using the locally decoded main video and the predicted plurality of motion vector candidates, and outputs the interpolated video to the comparison unit 14.

  As in the first embodiment, the comparison unit 14 compares each interpolated video with the sub-video of the original image, selects an interpolated video with the least difference from the original image, and selects the selected motion vector candidate from the plurality of motion vector candidates. The motion vector corresponding to the interpolated video is determined, and the index of the determined motion vector is output to the side information encoding unit 15 as side information.

  As in the first embodiment, the side information encoding unit 15 encodes the side information using the existing lossless encoding and outputs it as a bit stream (side stream) to the outside. When the side information is not encoded, the function of the side information encoding unit 15 is unnecessary, and the side information can be directly output as a side stream directly from the comparison unit 14.

  As described above, the frame thinning device 1 transmits a frame group with respect to a predetermined video together with predetermined side information.

(Video decoding device)
FIG. 9 is a block diagram showing an outline of a video decoding device 50 according to the second embodiment of the present invention. The video decoding device 50 is a device that decodes a video in which a frame group has been thinned out and encoded with respect to a predetermined video and interpolates the frame to restore the predetermined video. A decoding unit 52, a motion vector selection unit 53, a frame interpolation unit 54, a frame synthesis unit 55, and a video decoding unit 56 are provided. In FIG. 9, the same reference numerals are assigned to the same components as those in FIG.

  The video decoding unit 56 is a video in which frames are intermittently thinned and encoded with respect to a predetermined video such as a high frame rate image transmitted as a main stream (that is, a main video output from the video encoding device 10). ) Is decoded by a decoding process corresponding to the encoding process on the transmission side and output to the bidirectional motion vector prediction unit 51 and the frame synthesis unit 55.

  The bidirectional motion vector prediction unit 51 refers to the decoded main video frames that are temporally continuous with respect to the sub video frames to be restored (preferably, the decoded main video frames positioned before and after in time). Thus, a plurality of motion vectors of the prediction target block for restoring the sub video are predicted, and a plurality of motion vector candidates are output to the motion vector selection unit 53. The configuration and operation of the bidirectional motion vector prediction unit 51 can be the same as in the first embodiment.

  As in the first embodiment, the side information decoding unit 52 supports side information transmitted as a side stream (encoded side information output from the video encoding device 10) for encoding processing on the transmission side. Thus, side information indicating the optimal motion vector index of the prediction target block is acquired and output to the motion vector selection unit 53. When the side information is not encoded, the function of the side information decoding unit 52 is unnecessary, and the motion vector selection unit 53 can be configured to acquire the side information directly from the side stream.

  Similar to the first embodiment, the motion vector selection unit 53 refers to side information indicating the optimal motion vector index of each prediction target block, and generates a plurality of motion vector candidates generated by the bidirectional motion vector prediction unit 51. One motion vector is selected according to this index and output to the frame interpolation unit 54.

  As in the first embodiment, the frame interpolation unit 54 uses the received main video and the selected motion vector to restore the interpolated video frame corresponding to the sub video, and performs frame synthesis on the restored sub video. To the unit 55.

  The frame synthesizing unit 55 performs frame synthesis on the decoded main video and the restored sub video in the order thinned out on the transmission side, and restores the restored video obtained by restoring a predetermined video such as the high frame rate image to the outside Output to.

  Therefore, in the video encoding device 10 and the video decoding device 50 according to the second embodiment of the present invention, the same advantages as those in the first embodiment can be obtained. Further, in the second embodiment, when a predetermined video such as a high frame rate video is thinned and encoded and transmitted, the high frame rate video is encoded by the existing low frame rate codec alone. Thus, the coding efficiency can be improved as compared with the case of using the low frame rate codec in parallel.

  Although the present invention has been described with reference to specific examples, the present invention is not limited to the examples of the above-described embodiments, and various modifications can be made without departing from the technical idea thereof. For example, although an example in which the interval between frames to be thinned is 1 frame in 2 frames, the frequency of frames to be thinned may be set freely. Also, when the frequency of thinning out is 2 out of 3 frames, as in the above example, the bidirectional motion vector prediction units 12 and 51 decode the motion vectors of the block to be predicted positioned temporally forward and backward. Predict from the main video. At this time, as shown in FIG. 10, the search block for bidirectional motion prediction moves asymmetrically instead of moving point-symmetrically, and thereby an error map can be acquired. Similarly, the motion vector of the prediction target block may be predicted from two post-decoding main videos that are temporally forward. Therefore, instead of bi-directionally predicting motion, when determining motion vector candidates of a prediction target block, frames of a main video (or decoded main video) that are temporally continuous with respect to the frame of the prediction target block are used. It can be set as the motion prediction which predicts using multiple. In this way, motion prediction is performed not only with reference to the main video frame “positioned before and after in time” with respect to the sub-video frames to be thinned out, but also with reference to the main video frame “continuous in time”. Thus, a plurality of motion vectors of a prediction target block for restoring the sub video can be predicted, and a plurality of motion vector candidates can be determined.

  In the above example, the thinning interval is uniform, but it may not be uniform. In this case, thinning may be performed according to a thinning method set in advance on the transmitting side and the receiving side, or a thinning procedure may be transmitted as side information.

  In the above example, a plurality of motion vector candidates are generated by bidirectional motion prediction between two main videos (decoded main video), but bidirectional between three or more main videos (decoded main video). A plurality of motion vector candidates may be generated by motion prediction.

  In the above example, the transmission of the side information has been described as an example of transmission as a side information stream. However, for example, the transmission is performed in a form adapted to an existing transmission system such as multiplexing with respect to a main stream related to transmission of an image signal. be able to.

  In addition, the function of each component of the frame thinning device 1 and the frame interpolation device 5, the video encoding device 10, and the video decoding device 50 of each embodiment can be realized by a computer. A program for realizing each component according to the invention is stored in a memory (not shown) provided inside or outside the computer. By appropriately reading a program describing processing contents for realizing the function of each component under the control of a central processing unit (CPU) provided in the computer and executing the program, each embodiment The function of each component of the apparatus can be realized by a computer. Here, the function of each component may be realized by a part of hardware.

  According to the present invention, when transmitting a predetermined video by thinning out a frame, the amount of information of the main stream is reduced and the transmission efficiency is improved, and the judder is reduced by bidirectional motion prediction / motion compensation according to the present invention. Therefore, it is useful for applications that transmit high frame rate video.

DESCRIPTION OF SYMBOLS 1 Frame decimation apparatus 5 Frame interpolation apparatus 10 Video encoding apparatus 11 Frame division part 12 Bidirectional motion vector prediction part 13 Frame interpolation part 14 Comparison part 15 Side information encoding part 16 Video encoding part 17 Local decoding part 31 Bidirectional motion Prediction unit 32 Minimum point search unit 33 Threshold setting / processing unit 34 Labeling unit 35 Region-specific minimum point search unit 36 Motion vector storage unit 50 Video decoding device 51 Bidirectional motion vector prediction unit 52 Side information decoding unit 53 Motion vector selection unit 54 Frame interpolation unit 55 Frame composition unit 56 Video decoding unit

Claims (12)

A frame decimation device that decimates and transmits a frame group for a predetermined video,
Frame dividing means for dividing a predetermined video into a main video consisting of a frame group transmitted to the receiving side and a sub video consisting of a frame group not transmitted;
Refer to multiple temporal frames of the main video for the sub video, predict multiple motion vectors for the target block to restore the sub video by motion prediction, and determine multiple motion vector candidates Motion vector prediction means for
Frame interpolation means for generating respective interpolated videos corresponding to a plurality of motion vector candidates by motion compensation using a plurality of motion vector candidates predicted as a main video;
Comparing means for comparing each interpolated video with the sub video and determining a motion vector corresponding to the interpolated video with the smallest difference among the plurality of motion vector candidates;
An output means for outputting the determined motion vector index as side information to the outside together with the main video;
A frame thinning device comprising:   The motion vector prediction means performs motion prediction between input blocks that are referred to in the continuous main video frames, so that the prediction target block in the block to be interpolated with reference to the input block is detected as an error. A plurality of motion vector candidates including a motion vector candidate corresponding to a minimum point and a motion vector candidate corresponding to a minimum point that is minimum with respect to peripheral errors other than the minimum point are determined. The frame thinning device according to 1. The motion vector prediction means includes:
Motion prediction means for generating an error map related to the prediction target block by performing motion prediction between input blocks referred to in the continuous main video frames;
Minimum point search means for searching and extracting the minimum point of the entire error map, and determining the extracted minimum point coordinates as a first motion vector candidate;
Threshold setting / processing means for generating an error map related to the prediction target block after the new threshold processing by performing threshold processing with a predetermined value on the error map;
Among the regions formed in the error map divided by the threshold processing, a minimum point is searched for and extracted from regions excluding the region including the minimum point coordinate, and the extracted minimum point coordinate is the second and subsequent motion vector candidates. A region-by-region minimum point search means determined as:
The second and subsequent motion vector candidates obtained by operating the threshold value setting / processing means and the region-specific minimum point searching means at a specified number of times of one or more, and the first motion vector candidates Motion vector storage means for storing a plurality of motion vector candidates;
The frame thinning-out apparatus according to claim 1 or 2, further comprising: A frame interpolation device that interpolates a frame from a thinned video obtained by thinning a frame group with respect to a predetermined video and restores the predetermined video,
Input side information indicating the motion vector index of the prediction target block used to restore the main video in which frames are skipped intermittently with respect to a predetermined video and the sub video corresponding to the thinned frames. Input means to
Refer to multiple temporal frames of the main video for the sub video, predict multiple motion vectors for the target block to restore the sub video by motion prediction, and determine multiple motion vector candidates Motion vector prediction means for
Motion vector selection means for referring to the side information and selecting one motion vector for a prediction target block having the plurality of motion vector candidates according to the index;
Frame interpolation means for generating an interpolated video corresponding to the sub video using the main video and the selected motion vector;
Frame synthesis by performing frame synthesis by interpolating the main video with a frame of the interpolated video, and restoring the predetermined video; and
A frame interpolation apparatus comprising:   The motion vector prediction means performs motion prediction between input blocks that are referred to in the continuous main video frames, so that the prediction target block in the block to be interpolated with reference to the input block is detected as an error. A plurality of motion vector candidates including a motion vector candidate corresponding to a minimum point and a motion vector candidate corresponding to a minimum point that is minimum with respect to peripheral errors other than the minimum point are determined. 5. The frame interpolation apparatus according to 4. The motion vector prediction means includes:
Motion prediction means for generating an error map related to the prediction target block by performing motion prediction between input blocks referred to in the continuous main video frames;
Minimum point search means for searching and extracting the minimum point of the entire error map, and determining the extracted minimum point coordinates as a first motion vector candidate;
Threshold setting / processing means for generating an error map related to the prediction target block after the new threshold processing by performing threshold processing with a predetermined value on the error map;
Among the regions formed in the error map divided by the threshold processing, a minimum point is searched for and extracted from regions excluding the region including the minimum point coordinate, and the extracted minimum point coordinate is the second and subsequent motion vector candidates. A region-by-region minimum point search means determined as:
The second and subsequent motion vector candidates obtained by operating the threshold value setting / processing means and the region-specific minimum point searching means at a specified number of times of one or more, and the first motion vector candidates Motion vector storage means for storing a plurality of motion vector candidates;
The frame interpolating device according to claim 4, comprising: A video encoding device that thins out and encodes a group of frames for a predetermined video,
Frame dividing means for dividing a predetermined video into a main video consisting of a frame group transmitted to the receiving side and a sub video consisting of a frame group not transmitted;
Video encoding means for encoding the main video and outputting it externally;
Local decoding means for generating locally decoded main video from the encoded main video;
A plurality of motion vectors of a prediction target block for reconstructing the sub video by motion prediction are referred to by referring to a plurality of frames of the locally decoded main video that are temporally continuous with respect to the sub video, and a plurality of motions Motion vector prediction means for determining vector candidates;
Frame interpolation means for generating respective interpolated videos corresponding to a plurality of motion vector candidates by motion compensation using the locally decoded main video and a plurality of predicted motion vector candidates;
Comparing means for comparing each interpolated video with the sub video and determining a motion vector corresponding to the interpolated video with the smallest difference among the plurality of motion vector candidates;
An output means for outputting the determined motion vector index as side information to the outside;
A video encoding device comprising: A video decoding device for interpolating a frame from a thinned video obtained by thinning and encoding a frame group with respect to a predetermined video, and restoring the predetermined video,
Video decoding means for inputting a thinned video obtained by intermittently decimating and encoding a frame with respect to a predetermined video as a main video, performing a decoding process on the main video, and generating a main video after decoding;
Input means for inputting side information indicating an index of a motion vector of a prediction target block used for restoring a sub video corresponding to the thinned frame group;
Refer to multiple temporal frames of the main video for the sub video, predict multiple motion vectors for the target block to restore the sub video by motion prediction, and determine multiple motion vector candidates Motion vector prediction means for
Motion vector selection means for referring to the side information and selecting one motion vector for a prediction target block having the plurality of motion vector candidates according to the index;
Frame interpolation means for generating an interpolated video corresponding to the sub video using the decoded main video and the selected motion vector;
Frame synthesizing by interpolating the decoded main video with a frame of the interpolated video, and frame synthesizing means for restoring the predetermined video;
A video decoding device comprising: On the computer,
Dividing a predetermined video into a main video consisting of a frame group to be transmitted to the receiving side and a sub video consisting of a frame group not to be transmitted;
Refer to multiple temporal frames of the main video for the sub video, predict multiple motion vectors for the target block to restore the sub video by motion prediction, and determine multiple motion vector candidates And steps to
Generating each interpolated video corresponding to a plurality of motion vector candidates by motion compensation using a plurality of motion vector candidates predicted as a main video;
Comparing each interpolated video with the sub-video and determining a motion vector corresponding to an interpolated video with the smallest difference among the plurality of motion vector candidates;
Outputting the determined motion vector index as side information to the outside together with the main video;
A program for running On the computer,
Inputs the main video with intermittent frames thinned out for a given video and side information indicating the motion vector index of the prediction target block used to restore the sub video corresponding to the thinned frames. And steps to
Refer to multiple temporal frames of the main video for the sub video, predict multiple motion vectors for the target block to restore the sub video by motion prediction, and determine multiple motion vector candidates And steps to
Referring to the side information and selecting one motion vector for a prediction target block having the plurality of motion vector candidates according to the index;
Generating an interpolated video corresponding to the sub-video using the main video and the selected motion vector;
Performing frame synthesis by interpolating the main video with the frame of the interpolated video and restoring the predetermined video;
A program for running On the computer,
Dividing a predetermined video into a main video consisting of a frame group to be transmitted to the receiving side and a sub video consisting of a frame group not to be transmitted;
Encoding the main video and outputting to the outside;
Generating locally decoded main video from the encoded main video;
A plurality of motion vectors of a prediction target block for reconstructing the sub video by motion prediction are referred to by referring to a plurality of frames of the locally decoded main video that are temporally continuous with respect to the sub video, and a plurality of motions Determining vector candidates;
Generating each interpolated video corresponding to a plurality of motion vector candidates by motion compensation using the locally decoded main video and a plurality of predicted motion vector candidates;
Comparing each interpolated video with the sub-video and determining a motion vector corresponding to an interpolated video with the smallest difference among the plurality of motion vector candidates;
Outputting the determined motion vector index as side information to the outside;
A program for running On the computer,
Inputting a thinned video in which frames are intermittently thinned and encoded with respect to a predetermined video as a main video, performing a decoding process on the main video, and generating a decoded main video;
Inputting side information indicating a motion vector index of a prediction target block used to restore a sub-video corresponding to the thinned frame group;
Refer to multiple temporal frames of the main video for the sub video, predict multiple motion vectors for the target block to restore the sub video by motion prediction, and determine multiple motion vector candidates And steps to
Referring to the side information and selecting one motion vector for a prediction target block having the plurality of motion vector candidates according to the index;
Generating an interpolated video corresponding to the sub video using the decoded main video and the selected motion vector;
Performing frame synthesis by interpolating the decoded main video with a frame of the interpolated video and restoring the predetermined video;
A program for running