JP6055230B2 - Transmission device, reception device, and program - Google Patents

Description

  The present invention relates to a transmission device that thins out and transmits a part of a frame of an input video, a reception device that interpolates and generates a frame thinned out by the transmission device, and a program thereof.

  Conventionally, in video (moving image) compression, when a certain frame is encoded, information is reduced by using a time correlation with another frame at a close time. However, the larger the image size and the higher the frame rate, the more difficult it is to encode, and there is a limit to the image size and frame rate that can be processed, especially in real-time encoding processing. Therefore, in order to process video that exceeds the limits of image size and frame rate, it is necessary to encode the video by decimating the size and frame rate on the transmission side and interpolating the thinned pixels and frames on the reception side There is.

  For example, error correction technology was applied as a video encoding method that transmits frames with temporal information on the transmission side and side information to restore the thinned frames from undecimated frames (key frames). Distributed Video Coding has been proposed.

  On the other hand, there is a method of presenting the same frame a plurality of times in order to convert the image speed (frame frequency) to a higher frame frequency. For example, the frame frequency can be converted to 60 Hz by presenting each frame with a frame frequency of 30 Hz twice. There is also a technique in which the number of times the same frame is presented is variable. For example, in a technique called 2-3 conversion, an odd-numbered frame having a frame frequency of 24 Hz is presented twice and an even-numbered frame is presented three times, whereby an image having a frame frequency of 60 Hz can be obtained.

  There is also a method of determining the pixel value of the interpolation frame by the average of the pixel values of two frames adjacent to the time of the interpolation frame or the weighted average.

  Furthermore, a method of generating a smoother interpolation frame by performing motion compensation using a motion vector of a video is known (see, for example, Patent Document 1). In video coding with frame decimation, a technique is known in which the effectiveness of frame interpolation is evaluated on the transmission side and adaptive decimation is performed (see, for example, Patent Document 2).

Japanese Patent No. 3495485 Japanese Patent No. 4930409

  However, the conventional method of presenting the same frame multiple times has a problem that the motion of the video becomes awkward. Also, according to the method of generating an interpolation frame by averaging or weighted averaging, although the awkwardness of video motion is improved, there is a problem that the moving object becomes a double image in the interpolation frame and the resolution is lowered, and the interpolation frame There is a problem in that the difference in image quality from the original frame which is not interpolated looks like flicker.

  In addition, in the method as disclosed in Patent Document 1, although the awkwardness of motion and the double image are improved, the motion compensation error when the subject having different motion is included in the same block, and the coarseness of the motion estimation accuracy The boundary of the block used for motion compensation may be conspicuous due to the decimal accuracy error of the sample points caused by the above. If motion compensation is performed with decimal pixel accuracy, the decimal accuracy error of the sample point can be improved, but the blurring and ringing of the contour may occur due to the characteristics of the digital filter that moves the pixel position (phase shift) with decimal pixel accuracy. May occur.

  Further, in the technique such as Patent Document 2, the thinning control is performed so that the frame interpolation on the receiving side works effectively at the time of frame thinning, but the mechanism and the verification result for verifying the reference frame interpolation method on the transmitting side Is not provided to the receiver. For this reason, it has been difficult to design the receiving side to be in an appropriate state.

  In view of such circumstances, an object of the present invention is to provide a transmission capable of preventing a decrease in resolution and flicker and obtaining a sharp contour image including a high-frequency component when generating an interpolated image from an input video. An apparatus, a receiving apparatus, and a program thereof are provided.

In order to solve the above-described problem, a transmitting apparatus according to the present invention is a transmitting apparatus that thins out and transmits a part of a frame of an input video, and includes a transmission frame that is a transmitted frame of the input video and a frame that is not transmitted. A time division unit for determining a thinned-out image, reference frame determining means for determining a plurality of sets of a motion estimation original image, a motion estimation target image, and a motion compensation original image from the transmission frame, and the reference frame determination means For each of the plurality of sets determined in step (b), a corresponding region having a pixel value pattern similar to the block region of the motion estimation source image is determined from the motion estimation destination image, and the block region of the motion estimation source image and the motion estimation are determined. each motion estimation unit for obtaining the corresponding positional relationship between the corresponding region of the previous image, for a plurality of sets determined in the reference frame determining means, said corresponding position A motion compensation residual evaluation unit that calculates a residual between the image obtained by motion compensation of the motion compensated original image to a frame position of the thinned image based on a relationship and the thinned image for each block region; and a value of the residual And a determination unit that generates determination information indicating the effectiveness of motion compensation for each block region and transmits the determination information to the reception side .

  Furthermore, in the transmitting apparatus according to the present invention, the transmitter further includes a transmission simulation unit that simulates deterioration due to encoding when a frame is encoded and transmitted, and generates a frame deteriorated due to encoding. A plurality of sets of a motion estimation source image, a motion estimation destination image, and a motion compensation source image are determined from the frame generated by the transmission simulation unit.

  In order to solve the above-described problem, a receiving apparatus according to the present invention is a receiving apparatus that interpolates and generates a thinned image thinned out by the transmitting apparatus, and a motion estimation original image from a frame transmitted from the transmitting apparatus, Reference frame determination means for determining a plurality of sets of a motion estimation destination image and a motion compensation source image, and a block for which the determination information is acquired from the transmission apparatus and in which motion compensation is notified by the determination information A corresponding region having a pixel value pattern similar to the block region of the motion estimation source image is determined from the motion estimation destination image, and the corresponding positional relationship between the block region of the motion estimation source image and the corresponding region of the motion estimation destination image A motion estimator for obtaining a motion compensation source image for obtaining a superimposition destination coordinate of the motion compensation source image based on the corresponding positional relationship, and a plurality of sets of motion compensation source images. A superimposing unit that generates an interpolated image by superimposing on the superimposition destination coordinates, and a time multiplexing unit that time-multiplexes the frame transmitted from the transmission device and the interpolated image generated by the superimposing unit. Features.

  Furthermore, in the receiving apparatus according to the present invention, the superimposing unit weights the motion compensation image according to the time position of the motion estimation original image, the motion estimation destination image, the motion compensation original image, and the interpolated image. Then, it is superimposed on the superimposition destination coordinates to generate an interpolated image.

  Furthermore, in the receiving apparatus according to the present invention, the superimposing unit weights the motion compensated image for each block region according to the similarity between the block region of the motion estimation source image and the corresponding region of the motion estimation destination image. Then, it is superimposed on the superimposition destination coordinates to generate an interpolated image.

  In order to solve the above problems, a program according to the present invention causes a computer to function as the transmission device or the reception device.

  According to the present invention, it is possible to thin out some frames of an input video on the transmission device side and to interpolate frames thinned out on the reception device side with high accuracy. As a result, transmission with better rate distortion characteristics (RD characteristics) is possible than when all frames are transmitted.

It is a block diagram which shows the structural example of the transmitter which concerns on one Embodiment of this invention. It is a block diagram which shows the structural example of the residual evaluation part in the transmitter which concerns on one Embodiment of this invention. It is a figure explaining operation | movement of the residual evaluation part in the transmitter which concerns on one Embodiment of this invention. It is a figure which shows the bit pattern sequence which the determination part in the transmitter which concerns on one Embodiment of this invention outputs. It is a block diagram which shows the structural example in case the transmitter which concerns on one Embodiment of this invention has a transmission simulation part. It is a block diagram which shows the structural example of the receiver which concerns on one Embodiment of this invention. It is a block diagram which shows the structural example of the interpolation part in the receiver which concerns on one Embodiment of this invention. It is a figure explaining operation | movement of the interpolation part in the receiver which concerns on one Embodiment of this invention.

  Hereinafter, an embodiment of a transmission device and a reception device according to the present invention will be described in detail with reference to the drawings. In this specification, f indicates a frame number, and I (f) indicates an image of the frame number f. The frame time position is represented by a frame number.

[Transmitter]
FIG. 1 is a block diagram illustrating a configuration example of a transmission apparatus according to an embodiment of the present invention. As shown in FIG. 1, the transmission apparatus 1 includes a time division unit 10, a transmission frame memory 11, a non-transmission frame memory 12, a scanning unit 13, and K residual evaluation units 14 (14-1, 14). -2, ... 14-K) and a determination unit 15. For convenience of explanation, the transmission frame memory 11 and the non-transmission frame memory 12 are illustrated separately, but may be configured as one frame memory. Further, the transmission frame memory 11 and the non-transmission frame memory 12 may be provided outside the transmission device. The transmission apparatus 1 transmits a part of the frame of the input video by thinning it out.

  The time division unit 10 transmits a transmission frame that is a frame transmitted from the transmission device 1 without being thinned out of the input video, and a non-transmission frame (a thinning frame) that is a frame that is thinned out from the input video and is not transmitted from the transmission device 1. Image). Then, the transmission frame is stored in the transmission frame memory 11 and output to the outside, and the non-transmission frame is stored in the non-transmission frame memory 12.

  The transmission frame memory 11 stores a plurality of frames transmitted (or transmitted) from the transmission device 1. In FIG. 1, the transmission frame memory 11 is configured to be able to output the stored frames at three time points to the residual evaluation unit 14.

  The scanning unit 13 outputs block position information indicating the position of a block region (partial region) to be arranged on the input video to the residual evaluation unit 14.

  FIG. 2 is a block diagram illustrating a configuration example of the residual evaluation unit 14. As illustrated in FIG. 2, the residual evaluation unit 14 includes a block cutout unit 141, a motion estimation unit 142, and a motion compensation residual evaluation unit 143.

  FIG. 3 is a diagram for explaining an outline of the operation of the residual evaluation unit 14. The motion estimation original image is an image frame that is a motion estimation target source in the motion estimation unit 142. The motion estimation destination image is an image frame that is a target of motion estimation in the motion estimation unit 142. The motion compensation original image is an image frame that is subject to motion compensation in the motion compensation residual evaluation unit 143.

Although the frame number f _b of the frame number of the motion estimation original image f _a and the motion estimation target image must match one the frame number f _p of the motion estimation original image frame number f _a and the motion compensated original image may match, also it has a frame number f _p of the frame number f _b and the motion compensation based on image motion estimation target image may coincide. In FIG. 3, superscript ⁽¹⁾ such as I (f _a ⁽¹⁾ ) means processing by the residual evaluation unit 14-1, and superscript (2) is residual evaluation. This means processing by the unit 14-2.

The residual evaluation unit 14 extracts the motion estimation original image I (f _a ), the motion estimation destination image I (f _b ), and the motion compensation original image I (f _p ) from the transmission frame stored in the transmission frame memory 11. Reference frame determining means for determining a set is provided. These sets of reference frames are different for each residual evaluation unit 14. The residual evaluation unit 14 calculates a motion for each block region (partial region) in the image based on the motion estimation original image I (f _a ) and the motion estimation destination image I (f _b ), and based on the motion The motion compensation original image I (f _p ) and the thinned image I (f _q ) are aligned. Then, a residual e between the motion compensated original image I (f _p ) and the thinned image I (f _q ) subjected to the alignment is calculated and output to the determination unit 15.

The block cutout unit 141 cuts out the block region B at the position specified by the block position information input from the scanning unit 13 from the motion estimation original image I (f _a ) stored in the transmission frame memory 11 to perform motion estimation. Output to the unit 142. The pixel value at the image coordinate (x, y) is denoted as I (f, x, y), and the pixel value at the image coordinate (i, j) of the block area B of horizontal W pixel and vertical H pixel is defined as B (i, j ), The block cutout unit 141 cuts out the block region B, for example, as in the following equation (1) based on the upper left coordinates (s, t) of the block region B input from the scanning unit 13.

The motion estimation unit 142 uses a motion estimation destination stored in the transmission frame memory 11 as a corresponding region having a pixel value pattern similar to the block region B of the motion estimation original image I (f _a ) cut out by the block cutout unit 141. Search based on the similarity in the image I (f _b ). Then, the corresponding position relationship between the block area B of the motion estimation original image I (f _a ) and the corresponding area of the motion estimation destination image I (f _b ) is obtained, and the corresponding position information is output to the motion compensation residual evaluation unit 143. . The corresponding position information is expressed as (s, t, u, v), for example, with the upper left coordinates (s, t) of the block B input from the scanning unit 13 and the upper left coordinates (u, v) of the corresponding area as a set. can do. The corresponding position information is expressed as (s, t, u-s, vt) with the upper left coordinates (s, t) of the block B and the motion vector (u-s, v-t) as a set. Can do. The upper left coordinates (u, v) of the corresponding area are calculated by the following equation (2), for example.

  Note that Equation (2) finds the corresponding region by minimizing the sum of absolute error (SAD), but minimizing the sum of squared error (SSD) and normalized cross-correlation. The corresponding region may be obtained by maximizing (NCC: Normalized Cross Correlation).

  Further, the processing of the block cutout unit 141 and the motion estimation unit 142 may be integrated, and the calculations of the equations (1) and (2) may be realized by a single calculation as shown in the following equation (3).

The motion compensation residual evaluation unit 143 performs motion compensation on the motion compensated original image I (f _p ) based on the corresponding position information obtained by the motion estimation unit 142 up to the frame time position of the thinned image, and the non-transmission frame memory. The residual e with the stored thinned image I (f _q ) is calculated for each block area and output to the determination unit 15. The residual e can be estimated by the following equation (4), for example. At this time, the pixel position on the motion compensation original image I (f _p ) and the pixel position on the thinned image I (f _q ) are in accordance with the corresponding positional relationship and the time of each frame as shown in the following equation (5). Obtained by apportionment based on.

  Note that the image frame I is generally discretized into integer values with respect to image coordinates, whereas α, β, γ, and δ are not necessarily integers. Therefore, for example, the pixel value at the decimal pixel position may be defined by convolving a point spread function (for example, a two-dimensional Sinc function or a Gaussian function) with the video frame I, or α, β, γ, and δ may be defined. Integer (for example, rounding) may be performed. In this manner, each residual evaluation unit 14 estimates a residual for each block region B when frame interpolation is performed with reference to different frames.

The determination unit 15 determines the effectiveness of motion compensation for each block region B in accordance with the residual e calculated by the plurality of residual evaluation units 14 (the set of reference frames input to the residual evaluation unit 14). (Validity) is determined, and a bit pattern string indicating whether or not it is valid is output. For example, threshold processing is performed on the residual e of each residual evaluation unit 14 for each block region B of the motion compensation original image I (f _p ) of the scanning unit 13, and the residual e is equal to or less than the threshold value θ (or less). If it is, it is determined to be valid, and if it is an exception, it is determined to be invalid. For example, a block region set by the scanning unit 13 at the nth sample (n is an integer between 1 and N and N is a natural number) is set as _Bn . The residual value output by the residual evaluation unit 14-k (k is a natural number between 1 and K and K is a natural number) for the block region _Bn is set as e _n ^(k) . The determination unit 15 compares the threshold value θ and the residual value with e _n ^(k) and outputs a comparison result flag F _n ^(k) . The flag F _n ^(k) is expressed by, for example, the following formula (6) or the following formula (7).

The bit pattern string (P _n ) _{n = 1, 2,..., N} output from the determination unit 15 can be defined by the following equation (8), for example. P _n is the block area B _n , when the motion compensation evaluated by the residual evaluation unit 14-k is valid, the k-th bit is counted from the LSB side, and when it is not valid, the k-th bit is counted. The pattern is set to 0.

FIG. 4 is a diagram illustrating the bit pattern sequence P _n output from the determination unit 15. In FIG. 4, when K = 8, P _n is represented by a hexadecimal number, and in this example, P ₀ = FF, P ₁ = FA, and P ₂ = FC.

Also, the determination unit 15 determines whether each motion compensation (interpolation method using a set of reference frames evaluated by each residual evaluation unit 14) is effective for each pixel position of the thinned image I (f _q ), for example. The recorded bit pattern sequence may be output. In this case, as a method of determining each pixel position, for example, (1) a method of determining that L (L is a natural number) motion compensation with few residuals is valid and determining that other motion compensation is invalid, (2) A method of determining that motion compensation whose residual is less than (or less than) the threshold value θ is valid and invalidating other motion compensation, or (3) Of L motion compensation where L is small (L is a natural number) A method can be considered in which motion compensation whose residual is equal to or less than (or less than) the threshold value θ is valid and other motion compensation is judged invalid.

The bit pattern array of the determination result at the pixel position (x, y) is set as P (x, y). Also, placing the four pairs of corresponding positions relative to the block area _{B n} and _{(s n, t n, u} n, v n). In the case of the above method (2), the bit pattern array P (x, y) can be obtained by, for example, gradually applying the bit operation represented by the following equation (9). Note that the operator “| =” is the same as the operator “| =” in the C ++ language, and represents a bit operation for returning the result of the logical sum of the left side value and the right side value for each bit to the left side value.

The transmission apparatus 1 sends determination information (for example, bit pattern sequence (P _n ) _{n = 1, 2,..., N} or bit pattern array P (x, y)) indicating the determination result by the determination unit 15 to the reception apparatus. Transmit. Lossless compression may be performed during transmission.

As shown in FIG. 5, the transmission device 1 may further include a transmission simulation unit 16. The transmission simulation unit 16 simulates deterioration due to encoding when encoding and transmitting a frame, generates a frame deteriorated by encoding, and stores the frame in the transmission frame memory 11. For example, the transmission frame is MPEG-4 AVC / H. In the case of encoding by the H.264 system, the transmission frame is locally decoded (local decoding) similarly to the system, and the locally decoded frame image is stored in the transmission frame memory 11. Each of the residual evaluation units 14 includes a set of a motion estimation original image I (f _a ), a motion estimation destination image I (f _b ), and a motion compensation original image I (f _p ) from the frame generated by the transmission simulation unit 16. Decide

As described above, in the transmission device 1, the time division unit 10 determines a transmission frame that is a frame transmitted from the input video and a thinned image that is a frame that is not transmitted, and the residual evaluation unit 14 A plurality of sets of a motion estimation original image I (f _a ), a motion estimation destination image I (f _b ), and a motion compensation original image I (f _p ) are determined from the transmission frame, and the motion estimation original image I (f _a ) Corresponding region having a pixel value pattern similar to that of the block region is determined from the motion estimation target image I (f _b ), and the motion compensation original image I (f _p ) is based on the corresponding positional relationship between the block region and the corresponding region. and image motion compensation to the frame time and position of the thinned image, the residual e in the thinned image I (f _q) is calculated for each block area, the determination unit 15, from the residual of the motion compensation for each block area Effectiveness It generates determination information indicating. Therefore, according to the transmission apparatus 1, it is possible to determine a frame interpolation method to be performed on the reception apparatus side while referring to the original image of the thinned video frame. It is possible to thin out some frames and to interpolate frames thinned out on the receiving apparatus side with high accuracy. As a result, transmission with better rate distortion characteristics (RD characteristics) is possible than when all frames are transmitted.

  In addition, since the transmission apparatus 1 further includes the transmission simulation unit 16, when determining a frame interpolation method to be performed by the reception apparatus, it is possible to make a determination in consideration of deterioration due to encoding. Can be further improved.

  Note that a computer can be suitably used to cause the transmission device 1 to function as described above, and such a computer stores a program describing processing contents for realizing each function of the transmission device 1 in a storage unit of the computer. This can be realized by storing the program and executing it by the CPU of the computer.

[Receiver]
FIG. 6 is a block diagram illustrating a configuration example of a receiving apparatus according to an embodiment of the present invention. As illustrated in FIG. 6, the reception device 2 includes a frame memory 21, K interpolation units 22 (22-1, 22-2,..., 22-K), a superimposition unit 23, and a time multiplexing unit. 24. The number K of interpolation units 22 is the same as the number K of residual evaluation units 14 of the transmission apparatus 1. The receiving device 2 interpolates and generates the thinned image thinned out by the transmitting device 1 described above.

Frame memory 21, a plurality time points stores the video frame I _R transmitted from the transmitter 1. Note that the video frame I _R (f) is the same as the frame I (f) of the input video when the transmission means of the transmission frame is not deteriorated, but when the transmission means of the transmission frame performs irreversible compression. Has undergone degradation due to encoding.

  FIG. 7 is a block diagram illustrating a configuration example of the interpolation unit 22. As illustrated in FIG. 7, the interpolation unit 22 includes a block cutout unit 221, a motion estimation unit 222, and a motion compensation unit 223. The interpolation unit 22 refers to an image (motion estimation source image, motion estimation destination image, and motion compensation source image) having the same frame number as that of the kth residual evaluation unit 14-k. It shall be. In addition, the interpolation unit 22-k acquires block position information and determination information for the residual evaluation unit 14-k.

  FIG. 8 is a diagram for explaining an outline of the operation of the interpolation unit 22. The motion estimation original image is an image frame that is a motion estimation target source in the motion estimation unit 222. The motion estimation destination image is an image frame that is a target of motion estimation in the motion estimation unit 222. The motion compensation original image is an image frame that is a target of motion compensation. An interpolated image is an image frame that is not transmitted from the transmission device 1 and is generated by interpolation by the reception device 2.

Although the frame number f _b of the frame number of the motion estimation original image f _a and the motion estimation target image must match one the frame number f _p of the motion estimation original image frame number f _a and the motion compensated original image may match, also it has a frame number f _p of the frame number f _b and the motion compensation based on image motion estimation target image may coincide. In FIG. 8, superscript ⁽¹⁾ such as I _R (f _a ⁽¹⁾ ) means processing by the interpolation unit 22-1, and superscript (2) is the interpolation unit 22-. 2 means processing.

The interpolation unit 22 extracts the motion estimation original image I _R (f _a ), the motion estimation destination image I _R (f _b ), and the motion compensation original image I _R (f _p ) from the transmission frame stored in the frame memory 21. Reference frame determining means for determining a set is provided. These sets of reference frames differ for each interpolation unit 22, and the frame number of the reference frame of the interpolation unit 22-k is the same as the frame number of the reference frame of the residual evaluation unit 14k. The interpolation unit 22 acquires the determination information from the transmission device 1 and performs motion compensation only for the block that is notified by the determination information that motion compensation is effective.

Block cutting unit 221, the motion estimation original image stored in the frame memory 21 I _{R (f a),} cut block area B _R in the position specified by the block position information transmitted from the transmitting apparatus 1, the motion It outputs to the estimation part 222. However, the block cutout unit 221 performs block cutout only for a block for which motion compensation is determined to be effective in the determination information transmitted from the transmission device 1. Horizontal W pixels, the image coordinates (i, j) of the block area _{B R} of a vertical H pixels _B pixel value in R (i, j) When the block cutout section 221, block area that is input from the transmitter 1 based on the upper left coordinates (s, t) B, for example, cutting out a block area _{B R} by the following equation (10).

Motion estimation unit 222, the motion estimation based on image extracted by the block cutout unit 221 _I R _{(f a)} of the block area _{B R} and the pixel value pattern similar corresponding region motion estimation target image _I R of _(f b) search from, and we obtain the corresponding positional relationship between the corresponding region of the motion estimation original image I _{R (f a)} of the block area B _R and the motion estimation target image I _{R (f b),} the corresponding position information to the motion compensation unit 223 Output. Corresponding position information includes, for example, (s, t, u _R , t), a set of the upper left coordinates (s, t) of the block area B input from the transmitter 1 and the upper left coordinates (u _R , v _R ) of the corresponding area. v _R ). Moreover, the corresponding position information is the upper-left coordinates (s, t) block area B and the motion vector _{(u R -s, v R -t} ) as _{set, (s, t, u R} -s, v R -t ). The upper left coordinates (u _R , v _R ) of the corresponding region are obtained by the following equation (11), for example.

  The equation (11) obtains the corresponding region by minimizing the absolute value error sum (SAD), but the minimization of the square error sum (SSD) and the normalized cross-correlation. The corresponding region may be obtained by maximizing (NCC: Normalized Cross Correlation).

  Further, the processes of the block cutout unit 221 and the motion estimation unit 222 may be combined, and the calculations of the equations (10) and (11) may be realized by a single calculation as shown in the following equation (12).

The motion compensation unit 223 is a partial image that is a block region of the motion compensation original image I _R (f _p ) corresponding to the motion estimation original image I _R (f _a ) based on the corresponding position information obtained by the motion estimation unit 222. D is generated by the following equation (13), for example.

Note that the image frame I _R is normally discretized into integer values with respect to image coordinates, whereas α _R and β _R are not necessarily integers. Thus, for example, a point in the image frame I _R spread function (e.g., Sinc function or Gaussian function of two-dimensional) may be defined to pixel values of the sub-pel position by convoluting the integer to alpha _R and beta _R (For example, rounding) may be applied.

In addition, the motion compensation unit 223 calculates the superimposition destination coordinates (γ _R , δ _R ) of the partial image D of the compensation source image I _R (f _p ) based on the corresponding position information obtained by the motion estimation unit 222, for example: It calculates by Formula (14).

In addition, it is preferable that the motion compensation unit 223 also calculates the weight W of the partial image D when performing the superimposition process. Weight W is a frame time and position f _a motion estimation original _image, the frame time and position f _b of the motion estimation target _image, the motion frame time position f _p of the compensating an original _image, and the time interval of the frame time and position f _q of the interpolated image It can be calculated according to For example, the weight W can be calculated by the following equation (15) under the assumption that the reliability of motion estimation or motion compensation decreases as the frame interval for motion estimation or the frame interval for motion compensation increases.

Further, as a method for changing the weight values W for each partial area D, motion estimation original image _I R to be calculated by the motion estimation unit 222 _{(f a)} of the block area _{B R} and the motion estimation target image _I R _(f b) Depending on the optimum value (minimum value or maximum value) of the degree of similarity with the corresponding region, the weight W may be increased as the degree of similarity is higher (the error is smaller or the correlation is higher).

The superimposing unit 23 generates the interpolated image J (f _q ) by superimposing the partial image D generated by the interpolating unit 22 on the superimposition destination coordinates (γ _R , δ _R ) using the weight W. For the n-th (n is a natural number equal to or less than N, N is a natural number) block region _BRn , the k-th (k is a natural number equal to or less than K, K is a natural number) interpolation unit 22 performs Expressions (13) to (15). Assuming that the obtained partial image is D _n ^(k) , the weight is W _n ^(k) , and the superimposition destination coordinates are (γ _Rn ^(k) , δ _Rn ^(k) ), the superimposing unit 23 calculates the following equation (16 ) To generate an interpolated image J (f _q ).

Although the partial image D is superimposed on the superimposition destination coordinates (γ _Rn ^(k) , δ _Rn ^(k) ) for various k and n, superposition does not occur depending on the set of superimposition destination coordinates. The denominator of equation (16) becomes zero. Therefore, assuming such a case, the interpolated image M (x, y) generated by a known method is used for the image coordinates (x, y) where the denominator is zero as in the following equation (17). It is good also as a pixel value. For example, M (x, y) may be a pixel value of any frame stored in the frame memory 21, or a value derived from any of a plurality of frame pixel values (for example, an average value or a weighted value) Average value).

The time multiplexing unit 24 time-multiplexes the video frame transmitted from the transmission device 1 and stored in the frame memory 21 and the interpolated image J (f _q ) generated by the superimposing unit 23, thereby outputting an output video. Configure. For example, in the case where only even frames of the input video are transmitted from the transmission apparatus 1 and the odd frames are generated by interpolation by the superimposing unit 23, these frames are alternately time-multiplexed in the frame number order in the time multiplexing unit 24, Outputs video with the same frame rate as the input video.

As described above, the reception device 2 uses the interpolation unit 22 to estimate the motion estimation source image I _R (f _a ), the motion estimation destination image I _R (f _b ), and the motion compensation from the frame transmitted from the transmission device 1. A block of the motion estimation original image I _R (f _a ) is determined for a block in which a plurality of sets of the original image I _R (f _p ) are determined and motion compensation is notified by the determination information acquired from the transmission device 1. A corresponding region having a similar pixel value pattern to the region is determined from the motion estimation destination image I _R (f _b ), and the motion compensation source image I _R (f _p ) is based on the corresponding positional relationship between the block region and the corresponding region. Find the superimposition coordinates of. Then, the superimposing unit 23 superimposes a plurality of sets of motion compensation original images on the superimposing destination coordinates to generate an interpolated image J (f _q ), and the time multiplexing unit 24 interpolates the frame transmitted from the transmission device 1 and the interpolation. The image J (f _q ) is time-multiplexed. For this reason, according to the receiving device 2, it is possible to interpolate a frame that is not transmitted from the transmitting device 1 with high accuracy. As a result, transmission with better rate distortion characteristics (RD characteristics) is possible than when all frames are transmitted.

Here, the superimposing unit 23 performs the motion estimation original image I _R (f _a ), the motion estimation destination image I _R (f _b ), the motion compensation original image I _R (f _p ), and the interpolated image J (f _q ). The motion compensated image is weighted and superimposed according to the time position and the similarity between the block region of the motion estimation original image I _R (f _a ) and the corresponding region of the motion estimation destination image I _R (f _b ). Is preferred. As a result, the level of reliability according to the time interval between the image used for calculation of the interpolated image and the interpolated image and the similarity of motion estimation can be reflected by weighting. It becomes possible.

  In addition, a computer can be suitably used to cause the above-described function to function as the receiving device 2, and such a computer stores a program describing processing contents for realizing each function of the receiving device 2 in a storage unit of the computer. This can be realized by storing the program and executing it by the CPU of the computer.

  Although the above embodiments have been described as representative examples, it will be apparent to those skilled in the art that many changes and substitutions can be made within the spirit and scope of the invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and changes can be made without departing from the scope of the claims. For example, a plurality of constituent blocks described in the embodiments can be combined into one or divided.

  As described above, according to the present invention, it is possible to thin out some frames of the input video on the transmitting device side and to interpolate the thinned frames on the receiving device side with high accuracy. Useful for applications.

DESCRIPTION OF SYMBOLS 1 Transmission apparatus 2 Reception apparatus 10 Time division part 11 Transmission frame memory 12 Non-transmission frame memory 13 Scan part 14 Residual evaluation part 15 Judgment part 16 Transmission simulation part 21 Frame memory 22 Interpolation part 23 Superposition part 24 Time multiplexing part 141 Block cutting Output unit 142 Motion estimation unit 143 Motion compensation residual evaluation unit 221 Block cutout unit 222 Motion estimation unit 223 Motion compensation unit

Claims (7)

A transmission device that thins out and transmits a part of a frame of an input video,
A time division unit for determining a transmission frame that is a frame to be transmitted among input video and a thinned image that is a frame that is not transmitted;
Reference frame determining means for determining a plurality of sets of a motion estimation original image, a motion estimation destination image, and a motion compensation original image from the transmission frame;
For each of the plurality of sets determined by the reference frame determining means, a corresponding region having a pixel value pattern similar to the block region of the motion estimation original image is determined from the motion estimation destination image, and the block of the motion estimation original image A motion estimation unit for obtaining a corresponding positional relationship between a region and a corresponding region of the motion estimation destination image;
For each of the plurality of sets determined by the reference frame determination unit, an image obtained by performing motion compensation on the motion compensation source image to the frame time position of the thinned image based on the corresponding positional relationship, and a residual between the thinned image A motion compensation residual evaluation unit for calculating for each block region;
A determination unit that generates determination information indicating the effectiveness of motion compensation for each block region from the value of the residual and transmits the determination information to the reception side ;
A transmission device comprising: Simulating deterioration due to encoding when encoding and transmitting a frame, further comprising a transmission simulation unit that generates a frame deteriorated by encoding,
The reference frame determination unit determines a plurality of sets of a motion estimation source image, a motion estimation destination image, and a motion compensation source image from the frame generated by the transmission simulation unit. Transmitter. A receiving device for interpolating and generating a thinned image thinned out by the transmitting device according to claim 1,
Reference frame determining means for determining a plurality of sets of a motion estimation original image, a motion estimation destination image, and a motion compensation original image from a frame transmitted from the transmission device;
For the block for which the determination information is acquired from the transmission apparatus and motion compensation is notified by the determination information, a corresponding region having a pixel value pattern similar to the block region of the motion estimation source image is determined as the motion estimation destination. A motion estimation unit that determines a corresponding positional relationship between a block region of the motion estimation original image and a corresponding region of the motion estimation destination image, determined from an image;
A motion compensation unit for obtaining a superimposition destination coordinate of the motion compensation original image based on the corresponding positional relationship;
A superimposing unit that generates an interpolated image by superimposing the plurality of sets of motion compensation original images on the superimposition destination coordinates, respectively;
A time multiplexing unit that time-multiplexes the frame transmitted from the transmission device and the interpolation image generated by the superimposition unit;
A receiving apparatus comprising:   The superimposing unit weights and superimposes the motion compensation image on the superimposition destination coordinates according to the time position of the motion estimation original image, the motion estimation target image, the motion compensation original image, and the interpolation image, and performs interpolation. The receiving apparatus according to claim 3, wherein an image is generated.   The superimposing unit weights the motion compensation image for each block region according to the similarity between the block region of the motion estimation original image and the corresponding region of the motion estimation destination image, and superimposes the motion compensation image on the superimposition destination coordinates. The receiving apparatus according to claim 3, wherein an image is generated.   A program for causing a computer to function as the transmission device according to claim 1.   The program for functioning a computer as a receiver as described in any one of Claims 3-5.

Images