JP4181190B2 - Motion vector detection method and apparatus, interpolation image generation method and apparatus, and image display system - Google Patents

Description

  The present invention relates to a motion vector detection method, an interpolation image creation method by motion compensation using a motion vector, and an image display system.

  In general, the image display device includes an impulse type display device that continues to emit light during the afterglow time of the phosphor after image writing, and a hold type display device that continues to hold the display of the previous frame until a new image writing is performed. Broadly divided. Examples of the impulse display device include a CRT display and a field emission display device (hereinafter referred to as FED). Examples of the hold-type display device include a liquid crystal display device (hereinafter referred to as LCD) and an electroluminescence display (hereinafter referred to as ELD).

  The problem with the hold-type display device is a blurring phenomenon that occurs in moving image display. The occurrence of the blur phenomenon is illustrated in FIG. 25B when a moving object (indicated by an ellipse) exists in an image extending over a plurality of frames shown in FIG. 25A and the observer's eyes follow the movement of the moving object. As shown, images of a plurality of frames are overlapped and projected on the retina. Until the period when the display image switches from the previous frame to the next frame, the eye predicts the display of the image of the next frame even though the image of the same previous frame continues to be displayed. Is observed while moving in the moving direction of the moving object. In other words, the following movement of the eye is continuous, and sampling is performed finer than the frame interval. As a result, it is observed as blurring by visually recognizing an image between two adjacent frames.

  In order to solve these problems, the display frame interval may be shortened. As a specific method, it is conceivable to create an interpolated image using motion compensation used in MPEG (Motion Picture Experts Group) 1 or MPEG2 and perform interpolation between adjacent frames. In motion compensation, a motion vector detected by block matching is used.

  MPEG1 and MPEG2 are basically based on the premise that the compression rate is increased, and it does not matter much whether the motion vector accurately reproduces the actual motion. This is because an error of the motion vector appears as an error of a prediction signal created by motion compensation, and an error signal between the prediction signal and the input image signal is encoded by DCT. However, in an image display system such as a hold-type display device, there is a problem that if an interpolation image is created by motion compensation using such an inaccurate motion vector, the display image is deteriorated.

On the other hand, in MPEG4, since a motion vector is required to reproduce an actual motion, several highly accurate detection methods are considered. Koji et al. (Japan Broadcasting Corporation) have proposed a method of using an rhomboid data group formed by a plurality of frames and blocks in Japanese Patent Laid-Open No. 10-304369 (Patent Document 1). This method utilizes the point that a normal image moves at a constant speed (including a stationary state) in a plurality of frames, and is expected to improve motion vector detection accuracy.
JP-A-10-304369

  The motion vector detection technique described in Japanese Patent Laid-Open No. 10-304369 does not have means for verifying whether the detected motion vector is actually correct, so an interpolation image is created by motion compensation using an incorrect motion vector. As a result, there is still a possibility that the display image is greatly deteriorated. In addition, in order to increase the detection accuracy of motion vectors, processing over a large number of frames is necessary, and the processing time is long.

  Accordingly, an object of the present invention is to provide a motion vector detection method and apparatus capable of detecting a more accurate motion vector by eliminating a motion vector that is greatly different from the actual image motion with a small number of frames.

  Another object of the present invention is to create an interpolated image for interpolating an image by motion compensation using the detected accurate motion vector, which is more realistic for moving images and image quality for still images. An object of the present invention is to provide an interpolation image generation method and apparatus and an image display system capable of obtaining a high-quality image with little deterioration.

  In order to solve the above problems, the present invention provides motion vector detection for obtaining a motion vector between the mth frame (m is an arbitrary integer) and the m + nth frame (n is an integer of i + 1 or more, i is an integer of 1 or more). In the method, a plurality of motion vector candidates between the mth frame and the m + n frame are obtained, and the motion vector candidates are scaled according to a frame difference between the m + i frame and the m + n frame or between the m + i frame and the mth frame, An image on the m + i frame is virtually created according to the motion vector candidate after the scale conversion, and an optimal motion vector that maximizes the correlation between the virtually created image and the actual image on the m + i frame is the motion vector candidate. Basically select from.

  In a first aspect of the present invention, a motion vector detection method for obtaining a motion vector between an m-th frame (m is an arbitrary integer) and an m + n frame (n is an integer of i + 1 or more, i is an integer of 1 or more). In (a), the first block is extracted from the motion vector search region of the m-th frame, and (b) a plurality of second blocks having a larger correlation with the first block are extracted from the motion vector search region of the m + n frame. (C) detecting a plurality of first motion vectors between the first block and a plurality of second blocks; (d) extracting a third block spatially co-located with the first block from the m + i frame; e) calculating a plurality of second motion vectors which are (n−i) / n of the first motion vector, and (f) a fourth block which is the destination of the third block from the m + n frame according to the second motion vector. Extracted selects an optimum motion vector to maximize the correlation between the third block and the fourth block from the (g) a plurality of first motion vector.

  In the second viewpoint of the present invention, after processing similar to (a), (b) and (c) in the first viewpoint, (h) a plurality of third motion vectors which are i / n of the first motion vector (I) extract a fifth block that is the destination of the first block from the m + i frame according to the third motion vector, and (j) calculate the first block and the fifth block from the plurality of first motion vectors. The optimal motion vector that maximizes the correlation is selected.

  In the third viewpoint of the present invention, in consideration of the effects of both the first and second viewpoints, a plurality of first motion vectors having a high correlation among the first block 11, the second block 12, and the fifth block 15 are obtained. Then, the motion vector that maximizes the correlation between the third block 13 and the fourth block 14 is selected as the optimum motion vector.

  In the fourth aspect of the present invention, a more accurate motion vector is obtained by bidirectional motion vector detection including a method for obtaining a motion vector from the m + n frame to the m frame.

  In the fifth aspect of the present invention, a block pair having a high correlation between the mth frame and the m + n frame is extracted using the m + i frame as a fulcrum, and the correlation between the block pair and the block at the fulcrum position of the m + i frame is obtained. The maximum motion vector is set as the optimal motion vector.

  According to a sixth aspect of the present invention, in order to create an interpolation image at a temporal position without a frame image using the detected optimal motion vector, the mth frame (m is an arbitrary integer) and m + n of the original image. In the interpolated image creating method for creating an interpolated image to be interpolated at a temporal position of an m + k (k is an arbitrary real number) frame between frames (n is an integer of i + 1 or more, i is an integer of 1 or more), The optimal motion vector obtained by the motion vector method of any of the fifth viewpoints is scaled according to the temporal position of the m + k frame, and the moving destination of the first block from the m + i frame according to the scaled motion vector The twelfth block is extracted, and the twelfth block is assigned to the same position as the first block of the m + k frame to create an interpolation image.

  In the seventh aspect of the present invention, particularly for an interlaced image, an image to be extracted by thinning out is an even-numbered or odd-numbered field image, and a motion vector is obtained before line interpolation to obtain a first motion before extra processing is performed. Find a vector. That is, the temporal position of the m + k field (k is an arbitrary real number) field between the m-th field (m is an arbitrary integer) and the m + n field (n is an integer of i + 1 or more, i is an integer of 1 or more) of the original image. An interpolation image generation method for generating an interpolation image to be interpolated in the first step is performed by extracting a first block from a motion vector search region in an m-th field (m is an arbitrary integer) and m + 2n (n is an integer of 1 or more). A plurality of second blocks having a higher correlation with the first block are extracted from the motion vector search region of the field, a plurality of first motion vectors between the first block and the plurality of second blocks are detected, and m + 2n−1 A third block at a spatial position obtained by spatially shifting the first block in the vertical direction by 1/2 line from the field is extracted, and a plurality of second motion vectors that are 1/2 of the first motion vector are extracted. And the fourth block, which is the movement destination of the block in the same position as the first block in the m + 2n−1 field, is extracted from the m + 2n field according to the second motion vector, and the third block is defined as the first block. A thirteenth block is generated by linear interpolation in the vertical direction, a fourteenth block is generated by linear interpolation in the second vertical direction opposite to the first vertical direction, and the thirteenth block is generated from the plurality of first motion vectors. And an optimal motion vector that maximizes the correlation between the fourteenth block and the fourteenth block is selected. Then, the optimal motion vector is scale-converted according to the temporal position, the fifteenth block as the movement destination of the first block is extracted from the m + 2n field according to the motion vector after the scale conversion, and the first block of the m + k field An interpolation image is created by assigning the fifteenth block to the same spatial position.

  Furthermore, according to an eighth aspect of the present invention, there is provided an image display system for displaying an interpolated image and an original image created by the above-described interpolated image creating method.

  According to the present invention, it is possible to eliminate an erroneous motion vector different from the actual motion of an image and perform accurate motion vector detection. Also, by creating an interpolated image using the detected accurate motion vector, an output image signal with a high frame frequency can be created from an input image signal with a low frame frequency, providing a more realistic image in the video, As for a still image, an image with little image quality deterioration can be provided.

  Embodiments of the present invention will be described below with reference to the drawings.

(First embodiment)
As shown in FIG. 1, for example, in order to create an interpolated image between the mth frame 1 and the m + n (n is an integer of i + 1 or more, i is an integer of 1 or more) frame 2 of the original image, Consider detecting an optimal motion vector between frame 1 and m + n frame 2. In the first embodiment of the present invention, the optimal motion vector is detected by the procedure shown in FIG. Hereinafter, the processing procedure of this embodiment will be described with reference to FIGS. 1 and 2.

First, the first block 11 is extracted from the motion vector search region of the m-th frame 1 (step S101).
Next, a plurality of second blocks 12 having the same block size as the first block 11 in the m-th frame 1 and a larger correlation with the first block 11 are extracted from the motion vector search region of the m + n frame 2. (Step S102).
As a method of checking the magnitude of the correlation between blocks in step S102, for example, when the absolute value difference sum is calculated by calculating the absolute value difference of the data for each pixel in the block and the absolute value difference sum is small. A known method for determining that the correlation is small when the correlation is large and the sum of the absolute value differences is large can be used.

Next, a plurality of first motion vectors D between the first block 11 and the plurality of second blocks 12 are detected as motion vector candidates (step S103).
Next, the third block 13 in the same spatial position as the first block 11 in the m-th frame 1 is extracted from the m + i frame 3 (step S104).
Next, from the ratio of the frame difference (n−i) between the m + i frame 3 and the m + n frame 2 and the frame difference i between the mth frame 1 and the m + i frame 3, the first motion vector D is calculated as the m + i. A plurality of second motion vectors ((n−i) / n) * D converted to a motion vector on the m + n frame 2 of the third block 13 in the frame 3 are calculated (step S105). The frame difference is a difference between frame numbers of two frames.

  Next, in accordance with the second motion vector ((n−i) / n) * D, the fourth block 14 which is the movement destination of the third block 13 to the m + n frame 2 is extracted (step S106). The fourth block 14 is a part of an image virtually created on the m + i frame according to the second motion vector ((n−i) / n) * D.

  Here, if a certain first motion vector D reproduces a true motion, the same block as the first block 11 should exist in the (m + i) th frame 3, and the position thereof is a dotted line block in FIG. It becomes the position. Therefore, the relative positional relationship between the dotted line block and the third block 13 and the relative positional relationship between the second block 12 and the fourth block 14 are equal. Accordingly, in the present embodiment, the suitability of each of the plurality of first motion vectors D is determined using the third block 13 and the fourth block 14.

  Specifically, by repeating the process of examining the correlation between the third block 13 and the fourth block 14, a motion vector that maximizes the correlation between the third block 13 and the fourth block 14 is obtained. That is, the optimal motion vector that maximizes the correlation between the third block 13 and the fourth block 14 is selected from the plurality of first motion vectors D (step S107). The correlation between the third block 13 and the fourth block 14 is obtained by calculating the sum of absolute value differences of the pixels of both blocks. When the sum of absolute value differences is small, the correlation is large and when the sum of absolute value differences is large Is judged to have a low correlation.

In the motion vector detection method of the present embodiment described above, one first motion vector D that minimizes the absolute value difference sum E1 calculated by the following equation (1) is obtained as the optimum motion vector.

  Here, X is a block position vector, f (A, b) is a pixel value corresponding to each block position (A) and frame (b), and α and β are weighting factors, respectively. The definition of each parameter is also applied to each embodiment described later. The first term on the right side of Equation (1) is the sum of absolute value differences between the first block 11 and the second block 12, and the second term is the sum of absolute value differences between the third block 13 and the fourth block 14. The weighting factors α and β are used when weighting the absolute value difference of each block. Basically, the weighting coefficient α and β are important because the absolute value difference sum (correlation) between the third block 13 and the fourth block 14 is emphasized. It is preferable that ≧ α.

Next, the effect by this embodiment is demonstrated using FIG.3 and FIG.4.
For simplicity, i = 1 and n = 2. Assume that two motion vectors A and B are detected as the first motion vector D when a still image “TEST” is displayed as an image from the m-th frame 1 to the (m + 2) -th frame 2 as shown in FIG. The motion vector A is a motion vector between the first block 11 on the mth frame 1 and the second block 12A on the m + 2 frame 2. The motion vector B is a motion vector between the first block 11 on the mth frame 1 and the second block 12B different from the second block 12A on the m + 2 frame 2.

  When the image 14A to be displayed at the position of the third block 13 in the (m + 1) th frame 3 according to the motion vector between the mth frame 1 and the (m + 1) th frame 3 obtained by scaling the motion vector A, the “ES” Become part. On the other hand, when the image 14B to be displayed at the position of the third block 13 of the (m + 1) th frame 2 is created according to the motion vector between the mth frame 1 and the (m + 1) th frame obtained by scaling the motion vector B, "T" Become.

  In this embodiment, the images 14A and 14B thus created from the motion vectors A and B are compared with the actual image of the (m + 1) th frame 3. That is, the third block 13 is compared with each of the fourth blocks 14A and 14B. By this comparison, the motion vector B is selected as the optimum motion vector.

  Here, the reason why the motion vector A is erroneously detected is that the noise is not limited to the case where the same character (“T” in this example) is present in one still image as in the example of “TEST”. Will greatly affect. For example, as shown in FIG. 4, when “T” to be actually detected in the (m + 2) th frame 2 has noise and “T” indicated by the motion vector A has little noise, the motion vector A is detected. End up. On the other hand, according to the present embodiment, by actually checking the correlation between the third block 13 of the (m + 1) th frame 3 and the respective fourth blocks 14A and 14B, the probability that the motion vector A is erroneously detected is greatly reduced. . This effect was also confirmed from experiments conducted by the inventors.

  Next, FIG. 5 shows a configuration of a motion vector detection apparatus that performs the motion vector detection method according to the present embodiment. Here, in order to simplify the description, i = 1 and n = 2. The input image signal 31 is sequentially input to the m + 2 frame memory 32, the m + 1 frame memory 33, and the m frame memory 34, and the m + 2 frame, m + 1 frame, and m frame image signals from these memories 32, 33, and 34. 35, 36 and 37 are read out.

  The motion vector detection unit 40 receives the image signals 35 and 37 of the (m + 2) th frame and the mth frame, and performs the processing of steps S101 to S103 in FIG. 2, that is, the extraction of the first block from the image signal 37 of the mth frame. Then, the second block is extracted from the image signal 35 of the (m + 2) th frame, and a plurality of first motion vectors 41 between the first block and the second block are detected. The motion vector determination block extraction unit 43 extracts a block used for determination of a motion vector from the image signals 35 and 36 of the (m + 2) th frame and the (m + 1) th frame. Position designation of the first block 11 is performed by the first block position designation unit 38, and the address 39 of the first block position from the first block position designation unit 38, for example, the address of the upper left corner of the block position is the motion vector detection unit 40 and The result is output to the motion vector determination block extraction unit 43.

  The motion vector scale conversion unit 42 calculates a plurality of second motion vectors 44 (D / 2), which is the process of step S105 in FIG. 2, that is, a half motion of the first motion vector 41 (D). The motion vector determination block extraction unit 43 performs the processing of steps S104 and S106 in FIG. 2, that is, extraction of the third block 13 from the image signal 36 of the (m + 1) th frame, and the (m + 2) th frame according to the second motion vector 44. The fourth block 14 is extracted from the image signal 35, and the image signals 45 of the third and fourth blocks are output.

  The image signals 45 of the third block and the fourth block from the motion vector determination block extraction unit 43 are input to the optimum motion vector determination unit 46, and among the plurality of first motion vectors 41, the third block, the fourth block, The process of determining the motion vector that maximizes the correlation of the above as the optimal motion vector is performed. In accordance with the determination result 47 from the optimal motion vector determination unit 46, the motion vector selection unit 48 selects one motion vector from the plurality of first motion vectors 41 as the final optimal motion vector 49, and in step S107 in FIG. Processing is performed.

  As described above, in this embodiment, the motion vector can be accurately obtained from the three frames of the mth, m + i, and m + n frames. That is, the correlation between the third block extracted from the m + i frame and the fourth block extracted from the m + n frame using the second motion vector obtained by scaling the plurality of first motion vectors which are motion vector candidates is obtained. By verifying the suitability of each first motion vector, only one motion vector that maximizes this correlation is detected as the optimal motion vector, and other motion vectors are excluded, thereby enabling more accurate motion vector detection. Become.

(Second Embodiment)
Next, another procedure for detecting the optimal motion vector between the mth frame 1 and the m + n frame 2 as in the first embodiment as the second embodiment of the present invention will be described with reference to FIGS. 6 and 7. explain.

  In the present embodiment, the first block 11 is extracted from the motion vector search region of the mth frame 1, and the same block size as the first block 11 in the mth frame 1 is extracted from the motion vector search region of the m + n frame 2. In addition, a plurality of second blocks 12 having a larger correlation with the first block 11 are extracted, and a plurality of first motion vectors D between the first block 11 and the plurality of second blocks 12 are selected as optimum motion vector candidates. Processing until detection (steps S201 to S203) is the same as that in the first embodiment.

  In the present embodiment, the first motion vector D is then calculated from the ratio of the frame difference n between the mth frame 1 and the m + n frame 2 and the frame difference i between the mth frame 1 and the m + i frame 3. A plurality of third motion vectors (i / n) * D that have been scale-converted into motion vectors on the m + i frame 3 of the first block 11 in the frame 1 are calculated (step S204). Next, in accordance with the third motion vector (i / n) * D, the fifth block 15 that is the movement destination of the first block 11 to the m + n frame 2 is extracted (step S205).

  If the first motion vector D reproduces the true motion, the same block as the first block 11 should exist in the m + i frame 3, and the position thereof is the position of the fifth block 15. Therefore, the fifth block 15 and the first block 11 are equal. Therefore, in the present embodiment, the first block 11 and the fifth block 15 are used to determine suitability for each of the first motion vectors D.

  Specifically, the correlation between the first block 11 and the fifth block 15 is examined. By repeating this process, a motion vector that maximizes the correlation between the first block 11 and the fifth block 15 is obtained. That is, the optimal motion vector that maximizes the correlation between the first block 11 and the fifth block 15 is selected from the plurality of first motion vectors D (step S206). The correlation between the first block 11 and the fifth block 15 is obtained by calculating the absolute value difference sum of the pixels of both blocks. When the absolute value difference sum is small, the correlation is large and when the absolute value difference sum is large. Is judged to have a low correlation.

In the motion vector detection method of the present embodiment described above, one first motion vector D that minimizes the absolute value difference sum E2 calculated by the equation (2) is obtained as the optimum motion vector.

  The first term on the right side of Equation (2) is the sum of absolute value differences between the first block 11 and the second block 12, and the second term is the sum of absolute value differences between the first block 11 and the fifth block 15. The weighting coefficients α and β are used when weighting the absolute value difference of each block. Basically, since the correlation between the first block 11 and the fifth block 15 is emphasized, it is preferable that β ≧ α. .

  FIG. 8 shows the configuration of a motion vector detection apparatus that implements the motion vector detection method according to the present embodiment. In order to simplify the explanation, i = 1 and n = 2. The same reference numerals are assigned to elements corresponding to those in FIG. 5 and the differences from the first embodiment will be mainly described. The motion vector scale conversion unit 42 performs the process of step S204 in FIG. A process of calculating the third motion vector 51 (D / 2), which is a half of (D), is performed. The motion vector determination block extraction unit 43 extracts a block used for determination of the first motion vector from the image signals 36 and 37 of the (m + 1) th frame and the mth frame. More specifically, the motion vector determination block extraction unit 43 performs the process of step S205 in FIG. 7, that is, extracts the fifth block 15 from the image signal 36 of the (m + 1) th frame according to the third motion vector 51.

  The image signal 52 of the fifth block from the motion vector determination block extraction unit 43 is input to the optimal motion vector determination unit 46, where the first block 11 and the fifth block 15 among the plurality of first motion vectors 41 are input. Processing for determining the motion vector that maximizes the correlation as the optimal motion vector is performed. In accordance with the determination result 47 from the optimal motion vector determination unit 46, the motion vector selection unit 48 selects one motion vector from the plurality of first motion vectors 41 as the final optimal motion vector 49, and in step S207 in FIG. Processing is performed. Also according to this embodiment, the same effect as the first embodiment can be obtained.

(Third embodiment)
Next, as a third embodiment of the present invention, a motion vector detection method capable of detecting a more accurate motion vector by combining the first and second embodiments described above will be described with reference to FIGS. . In this embodiment, first, the first to fifth blocks 11 to 15 are extracted by the procedure described in the first and second embodiments (steps S301 to S308).

  Next, using the first block 11, the second block 12, and the fifth block 15 from the plurality of first motion vectors D detected in step S303, the correlation between these blocks 11, 12, and 15 is further increased. A plurality of first motion vectors to be selected are selected (step S309). Further, the first motion vector D selected in step S309 is judged to be appropriate using the third block 13 and the fourth block 14, and the first motion vector D that maximizes the correlation between the third block 13 and the fourth block 14 is determined. One motion vector D is selected as the optimum motion vector (step S310).

Therefore, in the motion vector detection method of the present embodiment described above, one first motion vector D that minimizes the absolute value difference sum E3 calculated by the following equation (3) is obtained as the optimum motion vector.

  The first term on the right side of equation (3) is the sum of absolute value differences between the first block 11 and the second block 12, the second term is the sum of absolute value differences between the first block 11 and the fifth block 15, and the third term. The term is the sum of absolute difference between the third block 13 and the fourth block 14. The weighting coefficients α and β are used when weighting the absolute value difference of each block. Basically, since the correlation between the third block 13 and the fourth block 14 is emphasized, it is preferable that β ≧ α. .

  FIG. 11 shows a configuration of a motion vector detection apparatus that performs the motion vector detection method according to the present embodiment. In order to simplify the explanation, i = 1 and n = 2. The same reference numerals are given to the elements corresponding to those in FIGS. 5 and 8 and the differences from the first and second embodiments will be mainly described. The motion vector detection unit 40 performs steps S301 to S303 in FIG. Processing, that is, extraction of the first block from the image signal 37 of the mth frame, extraction of the second block from the image signal 35 of the m + 2 frame, and a plurality of first motion vectors between the first block and the second block 41 (D) is detected, and the image signals 53 of the first and second blocks are output to the optimum motion vector determination unit 46 at this time.

  The motion vector scale conversion unit 42 performs the processing of steps S305 and S307 in FIG. 10, that is, the second motion vector 44 and the first motion vector i / n which are (n−i) / n of the first motion vector. The third motion vector 51 is calculated. In this case, since i = 1 and n = 2, both the second and third motion vectors are motion vectors (D) that are ½ of the first motion vector (D). / 2) is calculated.

  The motion vector determination block extraction unit 43 extracts a block used for determination of a motion vector from the image signals 35, 36, and 37 of the (m + 2) th frame, the (m + 1) th frame, and the mth frame. More specifically, the motion vector determination block extraction unit 43 follows the processing of steps S306 and S308 in FIG. 10, that is, the extraction of the third block 13 from the image signal 36 of the (m + 1) th frame, and the second motion vector 44. The fourth block 14 is extracted from the m + 2 frame image signal 35, and the fifth block 15 is extracted from the m + 1 frame image signal 36 according to the third motion vector 51. A 5-block image signal 54 is output.

  The image signals 54 of the third, fourth, and fifth blocks from the motion vector determination block extraction unit 43 are input to the optimum motion vector determination unit 46, where the processing of step S309 in FIG. Among the motion vectors 41, a plurality of motion vectors having a larger correlation between the first block 11, the second block 12, and the fifth block 15 are selected, and the correlation between the third block 13 and the fourth block 14 is maximized. A process of determining one motion vector as an optimal motion vector is performed. According to the determination result 47 from the optimal motion vector determination unit 46, the motion vector selection unit 48 performs the optimal motion vector selection processing in step S310 in FIG. 10, and the motion vector selected here is the final optimal motion vector. 49, and the process of step S310 in FIG. 10 is performed.

  As described above, in this embodiment, by combining the first and second embodiments, more accurate motion vector detection is possible.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described with reference to FIGS. In the present embodiment, when an interpolation image is created at the position of the m + i frame 3 between the mth frame 1 and the m + n frame 2, the first motion vector D and the m + n from the mth frame 1 to the m + n frame 2 are used. Both of the fourth motion vectors E from the frame 2 to the m-th frame 1 are obtained. The detection of the first motion vector D is performed in the same manner as in the first embodiment (steps S401 to S406, S412).

  On the other hand, in the detection of the fourth motion vector E, the sixth block 16 in the same spatial position as the first block 11 is extracted from the m + n frame (step S407), and the motion vector search region in the m-th frame 1 is extracted. The seventh block 17 having a larger correlation with the sixth block 16 is extracted from a plurality of blocks having the same block size as the sixth block 16 (step S408), and the fourth motion between the sixth block 16 and the seventh block 16 is extracted. Vector E is detected (step S409). The correlation between the blocks is obtained by calculating the absolute value difference sum of the pixels of both blocks. When the absolute value difference sum is small, the correlation is large, and when the absolute value difference sum is large, the correlation is determined to be small. .

  Next, from the ratio of the frame difference i on the mth frame 1 and the m + i frame 3 and the frame difference n on the m + n frame 2 and the mth frame 1, the fourth motion vector E is determined as the m + i frame of the sixth block 16. A fifth motion vector (i / n) * E that has been scale-converted to a moving amount from 3 to the m-th frame 1 is calculated (step S410). According to the fifth motion vector (i / n) * E, the third block 13 in the m + i frame 3 that is in the same spatial position as the first block 11 in the m-th frame 1 is the destination to the m-th frame 1 The eighth block 18 is extracted (step S411).

  In the selection of the optimum first motion vector in the next step S412, the absolute value difference sum between the third block 13 and the fourth block 14 is obtained and the correlation is examined as in the first embodiment, and the third block is determined from the first motion vector. One motion vector that maximizes the correlation between 13 and the fourth block 14 is obtained. On the other hand, in the selection of the optimum fourth motion vector in the next step S413, one motion vector that maximizes the correlation between the third block 13 and the eighth block 18 is selected from the fourth motion vector.

  Next, the absolute value difference sum E1 of the third block 13 and the fourth block 14 is compared with the absolute value difference sum E4 of the third block 13 and the eighth block 18 (step S414), and the smaller absolute value difference sum is calculated. Adopt motion vectors. That is, if E1 is smaller than E4, the optimum first motion vector detected in step S412 is selected as the optimum motion vector (step S415), and if E4 is smaller than E1, the optimum fourth motion detected in step S413 is selected. The inverse vector -E of the motion vector is selected as the optimal motion vector (step S416).

The absolute value difference sum E4 to be calculated in order to obtain the fourth motion vector in this embodiment can be expressed by the following equation.

  Here, it is preferable to use the same weighting factors α and β as the values used to obtain E1.

  FIG. 15 shows the configuration of a motion vector detection apparatus that implements the motion vector detection method according to the present embodiment. In order to simplify the explanation, i = 1 and n = 2. The same reference numerals are assigned to elements corresponding to those in FIG. 5 and the differences from the first embodiment will be mainly described. In FIG. 15, a motion vector detection unit 60, a motion vector scale conversion unit 62, and a motion vector determination A block extraction unit 63 and an optimum motion vector determination unit 66 are provided. The motion vector detection unit 40, the motion vector scale conversion unit 42, the motion vector determination block extraction unit 43, and the optimum motion vector determination unit 46 are the same as those in the first embodiment, and thus, steps S401 to S406 in FIG. And the process of S412 is performed.

  In the newly provided motion vector detection unit 60, the processing in steps S407 to S409 in FIG. 13, that is, the processing for extracting the sixth block 16 in the motion vector search region from the m + 2 frame, On the other hand, processing for extracting the seventh block 17 having the same block size in the motion vector search region in the m-th frame 1 and processing for detecting the fourth motion vector 61 between the sixth block 16 and the seventh block 17 are performed. .

  The position designation of the sixth block 16 is performed by the first block 11 designation unit 38, and the address of the first block position, for example, the address 39 of the upper left corner is output to the motion vector detection unit 60 and the motion vector determination block extraction unit 63. The

  The motion vector scale conversion unit 62 calculates a plurality of fifth motion vectors 64 which are the process of step S410 in FIG. The motion vector determination block extraction unit 63 performs the process of step S411 in FIG. 14, that is, extracts the eighth block 18 from the image signal 37 of the m-th frame according to the fifth motion vector 64, and the eighth block 18 The image signal 65 is output.

  The third block image signal 45A from the motion vector determination block extraction unit 43 and the eighth block image signal 65 from the motion vector determination block extraction unit 63 are input to the optimum motion vector determination unit 66, and a plurality of fourth signal signals are input. The process of step S413 in FIG. 14 is performed in which the motion vector that maximizes the correlation between the third block 13 and the eighth block 18 among the motion vectors 61 is determined as the optimum fourth motion vector.

  According to the determination results 47 and 67 from the optimal motion vector determination units 46 and 66, the motion vector selection unit 68 performs the optimal motion vector selection processing in steps S415 and S416 in FIG. The final optimum motion vector 69 is output.

  According to the present embodiment, in the process of obtaining the optimal fourth motion vector from the m-th frame to the m + n frame described in the first embodiment, the m + n-th frame to the m-th frame whose temporal direction is opposite to this. More accurate motion vector detection can be performed by adding a process for obtaining the optimum fourth motion vector for the frame and finally selecting the most suitable one of these.

(Fifth embodiment)
Next, a fifth embodiment of the present invention will be described with reference to FIGS. In this embodiment, in order to obtain a motion vector between the mth frame 1 and the m + n frame 2, first, the ninth block 19 is extracted from the motion vector search target of the m + i frame 3 (step S501). Next, a plurality of tenth blocks 20 are extracted from the motion vector search region of the m-th frame 1 (step S502).

  Next, a sixth motion vector F between the ninth block 19 and the tenth block 20 is detected as a motion vector from the ninth block 19 to the m-th frame 1 (step S503). Furthermore, as a motion vector from the ninth block 19 to the m + n frame 2, the direction is opposite to that of the sixth motion vector F, and the magnitude is a frame difference (n−i) between the m + i frame 3 and the m + n frame 2. A seventh motion vector − ((n−i) / i) * F that satisfies the ratio of the frame difference i between the m-th frame 1 and the m + i frame 3 is calculated (step S504). In accordance with the seventh motion vector − ((n−i) / i) * F, the eleventh block 21 as the movement destination of the ninth block 19 is extracted from the m + n frame 2 (step S505).

  Next, a plurality of sixth motion vectors F having a larger correlation between the tenth block 20 and the eleventh block 21 are detected (step S506). If the sixth motion vector F reproduces the true motion, the same block as the tenth block 20 or the eleventh block 21 should exist in the (m + i) th frame 3, and its position is that of the ninth block 19. Position.

  Therefore, in this embodiment, the image of the ninth block 19 is extracted from the m + i frame 3, and the suitability of the sixth motion vector F is determined using the images of the ninth block 19, the tenth block 20, and the eleventh block 21. . Specifically, the correlation between the tenth block 20 and the eleventh block 21 is examined to obtain a plurality of sixth motion vectors having a larger correlation, and further, the correlation between the ninth block 19 and the tenth block 20 is examined to further increase the correlation. One large sixth motion vector is selected as the optimal motion vector.

The absolute value difference sum E5 to be calculated in the present embodiment can be expressed as the following equation.

In equation (5), the correlation between the ninth block 19 and the tenth block 20 is examined, but the correlation between the ninth block 19 and the eleventh block 21 may be examined. The absolute value difference sum E6 to be calculated in that case can be expressed as the following equation.

  Here, with respect to the weighting factors α and β in the equations (5) and (6), it is preferable that β ≧ α.

Further, the correlation can be examined from the ninth block 19, the tenth block 20, and the ninth block 19 and the eleventh block 21, and the absolute value difference sum E7 to be calculated in this case can be expressed as the following equation. .

  Here, with respect to the weighting factors α and β, importance is placed on blocks having high correlation in the time direction. That is, i and ni are compared, and the smaller block between frames is emphasized. For example, when i = 1 and n = 3, since n−i is 2, it is preferable to place importance on the correlation between the ninth block 19 and the tenth block 20 and to make α larger than β.

  In the present embodiment, the motion vector F that minimizes E5, E6, or E7 shown in equations (5), (6), and (7) is obtained as the optimum motion vector.

  FIG. 18 shows the configuration of a motion vector detection apparatus that implements the motion vector detection method according to this embodiment. In order to simplify the explanation, i = 1 and n = 2. The same reference numerals are assigned to elements corresponding to those in FIG. 5, and the differences from the first embodiment will be mainly described. In the motion vector determination block extracting unit 83, the processing in step S501 in FIG. Processing for extracting the ninth block 19 in the motion vector search area from the image signal 36 of the frame is performed. The position of the ninth block 19 is specified by the ninth block 11 specifying unit 78, and the address of the ninth block position, for example, the address 79 of the upper left corner is output to the motion vector determination block extracting unit 83.

  The motion vector detection unit 80 detects the sixth motion vector F, which is the motion vector from step m502 to step m506 in FIG. 10th block 20 and m + n frame on the mth frame 1 with respect to the 9th block 19 by using the sixth motion vector and the seventh motion vector to calculate the seventh motion vector that is ½ motion in the direction The process of extracting the eleventh block 21 on 2 and the process of outputting the plurality of sixth motion vectors 81 and the image signal 82 of the tenth block 20 having a larger correlation between the tenth block 20 and the eleventh block 21 are performed.

  Next, the optimal motion vector determination unit 86 performs a process of determining a motion vector having the maximum correlation between the ninth block 19 and the tenth block 20 among the plurality of sixth motion vectors 81 as the optimal motion vector. According to the determination result 87 from the optimal motion vector determination unit 86, the motion vector selection unit 88 performs the optimal motion vector selection processing in step S507 in FIG. 17, and the motion vector selected here is the final optimal motion vector. 89 is output.

  Although the configuration for determining the correlation between the 10th block 20 and the 9th block has been described here, the same configuration can be used for determining the correlation between the 11th block 21 and the 9th block 19 as described above.

(Sixth embodiment)
Next, an interpolation image generation method using motion compensation based on motion vector detection will be described as a sixth embodiment of the present invention with reference to FIGS. 19 and 20. In the interpolation image creation method according to the present embodiment, for example, the twelfth block 22 is extracted using a motion vector obtained by scaling the optimal motion vector D obtained in the first embodiment, and the twelfth block 22 is extracted from the interpolation frame. Basically, an interpolated image is created by assigning it to the same spatial position as one block.

  More specifically, the m + k (k is an arbitrary real number) frame between the m-th frame (m is an arbitrary integer) and the m + n frame (n is an integer equal to or greater than i + 1, i is an integer equal to or greater than 1). When an interpolation image is created at a position, for example, the position of the m + k frame between the mth frame 1 and the m + i frame 3 as shown in FIG. 19, first, the mth frame 1 and the m + n as in the first embodiment. After obtaining the optimal motion vector D between the frames 2, the optimal motion vector D is scaled according to the temporal position of the m + k frame (steps S601 to S602).

  That is, from the ratio of the frame difference n between the mth frame 1 and the m + n frame 2 and the frame difference (i−k) between the m + k frame and the m + i frame 3, it is spatially the same position as the first block 11. The movement amount of the block on the m + k frame from the m + k frame to the m + i frame 3, that is, the motion vector ((i−k) / n) * D is calculated.

  Next, according to the motion vector ((i−k) / n) * D after the scale conversion, the 12th block 22 that is the movement destination of the first block 11 from the (m + i) th frame 3, that is, the first block 11 spatially. The twelfth block 22 that is the destination of the movement of the block on the m + k frame at the same position onto the m + i frame 3 is extracted (step S603).

  Next, an twelfth block 22 is allocated at the same spatial position as the first block of the m + k frame to create an interpolation image of the m + k frame (step S604). Since each block on the m + k frame is composed of blocks that are spatially located from the mth frame 1, the blocks are spread over the m + k frame, and an interpolation block is assigned to each block. By creating, it is possible to create interpolation images without gaps and without overlapping.

  The interpolated image can be created in a frame between the mth frame 1 and the m + i frame 3 as described above, or can be created in a frame between the m + i frame 3 and the m + n frame 2 using the same motion vector D. Can do. For example, if i = 1 and n = 2, using the motion vector D detected from the m-th frame 1 and the m + 2 frame, the m + 0.5 frame (between the m-th frame 1 and the m + 1-th frame) image, The m + 1.5th frame (between the (m + 1) th frame and the (m + 2) th frame) image is obtained almost simultaneously. In this method, after obtaining a motion vector between the mth frame 1 and the m + 1th frame, the m + 0.5 frame is created using the motion vector, and another motion vector between the m + 1th frame and the m + 2 frame is obtained. Thereafter, the calculation time can be shortened as compared with the case where the m + 1.5th frame is created using the motion vector.

As another method, when there is a margin in calculation processing, the m + 0.5th frame image and the m + 1.5th frame image are obtained using the motion vector D ₁ detected from the mth frame 1 and the m + 2 frame. was followed, using another motion vector D ₂ detected from the m + 1 frame and the m + 3 frame, obtains a first m + 1.5 frame image and the m + 2.5 frame image, and further two of the m + 1.5 frame image A method may be adopted in which the correlation between the m + 2 frame image is examined and each block is selected so that the correlation with the m + 2 frame image is high.

(Seventh embodiment)
Next, as a seventh embodiment of the present invention, a method for detecting a motion vector and creating an interpolated image when the original image is an interlaced image will be described with reference to FIGS.

  First, the first block 11 is extracted from the motion vector search region of the m-th field 1 (step S701), and the same block as the first block 11 in the motion vector search region of the m + 2n (n is an integer of 1 or more) field. A plurality of second blocks 12 having a larger correlation with the first block 11 are extracted from a plurality of blocks of size (step S702), and a vector connecting the first block 11 and the second block 12 is detected as the first motion vector D. (Step S703).

  Next, the third block 13 is extracted from the (m + 2n−1) -th field at a spatial position that is spatially shifted in the vertical direction (up or down) by ½ line with the first block 11 of the m-th frame 1 (step S704). ). Next, a second motion vector D / 2 having the same direction as the first motion vector D and having a motion amount of ½ is detected (step S705), and the third block from the (m + 2n) field is detected according to the second motion vector D / 2. A fourth block 14 is extracted as a movement destination to the m + 2n field of the block spatially located at 13 (step S706).

  Next, a thirteenth block 23 is created by linearly interpolating the third block 13 in the first vertical direction (for example, upward direction) (step S707), and the fourth block 14 is changed to the second direction opposite to the first vertical direction. A fourteenth block 24 linearly interpolated in the vertical direction (for example, the downward direction) is created (step S708). Then, the suitability of the first motion vector 11 is determined using the thirteenth block 23 and the fourteenth block 24. That is, for example, the correlation between both blocks is checked by obtaining the absolute value difference sum of the thirteenth block 23 and the fourteenth block 24, and the first motion vector D that maximizes the correlation is selected as the optimum motion vector (step S709). .

  Next, an interpolation image of the (m + k) th field is basically created using the same method as in the sixth embodiment. That is, similar to steps S602 to S604 in FIG. 20 described in the sixth embodiment, the optimal motion vector obtained in step S709 is scale-converted according to the temporal position of the m + k field to obtain the third motion vector. In step S710, the fifteenth block, which is the destination of the first block 11, is extracted from the m + 2n field according to the third motion vector after the scale conversion (step S711), and the first block 11 in the m + k field is spatially extracted. The 15th block is assigned to the same position to create an m + k field interpolation image (step S712).

  For example, when an interpolation image of the (m + 1.5) -th field is created with k = 0.5, the (m + 2n) -th motion vector has the same direction as the optimal motion vector of the first motion vector and has a motion amount of ¼. The fifth block to be moved to the field is extracted from the m + 2n field, and the fifth block is assigned to the m + 1.5 field. Similarly, each fifth block on the (m + 1.5) th field is obtained and assigned to the (m + 1.5) th field, thereby completing the interpolation image. However, this method is basically a case where a progressive image is obtained as a field image, and when interlace processing is performed also for a field of an interpolated image, lines must be thinned out in the vertical direction.

(Eighth embodiment)
Finally, as an eighth embodiment of the present invention, an image display system using the motion vector detection method described in each of the previous embodiments and the method of creating an interpolated image based thereon will be described.

  FIG. 24 shows a schematic configuration of the image display system. An input image signal 101 is input to the interpolation field image creation unit 102 and the image switching unit 104. In the interpolation field image creation unit 102, the interpolation image signal 103 is created by the procedure described so far, and the interpolation image signal 103 is output to the image switching unit 104. The image switching unit 104 controls whether the input image signal 101 is output as it is or whether the interpolated image signal 104 is output. An output image signal 105 from the image switching unit 104 is output to a high-speed refresh display device 106 that is a hold-type display device. The display device 106 displays an image at a different refresh rate in response to the synchronization signal included in the output image signal 105.

The figure explaining the motion vector detection method which concerns on the 1st Embodiment of this invention. A flowchart showing a motion vector detection procedure according to the embodiment. The figure which shows the creation image output result and the effect of the same embodiment The figure which shows the example of the image from which an incorrect motion vector is detected The block diagram which shows the structure of the motion vector detection apparatus which concerns on the same embodiment The figure explaining the motion vector detection method which concerns on the 2nd Embodiment of this invention. A flowchart showing a motion vector detection procedure according to the embodiment. The block diagram which shows the structure of the motion vector detection apparatus which concerns on the same embodiment The figure explaining the motion vector detection method which concerns on the 3rd Embodiment of this invention. A flowchart showing a motion vector detection procedure according to the embodiment. The block diagram which shows the structure of the motion vector detection apparatus which concerns on the same embodiment The figure explaining the motion vector detection method which concerns on the 4th Embodiment of this invention. A flowchart showing a motion vector detection procedure according to the embodiment. A flowchart showing a motion vector detection procedure according to the embodiment. The block diagram which shows the structure of the motion vector detection apparatus which concerns on the same embodiment The figure explaining the motion vector detection method which concerns on the 5th Embodiment of this invention. A flowchart showing a motion vector detection procedure according to the embodiment. The block diagram which shows the structure of the motion vector detection apparatus which concerns on the same embodiment The figure explaining the interpolation image creation method which concerns on the 6th Embodiment of this invention. A flowchart showing an interpolation image creation procedure according to the embodiment The figure explaining the motion vector detection method which concerns on the 7th Embodiment of this invention. A flowchart showing a motion vector detection procedure according to the embodiment. A flowchart showing a motion vector detection procedure according to the embodiment. The block diagram which shows the structure of the display system which concerns on one Embodiment of this invention. The figure explaining the blurring phenomenon in the conventional hold type display device

Claims (5)

In a motion vector detection method for obtaining an optimal motion vector between an mth frame (m is an arbitrary integer) and an m + n frame (n is an integer of i + 1 or more, i is an integer of 1 or more),
A block extracting unit extracting a ninth block from the motion vector search region of the m + i frame;
A step of a motion vector detection unit extracting a plurality of tenth blocks from the motion vector search region of the m-th frame;
The motion vector detection unit, determining a plurality of sixth motion vector from the ninth block to said plurality of first 10 blocks,
The motion vector detecting unit calculating a plurality of seventh motion vectors obtained by performing (ni) / i scale conversion of the plurality of sixth motion vectors in a reverse direction ;
The motion vector detection unit extracting, from the m + n frame, a plurality of eleventh blocks to which the ninth block is moved when the plurality of seventh motion vectors are used ;
The motion vector detection unit calculates a correlation between the tenth block and the eleventh block that have a one-to-one correspondence with the plurality of sixth motion vectors, and selects the plurality of sixth motion vectors selected in descending order of the correlation. A step to choose ;
The motion vector determination unit, for each of the selected plurality of sixth motion vectors , the first absolute value difference sum of the pixel values of the ninth block and the tenth block, the ninth block and the eleventh block A second absolute value difference sum of pixel values is calculated, the first absolute value difference sum is multiplied by a first weight that decreases according to the magnitude of i, and the second absolute value difference sum (n -I) is multiplied by a second weight that decreases in accordance with the magnitude of -i), and a first absolute value difference sum multiplied by the first weight and a second absolute value difference sum multiplied by the second weight; the motion vector detection method of a sum calculated as the evaluation value, and a step of determining an optimum motion vector sixth motion vector by n / i scaling the reversed to minimize the evaluation value. Interpolate at the temporal position of the m + k (k is an arbitrary real number) frame between the mth frame (m is an arbitrary integer) and the m + n frame (n is an integer of i + 1 or more, i is an integer of 1 or more) of the original image. In an interpolation image creation method for creating an interpolation image to be performed,
Scaling the optimal motion vector obtained by the motion vector detection method according to claim 1 according to a temporal position of the m + k frame;
Extracting a twelfth block as a movement destination of an interpolation block located in the same position as the tenth block in the m + k frame from the m + i frame according to the scale-converted motion vector;
And interpolating the interpolated image by allocating the twelfth block to the interpolated block located in the same spatial position as the tenth block within the m + k frame. An interpolated image creating unit that creates the interpolated image by the interpolated image creating method according to claim 2;
An image display system comprising: a display device that displays the interpolated image and the original image. In a motion vector detection device for obtaining an optimal motion vector between an mth frame (m is an arbitrary integer) and an m + n frame (n is an integer of i + 1 or more, i is an integer of 1 or more),
Means for extracting the ninth block from the motion vector search region of the m + i frame;
Means for extracting a plurality of tenth blocks from the motion vector search region of the m-th frame;
Means for determining a plurality of sixth motion vector from the ninth block to said plurality of first 10 blocks,
Means for calculating said plurality of sixth motion vector in the opposite direction (n-i) / i scaled plurality of seventh motion vector,
Means for extracting, from the m + n frame , a plurality of eleventh blocks that are the movement destinations of the ninth block when using the plurality of seventh motion vectors;
Means for calculating a correlation between the tenth block and the eleventh block corresponding to each other with respect to the plurality of sixth motion vectors, and selecting the plurality of sixth motion vectors in descending order of the correlation ;
For each of the selected plurality of sixth motion vectors, the first absolute value difference sum of the pixel values of the ninth block and the tenth block and the second of the pixel values of the ninth block and the eleventh block . An absolute value difference sum is calculated, the first absolute value difference sum is multiplied by a first weight that decreases according to the magnitude of i, and the magnitude of (ni) is given to the second absolute value difference sum. And the sum of the first absolute value difference sum multiplied by the first weight and the second absolute value difference sum multiplied by the second weight is used as an evaluation value. A motion vector detection apparatus comprising: means for calculating and calculating an optimum motion vector by performing n / i scale conversion on the sixth motion vector that minimizes the evaluation value in the reverse direction . Interpolate at the temporal position of the m + k (k is an arbitrary real number) frame between the mth frame (m is an arbitrary integer) and the m + n frame (n is an integer of i + 1 or more, i is an integer of 1 or more) of the original image. In an interpolated image creating apparatus for creating an interpolated image to be
Means for scaling the optimal motion vector obtained by the motion vector detection device according to claim 4 according to a temporal position of the m + k frame;
Means for extracting a twelfth block as a movement destination of an interpolation block located in the same position as the tenth block in the m + k frame from the m + i frame according to the scale-converted motion vector;
Means for creating the interpolated image by allocating the twelfth block to the interpolated block located in the same spatial position as the tenth block within the m + k frame.

Images