JP5005629B2 - Motion vector correction device, motion vector correction method, image interpolation device, television receiver, video reproduction device, control program, and computer-readable recording medium - Google Patents

Description

  The present invention relates to a motion vector correction apparatus and a motion vector correction method for determining a motion vector between original pictures, and in particular, an image interpolation apparatus, a television receiver, a video reproduction apparatus, a control program, and a computer-readable recording. It relates to the medium.

  When changing the number of frames of the original image material, such as when copying a 24 Hz film material to a 60 Hz or 120 Hz display device, or when copying a 60 Hz progressive signal generated by IP conversion or the like to a 120 Hz display device A technique is generally known in which a motion vector is obtained from a plurality of consecutive frames, and the number of frames of an image signal is converted by an interpolated image generated based on the motion vector.

  FIG. 9 is a diagram showing a relationship between an input 60 Hz original image and a 120 Hz output image. For example, when a 60 Hz progressive signal is copied to a 120 Hz display device, the original image 9-1 at time t-1 and the original image 9-2 at time t are used, and the time is exactly half (intermediate). The frame rate is increased by creating and outputting the interpolated image 9-3.

  Here, a general process for generating one interpolated image from two original images will be described. First, an original image serving as a reference for obtaining a motion vector is determined from the two images. Assuming that an image at time t in FIG. 9 is f (t), either f (t-1) or f (t) is generated when an interpolated image is generated from f (t-1) and f (t). Is the reference original image. Hereinafter, a case where f (t−1) is a reference image will be described.

  FIG. 10 is a diagram illustrating an example in which a reference image is divided into blocks in order to calculate a vector. As shown in FIG. 10, the reference image f (t−1) is divided into a plurality of blocks for calculating a motion vector. Like the portion 10-1 in FIG. 10, a portion surrounded by a thin line means a pixel of the original image, and in this example, a minimum block 10-2 is defined for each region of 8 pixels × 8 pixels.

  Next, a motion vector between f (t−1) and f (t) is obtained for each defined minimum block. As a method for obtaining the motion vector, a general method such as block matching or a gradient method is used.

  In this way, in the reference original image, a motion vector that seems to have the highest reliability is detected for each minimum block. As a process that is normally performed, filtering such as a median filter is performed on the motion vector determined in this way to prevent erroneous detection.

  Next, vector allocation processing is performed on each coordinate of the interpolated image using the motion vector obtained as described above. The unit of the assigned coordinates is arbitrary, but usually an area smaller than the block unit of the reference image is designated. In the example shown in FIG. 10, since the reference original image is defined by 8 pixels × 8 pixels, a vector is assigned to the interpolated coordinates for each small region such as 4 pixels × 2 pixels.

  The assignment of vectors to interpolation coordinates will be described with reference to FIG. 11 as one-dimensional processing in the horizontal direction for the sake of simplicity. FIG. 11 is a diagram illustrating an example of assigning vectors to interpolation coordinates. If the motion vector Vx indicated by the arrow 11-1 is obtained for the coordinate x of the reference original image f (t-1), the interpolated image is created in the middle of time t-1 and time t. , Vx, the interpolated coordinate N (x) is N (x) = x + Vx / 2.

  At this time, the vector Vx is assigned to the small block of the interpolation coordinates including N (x). By executing this processing for all the blocks of the reference original image, vectors are allocated on the small blocks of the interpolation coordinates. As a result, for each small block of interpolated coordinates, there are places where multiple vectors are assigned or not assigned at all, but usually when multiple assignments are made, of the assigned vectors, This is represented by the highest correlation between f (t-1) and f (t) calculated using the specific region. If no assignment is made, a zero vector is used instead. As a well-known process, the assigned vector is subjected to a filtering process such as a median filter to obtain a final vector.

  Finally, using the finally obtained allocation vector on the interpolated coordinates, calculate the coordinates of f (t-1) or f (t) and generate an interpolated image using one or both images To do.

  The above is a process for creating a general interpolated image, but a process that introduces a bidirectional vector has been proposed in order to further improve the reliability of the motion vector and reduce artifacts in the interpolated image. In FIG. 11, the bidirectional vector refers to the forward motion vector with reference to the image f (t) with f (t−1) as the reference image in FIG. 11, and f (t−1) with f (t) as the reference image. This refers to a vector in both directions with a backward motion vector referring to an image.

  By the way, when an interpolated image is generated and an image is interpolated, “disappearing area” and “appearing area” become a problem. For example, when an object moves between the original image f (t-1) and the original image f (t), the object is not hidden by f (t-1) due to the movement of the object, but f (t) Then, the background area hidden behind the object is called “disappearing area”, and the background area hidden behind the object at f (t−1) but not hidden at f (t) is called “appearing area”. Call. Therefore, “disappearing area” information (background area image data) exists only in f (t−1), and “appearing area” information (background area image data) exists only in f (t). Here, in the gradient method and block matching described above, there is no solution that there is no motion vector, and the solution in the calculation formula (gradient method) or the region with the smallest pixel correlation value (ie, the region with high pixel correlation) is used. It is a technique that leads as a solution.

  Therefore, when a motion vector is obtained from f (t−1) to f (t), the “disappearing region” has no corresponding point, and the motion vector is indicated toward any similar point in the vicinity, and f (t) When the motion vector is obtained from f (t−1), the “appearing region” has no corresponding point, and the vector is shown toward any similar point in the vicinity.

  When creating an interpolated image based on the motion vector thus determined, if an interpolated image is created using either the original image f (t-1) or the original image f (t), In the inset image, the object is projected in a region corresponding to the “disappearing region” or “appearing region”. In addition, when creating an interpolated image using both the original image f (t−1) and the original image f (t), the object is projected in both the “disappearing area” and the “appearing area”. There arises a problem that the accuracy of the generated interpolated image is lowered.

  On the other hand, configurations for interpolating images in consideration of “disappearing regions” and “appearing regions” are disclosed in, for example, Patent Documents 1 to 3.

Patent Document 4 discloses a technique for detecting a motion vector using a previous detection result (a detection result between an f (t-1) frame and an f (t) frame) as a candidate vector. Has been.
JP-A-8-9339 (January 12, 1996) JP 7-336688 (December 22, 1995) JP 9-214899 (August 15, 1997) JP 62-206980 (September 11, 1987)

  However, although the conventional configuration described above is an image interpolation method considering “disappearing area” and “appearing area”, there is a problem that an interpolated image cannot be generated with high accuracy.

  The configuration described in Patent Document 1 examines competing points on the original image from the vectors assigned to the interpolation points, and determines the direction of drawing, and the “disappearing area” and “appearing area” on the original image. Is not specified, and “disappearing area” and “appearing area” are not specified in the interpolated coordinates. Then, any of the competing vectors is drawn from one direction, and an operation is also performed on an area where vector expression can be correctly performed. For this reason, it is impossible to distinguish between useful vectors that have been correctly detected in the original image coordinates and non-useful vectors on the “disappearing area”, and it is necessary to exclude useful vectors from candidate vectors, resulting in reduced vector detection accuracy. Can result.

  Patent Document 2 describes a configuration for detecting an uncovered area (disappearing area / appearing area) in an original image. In the configuration described in Patent Document 2, a region where a motion vector is not detected or a region where a motion vector is not allocated and cannot be predicted is referred to as a “disappearing region” or “appearing region”. At this time, the “disappearing area” and “appearing area” in the original image are determined by a one-way motion vector. In an original image in which a similar region exists at a nearby point, the motion vector itself is easy to be erroneously detected, and there is a high possibility that “disappearing region” and “appearing region” are erroneously extracted.

  For example, when an object moves in a landscape (background) in which columns of the same shape are arranged at equal intervals between the original image f (t−1) and the original image f (t), Even when a column is erroneously detected by movement, a motion vector directed to a different column between f (t−1) and f (t) may be detected. That is, even if the start column and the end column of the motion vector are different between f (t−1) and f (t), the shape of the column is the same, so the start point of the motion vector And the end point pixel correlation value indicates “correlated”, and the motion vector is erroneously detected. Furthermore, due to erroneous detection of motion vectors, there are also points where two motion vectors are assigned in the interpolated coordinates.

  In this case, both the pixel correlation value of the start point and end point of the correct motion vector and the pixel correlation value of the start point and end point of the erroneously detected motion vector both show low values (that is, high correlation is shown). ) The correct motion vector cannot be discriminated, and the motion vector finally assigned to the interpolated coordinates is selected in accordance with the noise component and the phase difference depending on the vector detection accuracy. Sometimes. In some cases, motion vectors are assigned to interpolation points that should not be assigned. For this reason, it results in the failure of the interpolated image.

  The configuration described in Patent Document 3 is a configuration that predicts an uncovered area (disappearing area / appearing area) in the interpolated image based on the uncovered area and the bidirectional vector in the original image. Here, since the uncovered area in the original image is determined using the method described in Patent Document 2, that is, using only a one-way motion vector, “disappearing area” and “appearing in similar areas. The above-described problem of erroneously extracting “region” occurs.

  In the configuration described in Patent Document 3, “disappearing region” and “appearing region” are determined based on the original coordinates, the region is subjected to arithmetic processing, further moved to interpolation coordinates, and the vector is re-established at the interpolation point. It is a method of assignment and becomes complicated. Furthermore, when creating a plurality of interpolation images, a structure for area calculation and reallocation is required for each interpolation coordinate, which complicates the configuration and increases costs.

  Further, in the configuration described in Patent Document 4, a method of estimating a motion vector using a result detected in the past as a candidate vector is used, but in a vector from a past image f (t) to a future image f (t + 1), As a result of the previous detection, i.e., the detection between f (t-1) and f (t), the vector result indicated as "disappearing region" is assigned to similar neighboring points, Should not be a candidate for continued movement. In addition, in the vector from the future image f (t + 1) to the past image f (t), the result of detection between the previous detection f (t) and f (t−1) is indicated as “appearing region”. Vector results are assigned to similar neighbors and should not be candidates for motion continuation.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to determine a highly reliable motion vector in order to prevent the interpolation image from being broken and to accurately generate the interpolation image. An object of the present invention is to provide a motion vector correction device, a motion vector correction method, an image interpolation device, a television receiver, a video reproduction device, a control program, and a computer-readable recording medium.

  The motion vector correction device according to the present invention provides any one of the second original images at a second time after the first time, starting from each block of the first original image at the first time. A forward motion vector detecting means for detecting a forward motion vector having a block as an end point, and a backward direction starting from each block of the second original image and having any block of the first original image as an end point When the interval between the backward motion vector detecting means for detecting the motion vector, the start point of the forward motion vector, and the end point of the backward motion vector starting from the end point of the forward motion vector is larger than a predetermined threshold Determining means for determining the start point of the forward vector as a correction target block; and from the correction target block in a third original image at a third time before the first time. Appearance frequency of forward motion vector starting from a block within a predetermined range, and appearance frequency of forward motion vector starting from a block within a predetermined range from the correction target block in the first original image And the motion vector determining means for determining the forward motion vector having the lowest appearance frequency as the predicted motion vector, and the forward motion vector starting from the correction target block is replaced with the predicted motion vector. And a correction means.

  According to the above configuration, the forward motion vector detection means starts from each block of the first original image at the first time and starts the second original at the second time after the first time. A forward motion vector whose end point is one of the blocks of the image is detected. Then, the backward motion vector detecting means detects a backward motion vector starting from each block of the second original image and ending at any block of the first original image.

  Further, according to the above configuration, the determination unit is configured such that an interval between the start point of the forward motion vector and the end point of the backward motion vector starting from the end block of the forward motion vector is greater than a predetermined threshold. If it is larger, the start point of the forward vector is determined as a correction target block.

  When the region in the first original image is also present in the second original image, the start point of the forward motion vector and the end point of the backward motion vector starting from the end block of the forward motion vector are substantially the same. On the other hand, if they do not substantially match, the region existing in the first original image does not exist in the second original image. That is, the determination unit detects “disappearing area” (that is, an area hidden behind a moving object, etc.) in the first original image as the correction target block. In addition, as a predetermined threshold value, for example, the number of blocks may be preset according to the detection accuracy of the motion vector, and is not particularly limited.

  According to the above configuration, the motion vector predictor determining unit starts the block within the predetermined range from the correction target block in the third original image at the third time before the first time. Is compared with the appearance frequency of the forward motion vector starting from a block within the predetermined range from the correction target block in the first original image. The reduced forward motion vector is determined as the predicted motion vector. Then, the correcting unit replaces the forward motion vector in the correction target block with the predicted motion vector.

  For example, the predicted motion vector determination means includes the histogram data of the motion vector detected in a fixed region centered on the correction target block in the first original image, and the correction target in the first original image in the third original image. From the motion vector histogram data detected in a certain area centered on the block corresponding to the block (that is, the same coordinates as the correction target block in the first original image), the forward motion vector having the lowest appearance frequency Are detected as predicted motion vectors. The “disappearing region” is, for example, a background region that disappears due to the movement of an object, and a forward vector that originally indicates the movement of the background region should be detected. In addition, since the forward motion vector of the background region decreases with the passage of time around the “disappearing region”, the forward motion vector decreasing around the “disappearing region” is detected, and the “disappearing region” is detected. The motion vector in the past image (the original image of the past time with respect to the image to be interpolated) is corrected by replacing it with the motion vector erroneously detected in “.

  Thereby, in the motion vector correction apparatus according to the present invention, a motion vector erroneously detected in the “disappearing region” can be corrected to a motion vector with higher reliability. Therefore, when the “disappearing area” is included in the original image, it is possible to improve the accuracy of the interpolated image.

  The motion vector correction method according to the present invention includes any one of the second original images at a second time after the first time, starting from each block of the first original image at the first time. A forward motion vector detecting step for detecting a forward motion vector having a block as an end point, and a backward direction starting from each block of the second original image and having any block of the first original image as an end point When the interval between the backward motion vector detecting step for detecting the motion vector, the start point of the forward motion vector, and the end point of the backward motion vector starting from the end point of the forward motion vector is larger than a predetermined threshold A determination step of determining the start point of the forward vector as a correction target block, and the correction target in the third original image at a third time before the first time. The appearance frequency of a forward motion vector starting from a block within a predetermined range from the lock, and the forward motion vector starting from a block within the predetermined range from the correction target block in the first original image. A predicted motion vector determining step for comparing the appearance frequency and determining a forward motion vector having the lowest appearance frequency as a predicted motion vector; and a forward motion vector starting from the correction target block as the predicted motion vector. And a correction step to be replaced.

  According to said structure, there exists an effect similar to the motion vector correction apparatus which concerns on this invention.

  The motion vector correction device according to the present invention provides any one of the second original images at a second time after the first time, starting from each block of the first original image at the first time. A forward motion vector detecting means for detecting a forward motion vector having a block as an end point, and a backward direction starting from each block of the second original image and having any block of the first original image as an end point When the interval between the backward motion vector detecting means for detecting the motion vector, the start point of the backward motion vector, and the end point of the forward motion vector starting from the end point of the backward motion vector is larger than a predetermined threshold Determining means for determining the start point of the backward vector as a correction target block; and from the correction target block in a third original image at a third time before the first time. Appearance frequency of forward motion vector starting from a block within a predetermined range, and appearance frequency of forward motion vector starting from a block within a predetermined range from the correction target block in the first original image And a predicted motion vector determining means for determining a reverse vector of the forward motion vector having the highest appearance frequency as a predicted motion vector, and a backward motion vector starting from the correction target block as the predicted motion. It is characterized by comprising correction means for replacing with a vector.

  According to the above configuration, the forward motion vector detection means starts from each block of the first original image at the first time and starts the second original at the second time after the first time. A forward motion vector whose end point is one of the blocks of the image is detected. Then, the backward motion vector detecting means detects a backward motion vector starting from each block of the second original image and ending at any block of the first original image.

  Further, according to the above configuration, the determination unit is configured such that an interval between the start point of the backward motion vector and the end point of the forward motion vector starting from the end block of the backward motion vector is greater than a predetermined threshold. If it is larger, the starting point of the backward vector is determined as a correction target block.

  When a region in the second original image also exists in the first original image, the starting point of the backward motion vector and the end point of the forward motion vector starting from the end block of the backward motion vector are substantially the same. On the other hand, if they do not substantially match, the region existing in the second original image does not exist in the first original image. That is, the determination unit detects an “appearing area” (that is, an area coming out from behind a moving object or the like) in the second original image as the correction target block. In addition, as a predetermined threshold value, for example, the number of blocks may be preset according to the detection accuracy of the motion vector, and is not particularly limited.

  According to the above configuration, the motion vector predictor determining unit starts the block within the predetermined range from the correction target block in the third original image at the third time before the first time. Is compared with the appearance frequency of the forward motion vector starting from a block within the predetermined range from the correction target block in the first original image. An inverse vector of the increased forward motion vector is determined as a predicted motion vector. Then, the correcting unit replaces the forward motion vector in the correction target block with the predicted motion vector.

  For example, the predicted motion vector determination means includes the histogram data of the motion vector detected in a fixed region centered on the correction target block in the first original image, and the correction target in the first original image in the third original image. From the motion vector histogram data detected in a fixed area centered on the block corresponding to the block (that is, the same coordinates as the correction target block in the first original image), the forward motion vector having the highest appearance frequency Are detected as predicted motion vectors. The “appearing area” is, for example, a background area that appears due to the movement of an object, and a backward vector that originally indicates the movement of the background area should be detected. In the vicinity of the “appearing area”, the forward motion vector in the background area increases as time passes. Therefore, the forward motion vector that increases in the vicinity of the “appearing area” is detected and the inverse vector is used. A motion vector in a future image (an original image at a future time with respect to an image to be interpolated) is corrected by replacing a certain backward motion vector with a motion vector erroneously detected in the “appearing region”.

  Thereby, in the motion vector correction apparatus according to the present invention, a motion vector erroneously detected in the “appearing region” can be corrected to a motion vector with higher reliability. Therefore, when an “appearing region” is included in the original image, the accuracy of the interpolated image can be improved.

  The motion vector correction method according to the present invention includes any one of the second original images at a second time after the first time, starting from each block of the first original image at the first time. A forward motion vector detecting step for detecting a forward motion vector having a block as an end point, and a backward direction starting from each block of the second original image and having any block of the first original image as an end point When the backward motion vector detection step for detecting a motion vector, the interval between the start point of the backward motion vector and the end point of the forward motion vector starting from the end point of the backward motion vector is larger than a predetermined threshold A determination step of determining the start point of the backward vector as a correction target block, and the correction target in the third original image at a third time before the first time. The appearance frequency of a forward motion vector starting from a block within a predetermined range from the lock, and the forward motion vector starting from a block within the predetermined range from the correction target block in the first original image. A predicted motion vector determination step for comparing the appearance frequency and determining the inverse vector of the forward motion vector having the highest appearance frequency as a predicted motion vector, and a backward motion vector starting from the correction target block, And a correction step for replacing with a predicted motion vector.

  According to said structure, there exists an effect similar to the motion vector correction apparatus which concerns on this invention.

  In the image interpolation device according to the present invention, the determination means includes a forward motion vector starting from each block of the third original image and ending at any block of the first original image; The correction target block in the third original image is determined from the backward motion vector starting from each block of the original image of 1 and ending in any block of the third original image, and the front The direction motion vector detection means uses a vector other than the forward motion vector starting from the correction target block in the third original image as a candidate vector for estimating the forward motion vector in the first original image. Is preferred.

  According to the above configuration, the determination unit determines the correction target block in the third original image, that is, the “disappearing area” in the third original image, and the forward motion vector detection unit detects the third original image. Is not used as a candidate vector for detecting the forward motion vector in the first original image. That is, when detecting the forward motion vector of the original image at a certain time, the forward motion vector starting from the “disappearing area” of the original image at the previous time is not used as a candidate vector.

  Thus, when a forward motion vector in the first original image (current original image) is detected using the candidate vector, a false detection is made in the “disappearing region” of the third original image (previous original image). Since the forward motion vector thus set is not used as a candidate vector, it is possible to improve the detection accuracy of the forward motion vector in the first original image.

  In the image interpolation device according to the present invention, the determination means includes a forward motion vector starting from each block of the third original image and ending at any block of the first original image; The correction target block in the first original image is determined from the backward motion vector starting from each block of the original image of 1 and ending in any of the blocks of the third original image. The direction motion vector detection means uses a vector other than the backward motion vector starting from the correction target block in the first original image as a candidate vector for estimating the backward motion vector in the second original image. Is preferred.

  According to the above configuration, the determination unit determines the block to be corrected in the first original image, that is, the “appearing region” in the first original image, and the backward motion vector detection unit detects the first original image. Are not used as candidate vectors for detecting the backward motion vector in the second original image. That is, when the backward motion vector of the original image at a certain time is detected, the backward motion vector starting from the “appearing region” of the original image at the previous time is not used as a candidate vector.

  Thus, when a backward motion vector in the second original image (current original image) is detected using the candidate vector, a false detection is performed in the “appearing region” of the first original image (previous original image). Since the backward motion vector thus set is not used as a candidate vector, it is possible to improve the detection accuracy of the backward motion vector in the second original image.

  An image interpolation device according to the present invention includes a motion vector correction device, and includes a predicted motion vector starting from the correction target block, and a forward motion vector or a backward motion vector starting from a block other than the correction target block. An interpolation vector determining means for determining an interpolation vector to be allocated to each block of the interpolation image to be interpolated between the first original image and the second original image, and the interpolation vector determination And an interpolation image generation means for generating the interpolation image based on the interpolation vector determined by the means.

  According to the above configuration, the image interpolation device according to the present invention can select the predicted motion vector determined by the motion vector correction device as the interpolation vector to be allocated to each block of the interpolation image.

  As a result, the image interpolating apparatus according to the present invention can improve the reliability of the motion vector when the original image includes the “disappearing region” and the “appearing region”, and thus can generate an interpolation image with high accuracy. It becomes possible to do.

  In the image interpolation device according to the present invention, the interpolation vector determination means includes a predicted motion vector starting from the correction target block, and a forward motion vector or a backward motion vector starting from a block other than the correction target block. Are assigned to the same block of the interpolated image, a forward motion vector or a backward motion vector starting from a block other than the correction target block is determined as an interpolation vector to be assigned to the block. preferable.

  According to the above configuration, in the block of the interpolated image, the motion vector starting from an area other than the “disappearing area” and “appearing area” correctly detected in the original image is predicted by the motion vector correction apparatus. If a motion vector starting from the “disappearing region” or “appearing region” corrected using the predicted motion vector competes with the motion vector predicted by the motion vector correction device, it is accurately detected in the original image. The motion vector thus assigned is assigned to the block of the interpolated image.

  Thereby, since the motion vector with higher certainty is preferentially assigned to each block of the interpolated image, the accuracy of the interpolated image can be further improved.

  Moreover, it is preferable that the television receiver according to the present invention includes the image interpolation device.

  Moreover, it is preferable that the video reproduction apparatus according to the present invention includes the image interpolation apparatus.

  Note that the image interpolation device may be realized by a computer. In this case, a control program for realizing the image interpolation device in the computer by operating the computer as each of the above means and a computer-readable recording medium recording the control program also fall within the scope of the present invention.

  The motion vector correction device according to the present invention provides any one of the second original images at a second time after the first time, starting from each block of the first original image at the first time. A forward motion vector detecting means for detecting a forward motion vector having a block as an end point, and a backward direction starting from each block of the second original image and having any block of the first original image as an end point The backward motion vector detection means for detecting the motion vector, the interval between the start point of the forward motion vector and the end point of the backward motion vector starting from the end block of the forward motion vector is less than a predetermined threshold value. If it is larger, the determination means for determining the start point of the forward vector as the correction target block, and the correction target block in the third original image at the third time before the first time. Frequency of appearance of a forward motion vector starting from a block within a predetermined range from the block, and a forward motion vector starting from a block within the predetermined range from the correction target block in the first original image And the motion vector determining means for determining the forward motion vector having the lowest appearance frequency as the predicted motion vector, and replacing the forward motion vector in the correction target block with the predicted motion vector. And a correction means.

  In addition, the motion vector correction apparatus according to the present invention starts from each block of the first original image at the first time, and any of the second original images at the second time after the first time. Forward motion vector detecting means for detecting a forward motion vector whose end point is the block, and each block of the second original image as a start point, and any block of the first original image as an end point The distance between the backward motion vector detecting means for detecting the backward motion vector, the start point of the backward motion vector, and the end point of the forward motion vector starting from the end block of the backward motion vector is a predetermined threshold value. Is larger than the determination means for determining the start point of the backward vector as the correction target block, and the correction in the third original image at the third time before the first time. Appearance frequency of forward motion vector starting from a block within a predetermined range from the elephant block, and forward motion vector starting from a block within the predetermined range from the correction target block in the first original image The predicted motion vector determining means for determining the inverse vector of the forward motion vector having the highest appearance frequency as the predicted motion vector, and the backward motion vector in the correction target block as the predicted motion. It is characterized by comprising correction means for replacing with a vector.

  Therefore, in the motion vector correction apparatus according to the present invention, since the motion vector erroneously detected in the “disappearing region” and the “appearing region” can be corrected to a more reliable motion vector, “disappears” in the original image. When “region” and “appearing region” are included, the accuracy of the interpolated image can be improved.

[Embodiment 1]
(Outline of the image interpolation device 1)
FIG. 1 is a block diagram showing a configuration of an image interpolation apparatus 1 according to the present invention. An image interpolation apparatus 1 according to the present invention includes a delay unit 2, a motion vector detection unit 3 (backward motion vector detection unit), a motion vector detection unit 4 (forward motion vector detection unit), a state recording unit 5, and statistical data processing. A unit 6, a bidirectional vector comparison unit 7 (determination unit), a vector correction unit 8 (predicted motion vector determination unit, correction unit), a vector allocation unit 9, and a drawing unit 10

  The motion vector correction apparatus 100 includes a motion vector detection unit 3 (backward motion vector detection unit), a motion vector detection unit 4 (forward motion vector detection unit), a bidirectional vector comparison unit 7 (determination means), and vector correction. Part 8 (predicted motion vector determination means, correction means). In the present embodiment, the motion vector correction apparatus 100 is provided integrally with the image interpolation apparatus 1, and forms the image interpolation apparatus 1 together with the vector allocation unit 9 and the drawing unit 10.

  The image interpolation device 1 includes an original image f (t) (second original image, future image) at time t included in a video signal input from the outside, and an original image f ( t-1) An interpolated image is generated from (first original image, past image). In the image interpolation device 1, an external video signal is input to the delay unit 2 and the drawing unit 10. Hereinafter, processing when a video signal representing the original image f (t) at time t is input from the outside will be described.

  The delay unit 2 delays the input video signal from the outside and supplies it to the drawing unit 10. At a timing when a video signal representing the original image f (t) is input from the outside, the delay unit 2 outputs a video signal representing the original image f (t−1). The delay unit 2 is configured by a memory such as DDR, for example.

  The motion vector detection units 3 and 4 calculate bidirectional motion vectors using the original image f (t) and the original image f (t−1) as reference images, respectively. In addition, as shown in FIG. 2, the motion vector detection units 3 and 4 divide the original image into a plurality of blocks including a plurality of pixels, and detect a motion vector in units of blocks. The motion vector detection units 3 and 4 calculate a motion vector by block matching or a gradient method, but the motion vector may be calculated by another method. The motion vector detection units 3 and 4 output the detected motion vector to the bidirectional vector comparison unit 7 and the statistical data processing unit 6.

  The bidirectional vector comparison unit 7 compares the motion vectors detected by the motion vector detection units 3 and 4, and specifies “appearing region” and “disappearing region” (correction target block) based on the comparison result. Then, the bidirectional vector comparison unit 7 uses the motion vector detected by the motion vector detection units 3 and 4 together with the motion vector detected by the motion vector correction unit 8 and the statistical data processing unit 6 to identify the “appearing region” and “disappearing region” Output to.

  The statistical data processing unit 6 statistically processes the motion vector information from the motion vector detection units 3 and 4 and the information specifying the “appearing region” and “disappearing region” from the bidirectional vector comparison unit 7, Statistical data (for example, a histogram) is calculated and output to the state recording unit 5.

  The motion vector correction unit 8 corrects the motion vectors detected in the “appearing region” and “disappearing region” among the motion vectors detected by the motion vector detection units 3 and 4, and assigns the corrected motion vectors to the vectors. To the unit 9. At this time, the motion vector correction unit 8 reads the statistical data from the state recording unit 5 and determines new motion vectors (predicted motion vectors) in the “appearing region” and “disappearing region” based on the read statistical data. .

  The vector assignment unit 9 assigns the corrected motion vector from the motion vector correction unit 8 to each coordinate of the interpolated image, and outputs the motion vector assignment information to the drawing unit 10. The unit of the assigned coordinates is arbitrary, but in this embodiment, an area smaller than the block unit of the reference image is designated. When a plurality of motion vectors compete in the interpolated coordinates, the vector allocating unit 9 determines a motion vector to be allocated to the interpolated coordinates according to the priority set for the corrected motion vector.

  Note that if the conflict is not resolved only by the priority set for the corrected motion vector, the vector allocating unit 9 selects an optimal motion vector from the conflicting motion vectors. As a method for determining the optimum motion vector, the method having the highest correlation between the specific area of the original image f (t) and the original image f (t−1) designated by the motion vector assigned to the interpolation coordinates is representative. To do. In general, luminance data is used to calculate correlation data (correlation value). However, RGB data or YUV data may be used. As the correlation value, SAD (absolute sum of differences) obtained by calculating and adding the absolute value of the difference for each corresponding pixel, or DFD (difference of the difference) calculating and adding the absolute value of the difference for each corresponding pixel. Square sum) is calculated. The vector allocating unit 9 allocates a zero vector in the interpolated coordinates where the motion vector is not allocated. Further, the vector assigning unit 9 performs a filtering process using a median filter or the like, and assigns a motion vector to each coordinate of the interpolated image. Then, the vector allocating unit 9 outputs the motion vector information allocated to the interpolated coordinates to the drawing unit 10.

  Then, based on the motion vector allocation information from the vector allocation unit 9, the drawing unit 10 reads the video signal at each coordinate of the interpolated image from the video signal of the original image, and generates an interpolated image.

(Image interpolation processing)
First, of the image interpolation processing in the image interpolation apparatus 1 according to the present invention, processing contents similar to general image interpolation processing will be described.

(1) Motion Vector Detection FIG. 2 is a diagram showing an original image of a video in which the small object A moves, (a) is a diagram showing an original image f (t−1) at time t−1, (B) is a figure which shows the original image f (t) in the time t. In the example illustrated in FIG. 2, an interpolated image between time t and time t−1 is generated. Time t-1 is the time immediately before time t, the original image f (t-1) is a past image with respect to the interpolated image, and the original image f (t) is with respect to the interpolated image. It will be a future image.

  As shown in FIG. 2 (a), in the original image f (t-1), a portion surrounded by a thin line like the part 2-1 represents one pixel (pixel), and is 8 × 8 pixels. Detection blocks 2-2 to 2-7 are configured for each pixel group. Similarly, as shown in FIG. 2B, in the original image f (t), detection blocks 2-9 to 2-13 are configured for each pixel group of 8 × 8 pixels. A white pixel group in the original images f (t−1) and f (t) represents the moving small object A. In FIGS. 2A and 2B, a region indicated by diagonal lines represents the background region of the small object A. In FIGS. 2A and 2B, regions indicated by the same line type represent the same background region. 2A indicates the motion vector of the small object A, and the arrow 2-8 indicates the motion vector of the background area.

  FIG. 3 is a diagram for explaining an interpolation image drawing process. The original image and the interpolation image shown in FIGS. It is the figure displayed one-dimensionally. In FIG. 3, the past image f (t−1) and the future image f (t) are shown associated with each other by a motion vector. The motion vector is detected for each detection block between f (t−1) and f (t). As a method for obtaining the motion vector, a general method such as block matching or a gradient method is used.

  In motion vector detection, “disappearing areas” and “appearing areas” are problematic. For example, the image of the background area shown in the detection block 3-5 is a “disappearing area” that is hidden by a small object in the future image. That is, the background image shown in the detection block 3-5 of the original image f (t-1) is hidden by the moving small object in the original image f (t). Therefore, a correct motion vector cannot be detected for the detection block 3-5. However, there is no detection method that states that no solution exists. Therefore, for the detection block 3-5, a motion vector indicating a region similar to the detection block 3-5 is detected at f (t).

  The image interpolating apparatus 1 according to the present invention discriminates motion vectors in the “disappearing region” and the “appearing region” (that is, detects the “disappearing region” and the “appearing region”), and the “disappearing region”, “ The motion vector detected in the “appearing region” is corrected based on the statistical data.

(2) Allocation of motion vectors Hereinafter, allocation of motion vectors to interpolation coordinates will be described. As described above, the image interpolation apparatus 1 according to the present invention corrects the motion vectors detected in the “disappearing region” and the “appearing region” based on the statistical data, and then assigns the motion vector to the interpolation coordinates. Process.

  FIG. 3 shows a state in which motion vectors detected for the original image f (t) with respect to the original image f (t−1) are assigned to the interpolation coordinates. In the example shown in FIG. 3, the motion vector is obtained from the past image f (t−1) to the future image f (t), and is shown as ideally obtained. In FIG. 3, a solid line arrow indicates a motion vector detected in the original image and indicates a motion vector allocated in the interpolation block, and a broken line arrow indicates a motion vector detected in the original image, A motion vector that has not been assigned due to contention in the interpolation block is shown, and an arrow with a one-dot chain line indicates a motion vector that is not detected in the original image and that is assigned to the interpolation block according to a predetermined condition. Show.

  In FIG. 3, the original image f (t-1) includes detection blocks 3-1 to 3-5, and the original image f (t) includes detection blocks 3-6 to 3-10. The detection blocks 3-1 to 3-5 in FIG. 3 correspond to the detection blocks 2-2 to 2-6 in FIG. 2A, and the detection blocks 3-6 to 3-10 in FIG. This corresponds to the detection blocks 2-9 to 2-13 of 2 (b).

  In the example illustrated in FIG. 3, the interpolated image fi (t−1) is created at an intermediate point between the original image f (t−1) and the original image f (t). The interpolated image fi (t−1) includes interpolated blocks 3-11 to 3-20. A motion vector detected in the reference original image is assigned to the interpolation blocks 3-11 to 3-20. The interpolation block is an area smaller than the detection block, and includes fewer pixels than the detection block.

  In the example shown in FIG. 3, a vector indicating the motion of the background image is detected as the motion vector in the detection block 3-1. Then, the motion vector detected in the detection block 3-1 is allocated to the interpolation block 3-11 and is shown as a vector 3-21. Hereinafter, the motion vector assigned to the interpolation block is referred to as an interpolation vector.

  Further, a vector indicating the motion of the small object is detected as the motion vector in the detection block 3-2. The motion vectors detected in the detection block 3-2 are allocated to the interpolation blocks 3-15 and 3-16, and are indicated as interpolation vectors 3-32 and 3-33.

  Further, a vector indicating the background motion is detected as the motion vector in the detection block 3-3. And the motion vector detected in the detection block 3-3 is allocated to the interpolation blocks 3-14 and 3-15, and is shown as the interpolation vectors 3-23 and 3-28. Here, in the interpolation block 3-15, the vector 3-32 indicating the motion of the object competes with the vector 3-28 indicating the motion of the background, but there is finally one motion vector used for drawing. . Therefore, a condition for determining a motion vector used for drawing when a vector conflict occurs is determined in advance, and one motion vector is finally determined according to the condition. In the example illustrated in FIG. 3, it is assumed that a vector 3-32 indicating the motion of a small object is assigned to the interpolation block 3-15 according to a predetermined condition.

  Also, a vector indicating the background motion is detected as the motion vector in the detection block 3-4. The motion vectors detected in the detection block 3-4 are allocated to the interpolation blocks 3-16 and 3-17, and are indicated as interpolation vectors 3-29 and 3-34. Here, in the interpolation block 3-16, the vector 3-33 indicating the motion of the object competes with the vector 3-29 indicating the background motion. In the example illustrated in FIG. 3, it is assumed that a vector 3-33 indicating the motion of a small object is assigned to the interpolation block 3-16 according to a predetermined condition.

  FIG. 3 shows an example in which no vector to be allocated exists in the interpolation blocks 3-12, 3-13, 3-18, and 3-19. In general, if there is no vector assigned to the interpolation point, drawing becomes impossible. For this reason, in the motion vector allocation process, vector allocation conditions are set in advance so as to be allocated to a wider range of interpolation blocks. If the assignment is still not made, a zero vector may be assigned, or an average value of vectors assigned to surrounding interpolation blocks may be used.

  In the example shown in FIG. 3, an interpolation vector assigned to an interpolation block adjacent to an unallocated block is allocated to an interpolation block (unallocated block) to which no motion vector is allocated. Thereby, in the example shown in FIG. 3, with respect to the interpolation blocks 3-12, 3-13, 3-18, 3-19, respectively, the adjacent interpolation blocks 3-11, 3-14, 3-17 are provided. , 3-20, the same vector as the vector is allocated. The vectors assigned to the interpolation blocks 3-12, 3-13, 3-18, 3-19 are shown as interpolation vectors 3-26, 3-27, 3-30, 3-31, respectively.

(3) Drawing of Interpolated Image Next, a method of drawing an interpolated image using the interpolation vector will be described. As a method of drawing an interpolated image, there are a method of drawing from either one of the past image f (t-1) or the future image f (t) and a method of drawing from both the past image and the future image.

  The method of drawing from one side will be described as follows. In FIG. 2A, the upper left coordinate of the drawing is (0, 0), the right direction of the drawing is the positive direction of the x coordinate, and the lower direction of the drawing is the positive direction of the y coordinate. The pixel value of coordinates (x, y) is represented as Pp (x, y), and the pixel value of coordinates (x, y) in the future image f (t) is represented as Pf (x, y). In addition, in the interpolated image fi (t−1), the motion vector assigned to the coordinate (x, y) is (Vx, Vy), and the pixel value of the coordinate (x, y) is P (x, y). . Here, it is assumed that the interpolated image fi (t-1) is drawn in the middle (1/2 position) between the past image f (t-1) and the future image f (t).

At this time, when drawing the interpolated image fi (t−1) from only the past image f (t−1),
P (x, y) = Pp (x−Vx × 1/2, y−Vy × 1/2)
It becomes. Also, when drawing the interpolated image fi (t-1) from only the future image f (t),
P (x, y) = Pf (x + Vx × 1/2, y + Vy × 1/2)
It becomes. Furthermore, when drawing from the bidirectional of the past image f (t-1) and the future image f (t),
P (x, y) = α × Pp (x−Vx × 1/2, y−Vy × 1/2) + (1−α) × Pf (x + Vx × 1/2, y + Vy × 1/2)
It becomes.

  Here, Pp (x−Vx × 1/2, y−Vy × 1/2), Pf (x + Vx × 1/2, y + Vy × 1/2) does not indicate a single pixel, but a past image, It means a value that represents the future. For example, when the vector indicates a decimal value, it means a value obtained by linear interpolation of the original image, a value filtered by a Gaussian filter, or the like. Α is a mixing ratio of the past image and the future image, and is called a blend ratio. The value of α may be fixed to ½, or may be varied depending on the ratio of temporal internal dividing points of the interpolation.

  Furthermore, in order to improve the reliability of the motion vector and reduce artifacts in the interpolated image, the image may be drawn using a bidirectional vector. The bi-directional vector not only obtains the forward motion vector by referring to the image of f (t) with f (t−1) as the reference image, but also f (t−1) with f (t) as the reference image. ), The backward motion vector is obtained by referring to the image in (2). For example, for the interpolated coordinates, there are prepared those assigned only the forward motion vector and those assigned only the backward motion vector, and f (t−1) and f based on the assigned vectors. By determining a vector having a high image correlation with respect to the specific region (t) as the final vector, the reliability of the vector can be improved.

(Problems of disappearing and appearing areas)
As described above, in motion vector detection, there is a problem that an interpolated image breaks down due to erroneous detection of motion vectors caused by “disappearing regions” and “appearing regions”. In particular, in an original image in which an object moves against a background of a plurality of similar scenery existing in close proximity, it is difficult to accurately detect “disappearing region” and “appearing region” by the conventional method. The accuracy of the interpolated image generated based on the erroneously detected motion vector is significantly reduced.

  An example in which an interpolated image breaks down due to “disappearing area” and “appearing area” will be described in more detail with reference to FIG. FIG. 4 is a diagram illustrating a state in which an object moves with a plurality of columns having the same shape arranged at equal intervals in the background. FIG. 4A illustrates an original image f (t−1) and a time t at time t−1. FIG. 4B is a diagram illustrating an original image f (t) in FIG. 2, and FIG. 5B is a diagram illustrating an interpolated image generated from the original image illustrated in FIG. FIG. 4A shows a motion vector Va representing the motion of the object and motion vectors Vb and Vc representing the motion of the space between the two pillars. The start points of the motion vectors Vb and Vc are different background regions in the original image f (t−1), but the end points of all the motion vectors point to the same background region in the original image f (t).

  In the example shown in FIG. 4A, the starting point and the ending point of the motion vector Vc show different backgrounds. However, since they are very similar, the pixel correlation value indicates “correlated” and is erroneously detected. Yes. Then, when an interpolated image is generated based on the erroneously detected motion vector, as shown in FIG. 4 (b), the column disappears and is displayed, and the interpolated image breaks down.

  Therefore, in the image interpolation device 1 according to the present invention, the motion vectors detected in the “disappearing region” and the “appearing region” are more accurately determined, and the motion vector is corrected based on the statistical data. Below, the characteristic structure of the image interpolation apparatus 1 is demonstrated in detail using FIGS.

(Disappearing area detection and vector correction)
Hereinafter, detection of “disappearing region” and correction of a motion vector in the image interpolation device 1 according to the present invention will be described in detail.

  First, detection of “disappearing area” will be described. FIG. 5 is a diagram for explaining detection of “disappearing region” and correction of motion vectors, and shows bidirectional motion vectors detected in the original images f (t−1) and f (t). is there. FIG. 5 is a one-dimensional display of an image showing the movement of an object with the plurality of pillars shown in FIG. The original images f (t−1) and f (t) shown in FIG. 5 correspond to the original images f (t−1) and f (t) shown in FIG. 4A, respectively. FIG. 5 also shows an original image f (t-2) (third original image) immediately before the original image f (t-1).

  In FIG. 5, the original image f (t−1) is divided into detection blocks 5-14 to 5-26. The detection blocks 5-14 to 5-19 are regions representing moving objects shown in FIG. 4A, and the detection blocks 5-20, 5-23, and 5-24 are shown in FIG. The detection blocks 5-21, 5-22, 5-25, and 5-26 are areas that represent the columns shown in FIG. 4A. Similarly, the original image f (t) is divided into detection blocks 5-27 to 5-39. The detection blocks 5-27 to 5-36 are areas representing moving objects, the detection block 5-37 is an area representing between the columns, and the detection blocks 5-38 and 5-39 represent the columns. It is an area.

  FIG. 5 shows opposing motion vectors (bidirectional vectors) detected for each original image with reference to each of the original image f (t−1) and the original image f (t). FIG. 5 shows a motion vector detected for the original image f (t−1) with the original image f (t−2) as a reference.

  In the image interpolation apparatus 1 according to the present invention, the motion vector detection unit 4 detects a motion vector for the original image f (t) using the original image f (t−1) as a reference image, as shown in FIG. That is, a forward motion vector starting from each detection block of the past image f (t−1) and ending at any detection block of the future image f (t) is detected. Further, in the image interpolation apparatus 1, the motion vector detection unit 3 detects a motion vector for the original image f (t-1) using the original image f (t) as a reference image, as shown in FIG. That is, a backward motion vector is calculated that starts from each detection block of the future image f (t) and ends at any detection block of the past image f (t−1). Then, the bidirectional vector comparison unit 7 compares the bidirectional motion vectors detected by the motion vector detection units 3 and 4 for each detection block.

  The bidirectional vector comparison unit 7 is an original image that is an end point of each forward motion vector with respect to a forward motion vector starting from each detection block of the original image f (t−1) detected by the motion vector detection unit 4. Compared with the backward motion vector starting from the detection block of f (t), it is determined whether or not the opposing forward motion vector and backward motion vector are opposite vectors of equal magnitude. To do. That is, it is determined whether or not the end point of the backward motion vector starting from the end block of the forward motion vector matches the start point of the forward motion vector.

  It should be noted that the process of determining whether or not the opposing forward motion vector and the backward motion vector are equal in the bidirectional vector comparison unit 7 may have a width in consideration of the detection accuracy of the motion vector. That is, whether the interval between the start point of the forward motion vector and the end point of the backward motion vector starting from the end block of the forward motion vector is greater than a predetermined threshold (within a predetermined range) It is also possible to adopt a configuration in which it is determined whether or not the area disappears when the threshold value is larger than a predetermined threshold. Here, the predetermined threshold (predetermined range) may be configured such that the number of blocks corresponding to the motion vector detection accuracy can be set, and is not particularly limited.

An example of comparison processing in the bidirectional vector comparison unit 7 will be described with reference to FIG. FIG. 12 is a diagram for explaining the processing in the bidirectional vector comparison unit 7. As a result of detecting the forward motion vector from the original image f (t-1) to f (t), the forward motion vector (Vxft1, Vyft1) is detected in a certain block on f (t-1). To do. In FIG. 12, blocks 12-1 to 12-4 are fixed blocks on f (t) when backward detection is performed on the original images f (t) to f (t-1). Block 12-5 is an area allocated on f (t) when the vector detection result of forward detection in a certain block on f (t-1) indicates (Vxft1, Vyft1). That is, the block 12-5 is a block at the end of the forward motion vector (Vxft1, Vyft1) starting from f (t-1). Further, the backward vectors obtained in the blocks 12-1 to 12-4 are (Vxbt1, Vybt1) to (Vxbt4, Vybt4). At this time, the bidirectional vector comparison unit 7 performs a comparison process between the opposing forward motion vector and the backward motion vector,
| Vxft1 + Vxbt1 | <Vxth
｜ Vxft1 + Vxbt2 | <Vxth
｜ Vxft1 + Vxbt3 | <Vxth
｜ Vxft1 + Vxbt4 | <Vxth
Is true, and | Vyft1 + Vybt1 | <Vyth
| Vyft1 + Vybt2 | <Vyth
｜ Vyft1 + Vybt3 | <Vyth
｜ Vyft1 + Vybt4 | <Vyth
If either of the above holds, the opposite forward motion vector and the backward motion vector are equal, that is, the backward motion vector starting from the end block 12-5 of the forward motion vector (Vxft1, Vyft1). It is determined that the end point coincides (or exists within a predetermined range) with the start point of the forward motion vector. Here, Vxth and Vyth are thresholds predetermined as a reference for determining whether or not the opposing forward motion vector and backward motion vector are equal, and can be changed by setting.

  That is, when the difference between the detected forward motion vector and the backward vector of the block around the end point indicated by the forward motion vector is equal to or less than a certain value (because the forward and backward vectors have opposite directions, The result of the addition is ideally zero), and it is determined that the end point of the backward motion vector starting from the end block of the forward motion vector coincides with the start point of the forward motion vector.

  Here, the bidirectional vector comparison block is a 2 × 2 area, but depending on the calculation accuracy, a 3 × 3 area or a single block may be used.

  The bidirectional vector comparison unit 7 will be described below with reference to FIG. In the following, the comparison block is described as a single block for the sake of simplicity. The bidirectional vector comparison unit 7 uses the backward motion vector starting from the detection block of the end point of each forward motion vector, for the forward motion vector starting from the detection blocks 5-14 to 5-26 shown in FIG. Compare.

  For example, the end point of the forward motion vector 5-40 starting from the detection block 5-24 of the original image f (t-1) is the detection block 5-37 of the original image f (t). The unit 7 compares the backward motion vector 5-41 starting from the detection block 5-37 with the forward motion vector 5-40. In this case, the bidirectional vector comparison unit 7 is both a zero vector for the forward motion vector 5-40 and the backward motion vector 5-41, and the start point of the forward motion vector and the forward motion vector It is determined that the end point of the backward motion vector starting from the end point is equal.

  Further, for example, the end point of the forward motion vector 5-42 starting from the detection block 5-23 of the original image f (t-1) is the detection block 5-37 of the original image f (t). The vector comparison unit 7 compares the backward motion vector 5-41 starting from the detection block 5-37 with the forward motion vector 5-42. In this case, the bidirectional vector comparison unit 7 determines that the start point of the forward motion vector of the detection block 5-23 is different from the end point of the backward motion vector starting from the end point of the forward motion vector. .

  The bidirectional vector comparison unit 7 performs the same determination process for all detection blocks of the original image f (t−1). In the example illustrated in FIG. 5, the bidirectional vector comparison unit 7 determines the forward motion vector start point and the forward motion vector end point for the forward motion vector starting from the detection blocks 5-20 to 5-23. It is determined that the end point of the backward motion vector as the start point is different.

  When the start point of the forward motion vector and the end point of the backward motion vector starting from the end point of the forward motion vector are the same, the detection block of the original image f (t) indicated by the forward motion vector, The detection block of the original image f (t−1) indicated by the directional motion vector is the same object region or the same background region.

  On the contrary, when the start point of the forward motion vector is different from the end point of the backward motion vector starting from the end point of the forward motion vector, the original image f (t−1) that is the start point of the forward motion vector This means that the object or background area displayed in the detection block is not displayed in the original image f (t).

  That is, the forward motion vector starting from the detection block in the original image f (t−1), and the backward motion vector starting from the detection block in the original image f (t) that is the end point of the forward motion vector. To determine whether the forward motion vector and the backward motion vector are opposite in direction and equal in magnitude, thereby obtaining the original image f (t−1) at the previous time in time. Is displayed, but the “disappearing area” that is not displayed in the original image f (t) at a later time in time can be detected. In the example illustrated in FIG. 5, the detection blocks 5-20 to 5-23 in the original image f (t−1) are specified as “disappearing areas”.

  Next, correction of the motion vector in the “disappearing region” will be described. In FIG. 5, the forward motion vector detected in the detection blocks 5-20 to 5-23 of the original image f (t−1) which is the “disappearing region” is a falsely detected motion vector indicating a similar region. is there. Therefore, when the forward motion vector is used for drawing the interpolated image, it is necessary to correct the erroneously detected forward motion vector in the “disappearing region”.

  In the image interpolating apparatus 1 according to the present invention, when the motion vector in the “disappearing region” of the original image f (t−1) is corrected, the motion vector correction unit 8 performs the motion vector of the original image f (t−1). And the statistical data of the motion vector of the original image f (t-2) at the time immediately before the original image f (t-1).

  More specifically, the motion vector correction unit 8 includes the histogram data of the forward motion vector in the detection block within a predetermined range around the detection block of the “disappearing region” in the original image f (t−1). Compared with the histogram data of the forward motion vector in the detection block within the predetermined range centering on the detection block corresponding to the “disappearing area” in the original image f (t−2), the frequency is most decreased. The forward motion vector is determined as the motion vector in the “disappearing area” of the original image f (t−1).

  That is, in the “disappearing region”, the object enters, and the background region decreases with the passage of time. Therefore, the forward motion vector in the background area is reduced. Therefore, the motion vector in the “disappearing region” is corrected to the motion vector with the lowest frequency in the histogram data of the motion vector detected last time and the forward motion vector detected this time.

  In the present embodiment, the motion vector correction unit 8 reads out the histogram data of the motion vectors in the original image f (t−2) and the original image f (t−1) from the state recording unit 5 and compares them. The statistical data processing unit 6 receives the motion vector data detected by the motion vector detection units 3 and 4 and the data specifying the “disappearing region” determined by the bidirectional vector comparison unit 7, and receives the original image f. In (t-1), the motion vector histogram data in the detection block within a predetermined range centering on the detection block of the “disappearing region”, and the original image f (t−1) in the original image f (t−2). The motion vector histogram data in the detection block within a predetermined range centering on the detection block corresponding to the “disappearing area” is calculated and stored in the state recording unit 5. At this time, only the upper data of the histogram data may be stored in the state recording unit 5. Further, the statistical data processing unit 6 performs statistical processing on motion vectors in regions other than the “appearing region” and the “disappearing region” based on the “appearing region” and “disappearing region” information from the bidirectional vector comparison unit 7. The structure which calculates data may be sufficient. That is, the accuracy of statistical data can be further improved by excluding the motion vector data in the “appearing area” and “disappearing area” from the statistical data.

  In the example shown in FIG. 5, the motion vector correction unit 8 is, for example, 5 centered on each detection block included in the “disappearing area” (detection blocks 5-20 to 5-23) of the original image f (t−1). Histograms of motion vectors within the block range and areas corresponding to “disappearing areas” of the original image f (t−1) in the original image f (t−2) (detection blocks 5-7 to 5-10) Each detected block is compared with a histogram of motion vectors within a range of 5 blocks. In the example illustrated in FIG. 5, the appearance frequency is within a range of 5 blocks centering on each detection block included in the “disappearing region” between the original image f (t−2) and the original image f (t−1). The most reduced motion vector is the zero vector. Therefore, the motion vector correction unit 8 sets a zero vector as a motion vector in the “disappearing region” (detection blocks 5-20 to 5-23) of the original image f (t−1).

  In the above description, in order to simplify the description, a configuration for calculating histogram data within a range of 5 blocks centering on the “disappearing region” based on FIG. 5 in which the original image shown in FIG. 4 is modeled. As described above, the range for calculating the histogram data is not limited to 5 blocks centered on the “disappearing area”, and the number of blocks for calculating the histogram data is set according to the type of the original image. There may be, and it does not specifically limit.

  Then, the motion vector correction unit 8 supplies the corrected motion vector to the vector allocation unit 9. At this time, the motion vector correction unit 8 uses an indicator that lowers the priority in the allocation process to the interpolated image for the motion vector of the “disappearing region”, that is, the interpolation vector when contention occurs in the interpolation block. Is supplied to the vector allocating unit 9. The vector allocation unit 9 performs allocation processing to the interpolation block based on the priority instruction information from the motion vector correction unit 8 and supplies the motion vector allocated to the interpolated image, that is, the interpolation vector, to the drawing unit 10. To do.

  The motion vector correction unit 8 also draws the motion vector of the “disappearing area” from the past image, that is, the original image f (t−1) at a time before the interpolation time. Is supplied to the vector allocation unit 9. As the drawing instruction information, for example, information instructing to draw with the blend ratio α = 1 in the formula in the case of drawing from the past image and the future image is supplied. When the motion vector of the “disappearing area” is assigned to the interpolated image, the vector assigning unit 9 supplies the drawing instruction information together with the interpolation vector to the drawing unit 10. Then, the drawing unit 10 draws an interpolation image using the interpolation vector and the drawing instruction information from the vector allocation unit 9.

  FIG. 6 is a diagram for explaining the “disappearing region” detection and the motion vector correction, and shows how a motion vector is assigned to an interpolated image using the corrected motion vector. Detection blocks 6-1 to 6-6 in the original image f (t-1) in FIG. 6 correspond to the detection blocks 5-19 to 5-24 in the original image f (t-1) in FIG. As described above, a zero vector is set in the “disappearing area” of the original image f (t−1) (detection blocks 5-20 to 5-23, that is, detection blocks 6-2 to 6-5).

  FIG. 6 illustrates the interpolated images f (t-0.75), f (t-0.5), and f (t−t−) at the times t−0.75, t−0.5, and t−0.25. 0.25) is shown as an example of interpolation. As shown in FIG. 6, no motion vector competition occurs in the interpolation block 6-7 of the interpolated image f (t-0.25), and the vector allocating unit 9 performs the original image f (t-1). The motion vector in the detection block 6-3 which is the “disappearing area” of FIG. On the other hand, motion vector contention occurs in the interpolation block 6-8 of the interpolated image f (t-0.5) and the interpolation block 6-9 of the interpolated image f (t-0.75). ing.

  In this case, as described above, the vector allocation unit 9 instructs the motion vector correction unit 8 not to select the motion vector of the “disappearing region” as the interpolation vector when a conflict occurs in the interpolation block. Receives priority indication information. Therefore, in the interpolation block 6-8, the motion vector in the detection block 6-3 that is the “disappearing region” is not selected as the interpolation vector, and the motion vector detected in the detection block 6-1 is used as the interpolation vector. select. Similarly, in the interpolation block 6-9, the motion vector in the detection block 6-3 that is the “disappearing region” is not selected as the interpolation vector.

  Further, as described above, the drawing unit 10 receives the drawing instruction information for drawing from the past image and the past image with respect to the motion vector of the “disappearing area”. Therefore, for example, the drawing unit 10 detects the original image f (t−1) for the interpolation block 6-7 of the interpolation image f (t−0.25) to which the motion vector of the “disappearing region” is assigned. Drawing is performed using the image of block 6-3.

  Thereby, in the image interpolation apparatus 1 according to the present invention, a motion vector erroneously detected in the “disappearing region” can be corrected to a motion vector with higher reliability. Therefore, when the “disappearing area” is included in the original image, it is possible to improve the accuracy of the interpolated image.

(Detection of appearing area and vector correction)
Hereinafter, detection of “appearing regions” and correction of motion vectors in the image interpolation device 1 according to the present invention will be described in detail.

  First, detection of “appearing area” will be described. FIG. 7 is a diagram for explaining detection of “appearing region” and correction of motion vectors, and shows bidirectional motion vectors detected in the original images f (t−1) and f (t). is there. FIG. 7 is a one-dimensional display of an image showing how an object moves against a background of a plurality of columns as shown in FIG. FIG. 7 also shows an original image f (t-2) (third original image) immediately before the original image f (t-1).

  In FIG. 7, the original image f (t-1) is divided into detection blocks 7-14 to 7-26. The detection blocks 7-21 to 7-26 are areas representing moving objects shown in FIG. 4A, and the detection blocks 7-15, 7-16, 7-19, and 7-20 are shown in FIG. It is an area | region showing the pillar shown to (a), and detection block 7-14, 7-17, and 7-18 are areas showing between the pillars shown to Fig.4 (a). Similarly, the original image f (t) is divided into detection blocks 7-27 to 7-39. The detection blocks 7-38 to 7-39 are areas representing moving objects, and the detection blocks 7-28, 7-29, 7-32, 7-33, 7-36, and 7-37 are columns. The detection blocks 7-27, 7-30, 7-31, 7-34, and 7-35 are areas representing between the columns.

  FIG. 7 shows opposing motion vectors (bidirectional vectors) detected for the original images with reference to the original image f (t−1) and the original image f (t). FIG. 5 shows a motion vector detected for the original image f (t−1) with the original image f (t−2) as a reference.

  In the image interpolation apparatus 1 according to the present invention, the motion vector detection unit 4 detects a motion vector for the original image f (t) using the original image f (t−1) as a reference image, as shown in FIG. That is, a forward motion vector starting from each detection block of the past image f (t−1) and ending at any detection block of the future image f (t) is detected. Further, in the image interpolation apparatus 1, the motion vector detection unit 3 detects a motion vector for the original image f (t-1) using the original image f (t) as a reference image, as shown in FIG. That is, a backward motion vector is calculated that starts from each detection block of the future image f (t) and ends at any detection block of the past image f (t−1). Then, the bidirectional vector comparison unit 7 compares the bidirectional motion vectors detected by the motion vector detection units 3 and 4 for each detection block.

  The bidirectional vector comparison unit 7 is an original image that is an end point of each forward motion vector with respect to a forward motion vector starting from each detection block of the original image f (t−1) detected by the motion vector detection unit 4. Compared with the backward motion vector starting from the detection block of f (t), it is determined whether or not the opposing forward motion vector and the backward motion vector are opposite in the opposite direction and equal in magnitude. To do. That is, it is determined whether or not the end point of the forward motion vector starting from the end block of the backward motion vector matches the start point of the backward motion vector.

  Note that the above-described determination process of whether or not the opposing forward motion vector and the backward motion vector are equal in the bidirectional vector comparison unit 7 may have a width in consideration of the detection accuracy of the motion vector. That is, whether the interval between the start point of the backward motion vector and the end point of the forward motion vector starting from the end block of the backward motion vector is larger than a predetermined threshold (within a predetermined range) It is also possible to adopt a configuration in which the “appearing region” is determined when the value is larger than a predetermined threshold. Here, the predetermined threshold (predetermined range) may be configured such that the number of blocks corresponding to the motion vector detection accuracy can be set, and is not particularly limited.

An example of comparison processing in the bidirectional vector comparison unit 7 will be described with reference to FIG. FIG. 12 is a diagram for explaining the processing in the bidirectional vector comparison unit 7. It is assumed that the backward motion vector (Vxft1, Vyft1) is detected in a certain block on f (t) as a result of detecting the backward motion vector from the original image f (t) to f (t-1). In FIG. 12, blocks 12-1 to 12-4 are fixed blocks on f (t-1) when forward detection is performed on the original images f (t-1) to f (t). The block 12-5 is an area allocated on f (t-1) when the vector detection result of backward detection in a certain block on f (t) indicates (Vxft1, Vyft1). That is, the block 12-5 is a block at the end of the backward motion vector (Vxft1, Vyft1) starting from f (t). Further, the forward vectors obtained in the blocks 12-1 to 12-4 are (Vxbt1, Vybt1) to (Vxbt4, Vybt4). At this time, the bidirectional vector comparison unit 7 performs a comparison process between the opposing forward motion vector and the backward motion vector,
| Vxft1 + Vxbt1 | <Vxth
｜ Vxft1 + Vxbt2 | <Vxth
｜ Vxft1 + Vxbt3 | <Vxth
｜ Vxft1 + Vxbt4 | <Vxth
Is true, and | Vyft1 + Vybt1 | <Vyth
| Vyft1 + Vybt2 | <Vyth
｜ Vyft1 + Vybt3 | <Vyth
｜ Vyft1 + Vybt4 | <Vyth
Is true, the forward motion vector and the backward motion vector that face each other are equal, that is, the forward motion vector starting from the end block 12-5 of the backward motion vector (Vxft1, Vyft1). It is determined that the end point coincides (or exists within a predetermined range) with the start point of the backward motion vector. Here, Vxth and Vyth are thresholds predetermined as a reference for determining whether or not the opposing forward motion vector and backward motion vector are equal, and can be changed by setting.

  That is, when the difference between the detected backward motion vector and the forward vector of the block around the end point indicated by the backward motion vector is equal to or less than a predetermined value (because the forward and backward vectors have opposite directions, It is ideal that the result of the addition is zero), and it is determined that the end point of the forward motion vector starting from the end block of the backward motion vector coincides with the start point of the backward motion vector.

  Here, the bidirectional vector comparison block is a 2 × 2 area, but depending on the calculation accuracy, a 3 × 3 area or a single block may be used.

  The bidirectional vector comparison unit 7 will be described below with reference to FIG. In the following, the comparison block is described as a single block for the sake of simplicity. The bidirectional vector comparison unit 7 uses the forward motion vector starting from the detection block at the end point of each backward motion vector as to the backward motion vector starting from the detection block 7-27 to 7-39 shown in FIG. Compare.

  For example, the end point of the backward motion vector 7-40 starting from the detection block 7-33 of the original image f (t) is the detection block 7-20 of the original image f (t-1). The unit 7 compares the forward motion vector 7-41 starting from the detection block 7-20 with the backward motion vector 7-40. In this case, the bidirectional vector comparison unit 7 is both a zero vector for the backward motion vector 7-40 and the forward motion vector 7-41, and the starting point of the backward motion vector and the backward motion vector It is determined that the end point of the forward motion vector starting from the end point is equal.

  For example, the end point of the backward motion vector 7-42 starting from the detection block 7-37 of the original image f (t) is the detection block 7-20 of the original image f (t-1), and is bidirectional. The vector comparison unit 7 compares the forward motion vector 7-41 starting from the detection block 7-20 with the backward motion vector 7-42. In this case, the bidirectional vector comparison unit 7 determines that the start point of the backward motion vector is different from the end point of the forward motion vector starting from the end point of the backward motion vector.

  The bidirectional vector comparison unit 7 performs the same determination process for all detection blocks of the original image f (t). In the example illustrated in FIG. 7, the bidirectional vector comparison unit 7 determines the backward motion vector start point and the backward motion vector end point for the backward motion vector starting from the detection blocks 7-34 to 7-37. It is determined that the end point of the forward motion vector as the start point is different.

  When the start point of the backward motion vector and the end point of the forward motion vector starting from the end point of the backward motion vector are the same, the detection block of the original image f (t−1) indicated by the backward motion vector; The detection block of the original image f (t) indicated by the forward motion vector is the same object region or the same background region.

  On the other hand, when the start point of the backward motion vector is different from the end point of the forward motion vector starting from the end point of the backward motion vector, the detection block of the original image f (t) serving as the start point of the backward motion vector. This means that the object or background area displayed in is not displayed in the original image f (t−1).

  That is, the backward motion vector starting from the detected block in the original image f (t), and the forward motion vector starting from the detected block in the original image f (t−1) serving as the end point of the backward motion vector Are not displayed in the original image f (t−1) at the previous time in time by determining whether or not the backward motion vector and the forward motion vector are opposite in direction and equal in magnitude. The “appearing region” displayed in the original image f (t) at a later time in time can be detected. In the example illustrated in FIG. 7, the detection blocks 7-34 to 7-37 in the original image f (t) are identified as “appearing areas”.

  Next, motion vector correction in the “appearing region” will be described. In FIG. 7, the backward motion vectors detected in the detection blocks 7-34 to 7-37 of the original image f (t) that is an “appearing region” are erroneously detected motion vectors that indicate similar regions. Accordingly, when the backward motion vector is used for drawing the interpolated image, it is necessary to correct the backward motion vector that is erroneously detected in the “appearing region”.

  In the image interpolating apparatus 1 according to the present invention, when the motion vector in the “appearing region” of the original image f (t) is corrected, the motion vector correction unit 8 calculates the motion vector statistics of the original image f (t−1). Correction is performed based on the data and the statistical data of the motion vector of the original image f (t-2) at the time immediately before the original image f (t-1).

  More specifically, the motion vector correction unit 8 is a histogram of the forward motion vector in the detection block within a predetermined range centering on the detection block corresponding to the “appearing region” in the original image f (t−1). The data is compared with the histogram data of the forward motion vector in the detection block within a predetermined range centering on the detection block corresponding to the “appearing region” in the original image f (t−2). A vector in the reverse direction of the increased forward motion vector is determined as a motion vector in the “appearing region” of the original image f (t).

  That is, in the “appearing area”, the object moves away, and the background area increases with time. Therefore, the forward motion vector of the background area increases. Therefore, the motion vector in the “appearing region” is corrected to the inverse vector of the motion vector with the highest frequency in the histogram data of the motion vector detected last time and the forward motion vector detected this time.

  In the present embodiment, the motion vector correction unit 8 reads out the histogram data of the motion vectors in the original image f (t−2) and the original image f (t−1) from the state recording unit 5 and compares them. The statistical data processing unit 6 receives the motion vector data detected by the motion vector detection units 3 and 4 and the data specifying the “appearing region” determined by the bidirectional vector comparison unit 7, and receives the original image f In (t-1), the motion vector histogram data in a detection block within a predetermined range centering on the detection block corresponding to the “appearing region”, and the original image f (t−) in the original image f (t−2). The motion vector histogram data in the detection block within a predetermined range centering on the detection block corresponding to the “appearing area” of 1) is calculated and stored in the state recording unit 5. At this time, only the upper data of the histogram data may be stored in the state recording unit 5.

  In the example illustrated in FIG. 7, the motion vector correction unit 8, for example, in the original image f (t−1), an area (detection blocks 7-21 to 7-24) corresponding to the “appearing area” of the original image f (t). Histograms of motion vectors within a range of 5 blocks centering on each detection block included in), and an area (detection block 7) corresponding to the “appearing area” of the original image f (t) in the original image f (t−2). The detected blocks included in -8 to 7-11) are compared with a histogram of motion vectors within a range of 5 blocks centered on each detected block. In the example shown in FIG. 7, the appearance frequency is the highest in the range of 5 blocks centered on the detection block corresponding to the “appearing region” between the original image f (t−2) and the original image f (t−1). The increased motion vector is a zero vector, and the vector in the opposite direction is also a zero vector. Therefore, the motion vector correction unit 8 sets a zero vector as a motion vector in the “appearing region” (detection blocks 7-34 to 7-37) of the original image f (t).

  In the above description, in order to simplify the description, histogram data within a range of 5 blocks centering on the “appearing region” is calculated based on FIG. 7 in which the original image as shown in FIG. 4 is modeled. However, the range for calculating the histogram data is not limited to 5 blocks centering on the “appearing region”, and the number of blocks for calculating the histogram data is set according to the type of the original image and the like. It may be a configuration and is not particularly limited.

  Then, the motion vector correction unit 8 supplies the corrected motion vector to the vector allocation unit 9. At this time, the motion vector correction unit 8 uses an indicator that lowers the priority in the allocation process to the interpolated image with respect to the motion vector of the “appearing region”, that is, the interpolation vector when contention occurs in the interpolation block. Is supplied to the vector allocating unit 9. The vector allocation unit 9 performs allocation processing to the interpolation block based on the priority instruction information from the motion vector correction unit 8 and supplies the motion vector allocated to the interpolated image, that is, the interpolation vector, to the drawing unit 10. To do.

  In addition, the motion vector correction unit 8 vectorizes the drawing instruction information for instructing the drawing of the motion vector of the “appearing region” from the future image, that is, the original image f (t) at a time later than the interpolation time. Supply to the allocation unit 9. Then, when the motion vector of the “appearing region” is assigned to the interpolated image, the vector assigning unit 9 supplies the drawing instruction information together with the interpolation vector to the drawing unit 10. Then, the drawing unit 10 draws an interpolation image using the interpolation vector and the drawing instruction information from the vector allocation unit 9.

  FIG. 8 is a diagram for explaining detection of an “appearing region” and correction of a motion vector, and shows a state in which a motion vector is assigned to an interpolated image using the corrected motion vector. The detection blocks 8-1 to 8-6 in the original image f (t-1) in FIG. 8 correspond to the detection blocks 7-20 to 7-24 in the original image f (t-1) in FIG. As described above, zero vectors are set in the “appearing regions” (detection blocks 7-34 to 7-37, that is, detection blocks 8-11 to 8-14) of the original image f (t−1).

  FIG. 8 shows the interpolated images f (t-0.75), f (t-0.5), and f (t−t−) at each time of t−0.75, t−0.5, and t−0.25. 0.25) is shown as an example of interpolation. As shown in FIG. 8, in the interpolation block 8-9 of the interpolated image f (t-0.75), no motion vector conflict has occurred, and the vector allocating unit 9 performs the processing for the original image f (t). A motion vector in the detection block 8-12 which is an “appearing region” is assigned as an interpolation vector. In addition, in the interpolation block 8-8 of the interpolated image f (t-0.5), no motion vector conflict has occurred. Therefore, the vector allocating unit 9 displays the “appearing region of the original image f (t). The motion vector in the detection block 8-12 is assigned as an interpolation vector. On the other hand, motion vector contention occurs in the interpolation block 8-7 of the interpolated image f (t-0.25).

  In this case, as described above, the vector allocation unit 9 instructs the motion vector correction unit 8 not to select the motion vector of the “appearing region” as the interpolation vector when competition occurs in the interpolation block. Receives priority indication information. Therefore, in the interpolation block 8-7, the motion vector in the detection block 8-12 that is an “appearing region” is not selected as the interpolation vector, and the motion vector detected in the detection block 8-2 is used as the interpolation vector. select.

  A general formula for assigning a vector (Vx, Vy) from coordinates (x, y) to a frame at a position 1 / k in time between f (t) and f (t + 1) is (x + 1 / k × Vx, y + 1 / k × Vy), but the “appearing area” is allocated as (x− (1-1 / k) × Vx, y− (1-1 / k) × Vy).

  Further, as described above, the drawing unit 10 receives drawing instruction information for drawing from a future image with respect to the motion vector of the “appearing region”. Therefore, for example, the drawing unit 10 inserts the interpolation block 8-8 and the interpolation image f (t-0.75) of the interpolation image f (t-0.5) to which the motion vector of the “appearing region” is assigned. The interpolation block 8-9 is drawn using the image of the detection block 8-12 of the original image f (t).

  Thereby, in the image interpolation device 1 according to the present invention, a motion vector erroneously detected in the “appearing region” can be corrected to a motion vector with higher reliability. Therefore, when an “appearing region” is included in the original image, the accuracy of the interpolated image can be improved.

(Modification)
The image interpolation apparatus 1 according to the present invention may be configured to use a previously detected motion vector as a candidate vector for estimation of a newly detected motion vector when a new motion vector is detected. For example, when detecting a motion vector between the original images f (t−1) and f (t), the previous detection result, that is, the original images f (t−2) and f (t−1) The motion vector detected between them is used as a candidate vector.

  That is, the forward motion vector starting from the original image f (t-2) and ending at the original image f (t-1) is set as the forward motion vector, and the original image f (t-1) is set as the starting point and the original image f (t) is set as the end point. Can be used as a candidate vector of the forward motion vector, and a backward motion vector starting from the original image f (t-1) and ending at the original image f (t-2) is used as the original image f (t). Can be used as a candidate vector of a backward motion vector starting from the original image f (t-1).

  Here, in the image interpolation device 1, when detecting a forward motion vector to the future image f (t) with reference to the past image f (t-1), the original image f (t-2) to f (t- Of the forward motion vectors detected for 1), a motion vector starting from a detection block included in the “disappearing region” is prohibited from being referred to as a candidate vector. For the motion vector of the detection block included in the “appearing region”, the convergence of the vector is accelerated by actively searching for new candidates such as changing the weighting or increasing the number of candidate vectors.

  Further, in the image interpolation device 1, when detecting a backward motion vector to the past image f (t−1) with the future image f (t) as a reference, the original image f (t−1) to f (t−2) is detected. Among the backward motion vectors detected for), the motion vector starting from the detection block included in the “appearing region” is prohibited from being referred to as a candidate vector. Alternatively, in the histogram data of the previous detection (detection between f (t-2) and f (t-1)) and the current detection (detection between f (t-1) and f (t)) A configuration may be employed in which convergence is improved and erroneous detection is prevented by substituting a motion vector having increased frequency as a candidate vector.

  In addition, the image interpolation device 1 according to the present invention may be configured, for example, in a television receiver to interpolate an image to be displayed based on a video signal included in a received broadcast signal. Alternatively, the image interpolation device 1 according to the present invention is provided in a video playback device such as a DVD playback device or a video playback device, and interpolates an image to be played back based on video data recorded on a recording medium. There may be.

  The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

  Finally, each block of the image interpolation apparatus 1 may be configured by hardware logic, or may be realized by software using a CPU as follows.

  That is, the image interpolation apparatus 1 includes a CPU (central processing unit) that executes instructions of a control program that realizes each function, a ROM (read only memory) that stores the program, and a RAM (random access memory) that expands the program. And a storage device (recording medium) such as a memory for storing the program and various data. An object of the present invention is to provide a recording medium on which a program code (execution format program, intermediate code program, source program) of a control program of the image interpolation apparatus 1 which is software for realizing the above-described functions is recorded so as to be readable by a computer. This can also be achieved by supplying the image interpolation apparatus 1 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

  Examples of the recording medium include a tape system such as a magnetic tape and a cassette tape, a magnetic disk such as a floppy (registered trademark) disk / hard disk, and an optical disk such as a CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

  Further, the image interpolation apparatus 1 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Also, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The image interpolating apparatus according to the present invention can generate an interpolated image from an original image included in a video signal with high accuracy, and thus can be suitably used in a television receiver, a DVD reproducing apparatus, and the like.

It is a block diagram which shows the structure of the image interpolation apparatus 1 which concerns on this invention. It is a figure which shows the original image of the image | video which a small object moves, (a) is a figure which shows the past original image f (t-1) temporally with respect to an interpolation image, (b), It is a figure which shows the future original image f (t) temporally with respect to an interpolation image. It is a figure for demonstrating the drawing process of an interpolation image, Comprising: The original image and interpolation image which were shown to Fig.2 (a) and (b) were taken, and the cross section of the moving direction of the small object A was taken and displayed one-dimensionally FIG. It is a figure which shows a mode that an object moves against the background of the several column of the same shape arranged in equal intervals, (a) is the original image f (t-1) in the time t-1, and the original image f in the time t It is a figure which shows (t), (b) is a figure which shows the interpolation image produced | generated from the original image shown to (a). It is a figure for demonstrating the detection of a "disappearing area | region", and correction | amendment of a motion vector, Comprising: It is a figure which shows the bidirectional | two-way motion vector detected in the original images f (t-1) and f (t). It is a figure for demonstrating the detection of a "disappearing area | region", and correction | amendment of a motion vector, Comprising: It is a figure which shows a mode that a motion vector is allocated to an interpolation image using the corrected motion vector. It is a figure for demonstrating the detection of an "appearing area | region", and correction | amendment of a motion vector, Comprising: It is a figure which shows the bidirectional | two-way motion vector detected in the original images f (t-1) and f (t). It is a figure for demonstrating detection of an "appearing area | region", and correction | amendment of a motion vector, Comprising: It is a figure which shows a mode that a motion vector is allocated to an interpolation image using the motion vector after correction | amendment. It is a figure which shows the relationship between the 60Hz original image and 120Hz output image which are input. It is a figure which shows the example which divides | segments a reference | standard image into a block, in order to calculate a motion vector. It is a figure which shows the example of allocation of the vector to an interpolation coordinate. It is a figure for demonstrating the process in a bidirectional vector comparison part.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image interpolation apparatus 2 Delay part 3 Motion vector detection part (backward motion vector detection means)
4. Motion vector detection unit (forward motion vector detection means)
5 Status recording unit 6 Statistical data processing unit 7 Bidirectional vector comparison unit (determination means)
8 Motion vector correction unit (predicted motion vector determination means, correction means)
9 Vector allocation unit 10 Drawing unit 100 Motion vector correction device

Claims (12)

A forward motion vector starting from each block of the first original image at the first time and ending at any block of the second original image at the second time after the first time Forward motion vector detecting means for detecting;
A backward motion vector detecting means for detecting a backward motion vector starting from each block of the second original image and ending at any block of the first original image;
When the interval between the start point of the forward motion vector and the end point of the backward motion vector starting from the end point of the forward motion vector is larger than a predetermined threshold, the start point of the forward vector is determined as the correction target block Determination means to perform,
In the third original image at a third time before the first time, the appearance frequency of the forward motion vector starting from a block within a predetermined range from the correction target block, and the first original image Prediction motion in which the forward motion vector having the lowest appearance frequency is determined as a predicted motion vector by comparing with the appearance frequency of the forward motion vector starting from a block within the predetermined range from the correction target block in the image Vector determination means;
A motion vector correction apparatus comprising: correction means for replacing a forward motion vector starting from the correction target block with the predicted motion vector. A forward motion vector starting from each block of the first original image at the first time and ending at any block of the second original image at the second time after the first time Forward motion vector detecting means for detecting;
A backward motion vector detecting means for detecting a backward motion vector starting from each block of the second original image and ending at any block of the first original image;
If the interval between the start point of the backward motion vector and the end point of the forward motion vector starting from the end point of the backward motion vector is larger than a predetermined threshold, the start point of the backward vector is determined as the correction target block. Determination means to perform,
In the third original image at a third time before the first time, the appearance frequency of the forward motion vector starting from a block within a predetermined range from the correction target block, and the first original image Compare the appearance frequency of the forward motion vector starting from a block within the predetermined range from the correction target block in the image, and determine the reverse vector of the forward motion vector with the highest appearance frequency as the predicted motion vector Predictive motion vector determining means to perform,
A motion vector correction apparatus comprising: correction means for replacing a backward motion vector starting from the correction target block with the predicted motion vector. The determination means is
Starting from each block of the third original image, the forward motion vector starting from any block of the first original image and each block of the first original image as the starting point, the third The correction target block in the third original image is determined from the backward motion vector whose end point is any block of the original image of
The forward motion vector detection means includes
2. The forward motion vector other than the forward motion vector starting from the correction target block in the third original image is used as a candidate vector for estimating the forward motion vector in the first original image. Or the motion vector correction apparatus according to 2; The determination means is
Starting from each block of the third original image, the forward motion vector starting from any block of the first original image and each block of the first original image as the starting point, the third Determining the correction target block in the first original image from a backward motion vector whose end point is any block of the original image of
The backward motion vector detecting means is
2. A vector other than a backward motion vector starting from the correction target block in the first original image is used as a candidate vector for estimating a backward motion vector in the second original image. Or the motion vector correction apparatus according to 2; An image interpolation apparatus comprising the motion vector correction apparatus according to any one of claims 1 to 4,
Using the predicted motion vector starting from the correction target block and the forward or backward motion vector starting from a block other than the correction target block, the first original image and the second original image are used. An interpolation vector determining means for determining an interpolation vector to be assigned to each block of the interpolated image to be interpolated between the images;
An image interpolation device comprising: an interpolation image generation unit configured to generate the interpolation image based on the interpolation vector determined by the interpolation vector determination unit. The interpolation vector determination means is
When both the predicted motion vector starting from the correction target block and the forward motion vector or the backward motion vector starting from a block other than the correction target block are allocated to the same block of the interpolated image, 6. The image interpolation apparatus according to claim 5, wherein a forward motion vector or a backward motion vector starting from a block other than the correction target block is determined as an interpolation vector to be assigned to the block.   A television receiver comprising the image interpolation device according to any one of claims 1 to 6.   A video reproduction device comprising the image interpolation device according to any one of claims 1 to 6. A forward motion vector starting from each block of the first original image at the first time and ending at any block of the second original image at the second time after the first time A forward motion vector detection step to detect;
A backward motion vector detecting step of detecting a backward motion vector starting from each block of the second original image and ending at any block of the first original image;
When the interval between the start point of the forward motion vector and the end point of the backward motion vector starting from the end point of the forward motion vector is larger than a predetermined threshold, the start point of the forward vector is determined as the correction target block A determination step to:
In the third original image at a third time before the first time, the appearance frequency of the forward motion vector starting from a block within a predetermined range from the correction target block, and the first original image Prediction motion in which the forward motion vector having the lowest appearance frequency is determined as a predicted motion vector by comparing with the appearance frequency of the forward motion vector starting from a block within the predetermined range from the correction target block in the image A vector determination step;
A motion vector correction method comprising: a correction step of replacing a forward motion vector starting from the correction target block with the predicted motion vector. A forward motion vector starting from each block of the first original image at the first time and ending at any block of the second original image at the second time after the first time A forward motion vector detection step to detect;
A backward motion vector detecting step of detecting a backward motion vector starting from each block of the second original image and ending at any block of the first original image;
If the interval between the start point of the backward motion vector and the end point of the forward motion vector starting from the end point of the backward motion vector is larger than a predetermined threshold, the start point of the backward vector is determined as the correction target block. A determination step to:
In the third original image at a third time before the first time, the appearance frequency of the forward motion vector starting from a block within a predetermined range from the correction target block, and the first original image Compare the appearance frequency of the forward motion vector starting from a block within the predetermined range from the correction target block in the image, and determine the reverse vector of the forward motion vector with the highest appearance frequency as the predicted motion vector A prediction motion vector determination step to be performed;
A motion vector correction method, comprising: a correction step of replacing a backward motion vector starting from the correction target block with the predicted motion vector.   A control program for operating the motion vector correction apparatus according to any one of claims 1 to 4, wherein the computer functions as each of the above-described means.   The computer-readable recording medium which has recorded the control program of Claim 11.