JP2010178216A - Vector correction device, vector correction method, image interpolation device, television receiver, video reproducer, control program, and computer readable recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a motion vector device capable of determining a highly reliable motion vector. <P>SOLUTION: In this vector correction device 100, a motion vector detecting section 3 detects a motion vector from original images f(t) and f(t+1). A statistics processing section 8 analogizes an analogical vector to be allocated to an undetermined block of an interposed image to which the detected motion vector is not allocated from the appearance frequency of motion vectors from an original image f(t-1) to the original image f(t) and the appearance frequency of motion vectors from an original image (t+2) to the original image f(t+1) in accordance with a determination result of a plotting direction in a plotting direction determining section 4. A vector allocating section 6 allocates the detected motion vector and the analogical vector to each block of the interpolated image. A vector correcting section 7 corrects an interpolation vector by replacing the analogical vector allocated to the undetermined block with the closest detected motion vector among detected motion vectors allocated to peripheral blocks. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a vector correction apparatus and vector correction method for correcting a vector assigned to each block of an interpolated image, and in particular, an image interpolation apparatus, a television receiver, a video reproduction apparatus, a control program, and a computer-readable program. The present invention relates to a simple recording medium.

  When changing the number of frames of the original image material, such as when copying a 24 Hz film material to a 60 Hz or 120 Hz display device, or when copying a 60 Hz progressive signal generated by IP conversion or the like to a 120 Hz display device A technique is generally known in which a motion vector is obtained from a plurality of consecutive frames, and the number of frames of an image signal is converted by an interpolated image generated based on the motion vector.

  FIG. 11 is a diagram showing a relationship between an input 60 Hz original image and a 120 Hz output image. For example, when a 60 Hz progressive signal is copied to a 120 Hz display device, the original image 11-1 at time t-1 and the original image 11-2 at time t are used, and the time is exactly half (intermediate). The frame rate is increased by creating and outputting the interpolated image 11-3.

  Here, a general process for generating one interpolated image from two original images will be described. First, an original image serving as a reference for obtaining a motion vector is determined from the two images. Assuming that an image at time t in FIG. 11 is f (t), either f (t−1) or f (t) is generated when an interpolated image is generated from f (t−1) and f (t). Is the reference original image. Hereinafter, a case where f (t−1) is a reference image will be described.

  FIG. 12 is a diagram illustrating an example in which a reference image is divided into blocks in order to calculate a vector. As shown in FIG. 12, the reference image f (t−1) is divided into a plurality of blocks for calculating a motion vector. A portion surrounded by a thin line like a portion 12-1 in FIG. 12 means a pixel of the original image. In this example, a minimum block 12-2 is defined for each area of 8 pixels × 8 pixels.

  Next, a motion vector between f (t−1) and f (t) is obtained for each defined minimum block. As a method for obtaining the motion vector, a general method such as block matching or a gradient method is used.

  In this way, in the reference original image, a motion vector that seems to have the highest reliability is detected for each minimum block. As a process that is normally performed, filtering such as a median filter is performed on the motion vector determined in this way to prevent erroneous detection.

  Next, vector allocation processing is performed on each coordinate of the interpolated image using the motion vector obtained as described above. The unit of the assigned coordinates is arbitrary, but usually an area smaller than the block unit of the reference image is designated. In the example shown in FIG. 12, since the reference original image is defined by 8 pixels × 8 pixels, a vector is assigned to the interpolation coordinates for each small region such as 4 pixels × 2 pixels.

  The assignment of vectors to interpolated coordinates will be described with reference to FIG. 13 as a one-dimensional process in the horizontal direction for simplification. FIG. 13 is a diagram illustrating an example of assigning vectors to interpolation coordinates. If the motion vector Vx indicated by the arrow 13-1 is obtained for the coordinate x of the reference original image f (t-1), the interpolated image is created in the middle of time t-1 and time t. , Vx, the interpolated coordinate N (x) is N (x) = x + Vx / 2.

  At this time, the vector Vx is assigned to the small block of the interpolation coordinates including N (x). By executing this processing for all the blocks of the reference original image, vectors are allocated on the small blocks of the interpolation coordinates. As a result, for each small block of interpolated coordinates, there are places where multiple vectors are assigned or not assigned at all, but usually when multiple assignments are made, of the assigned vectors, This is represented by the highest correlation between f (t-1) and f (t) calculated using the specific region. If no assignment is made, a zero vector is used instead. As a well-known process, the assigned vector is subjected to a filtering process such as a median filter to obtain a final vector.

  Finally, using the finally obtained allocation vector on the interpolated coordinates, calculate the coordinates of f (t-1) or f (t) and generate an interpolated image using one or both images To do.

  The above is a process for creating a general interpolated image, but a process that introduces a bidirectional vector has been proposed in order to further improve the reliability of the motion vector and reduce artifacts in the interpolated image. In FIG. 13, the bidirectional vector refers to the forward motion vector with reference to the image f (t) with f (t−1) as the reference image in FIG. 13, and f (t−1) with f (t) as the reference image. This refers to a vector in both directions with a backward motion vector referring to an image.

  By the way, when an interpolated image is generated and an image is interpolated, “disappearing area” and “appearing area” become a problem. For example, when an object moves between the original image f (t-1) and the original image f (t), the object is not hidden by f (t-1) due to the movement of the object, but f (t) Then, the background area hidden behind the object is called “disappearing area”, and the background area hidden behind the object at f (t−1) but not hidden at f (t) is called “appearing area”. Call. Therefore, “disappearing area” information (background area image data) exists only in f (t−1), and “appearing area” information (background area image data) exists only in f (t). Here, in the gradient method and block matching described above, there is no solution that there is no motion vector, and the solution in the calculation formula (gradient method) or the region with the smallest pixel correlation value (ie, the region with high pixel correlation) is used. It is a technique that leads as a solution.

  Therefore, when a motion vector is obtained from f (t−1) to f (t), the “disappearing region” has no corresponding point, and the motion vector is indicated toward any similar point in the vicinity, and f (t) When the motion vector is obtained from f (t−1), the “appearing region” has no corresponding point, and the vector is shown toward any similar point in the vicinity.

  When creating an interpolated image based on the motion vector thus determined, if an interpolated image is created using either the original image f (t-1) or the original image f (t), In the inset image, the object is projected in an area corresponding to the “disappearing area” or the “appearing area”. In addition, when creating an interpolated image using both the original image f (t−1) and the original image f (t), the object is projected in both the “disappearing area” and the “appearing area”. There arises a problem that the accuracy of the generated interpolated image is lowered.

  On the other hand, configurations for interpolating images in consideration of “disappearing regions” and “appearing regions” are disclosed in, for example, Patent Documents 1 to 3.

JP-A-8-9339 (January 12, 1996) JP 7-336688 A (December 22, 1995) JP-A-9-214899 (August 15, 1997)

  However, although the conventional configuration described above is an image interpolation method considering “disappearing area” and “appearing area”, there is a problem that an interpolated image cannot be generated with high accuracy.

  For example, the original image temporally before is f (t), the original image temporally after is f (t + 1), the original image immediately before f (t) is f (t−1), Let f (t + 2) be the original image immediately after f (t + 1). In the configurations described in Patent Document 1 and Patent Document 3, when an interpolation is generated between f (t) and f (t + 1), f (t−1) is assigned to a vector assigned to “disappearing region” and “appearing region”. And a motion vector obtained between f (t + 1) and f (t + 2). This is based on the premise that the object and the background area move at a constant speed. In an actual video, there are few cases where the object is moving at a uniform speed, and the background area relatively accelerates / decelerates when the camera follows the object that is accelerating / decelerating.

  FIG. 4 is a diagram for explaining a state in which the amount of motion of the background changes. The original image f (t−1) at time t−1, the original image f (t) at time t, and the original image at time t + 1. It is a figure which shows an example of the image f (t + 1), the original image f (t + 2) in the time t + 2, and the original image f (t + 3) in the time t + 3.

  In FIG. 4, a portion surrounded by a square denoted by reference numeral 4-1 indicates one pixel group that performs motion vector detection. A pixel group in which alphabets are written is set as a background area, and a portion indicated by hatched lines 4-2 is set as an object area. An arrow 4-3 indicates an object motion vector, and a reference 4-4 indicates a background motion vector. Now, as an image, a case where an object accelerates in the background as indicated by the alphabet will be described as an example. It is assumed that the camera follows the object so as to capture the object as the center. It is assumed that the acceleration speed is 2 blocks / frame every other frame.

  In the original image f (t-1) and the original image f (t), the accelerating object is imaged stationary, and the background is relatively moving leftward at a speed of 2 blocks / frame. In the original image f (t) and the original image f (t + 1), the accelerating object is imaged stationary, and the background relatively moves at a speed of 4 blocks / frame. In the original image f (t + 1) and the original image f (t + 2), the accelerating object is imaged stationary, and the background is relatively moving at a speed of 6 blocks / frame. In the original image f (t + 2) and the original image f (t + 3), the accelerating object is imaged stationary, and the background moves relatively at a speed of 8 blocks / frame.

  A case where an interpolated image at a half position of two original images in terms of time is created using this frame will be described with reference to FIG. FIG. 5 is a diagram for explaining a problem that occurs when an interpolated image is created from the original image of FIG. 4, and shows a state when the interpolated image is created at an intermediate point in time of the original image. . In FIG. 5, the interpolated image f (t−0.5) at time t−0.5, the interpolated image f (t + 0.5) at time t + 0.5, and the interpolated image f (t + 1. 5) and the original image f (t + 2.5) at time t + 2.5.

  In FIG. 5, reference numeral 5-1 indicates one pixel group that performs motion vector detection. The pixel group in which the alphabet is written is the background and corresponds to the pixel group 4-1 shown in FIG. Further, the portion indicated by the hatched line of reference numeral 5-2 represents an object and corresponds to a region 4-2 in FIG.

  An arrow indicated by a solid line of reference numeral 5-3A indicates that the interpolated image f (t-0.5) of the object part refers to the original image f (t-1) in the past. An arrow indicated by a solid line 5-3B indicates that the interpolation image f (t-0.5) of the object part refers to the original image f (t) of the future in time. Arrows 5-3A and 5-3B correspond to the motion vector 4-3 in FIG. An arrow indicated by a solid line 5-4A indicates that the interpolated image f (t-0.5) in the background portion refers to the past original image f (t-1) in terms of time. An arrow indicated by a solid line 5-4B indicates that the interpolated image f (t-0.5) in the background portion refers to the original image f (t) of the future in time. Arrows 5-4A and 5-4B correspond to the motion vector 4-4 in FIG. An arrow indicated by a dotted line 5-11 is a “disappearing area” of the interpolation, and indicates that the drawing is performed only from the past original image f (t−1) in terms of time.

  Here, a description will be given focusing on the “disappearing area” of the background hidden by the object. First, between the original image f (t) and the original image f (t + 1), the object and background 'G' are blended in both directions to the interpolated image f (t + 0.5). The space between the object and ‘G’ is recognized as the “disappearing region”, and the image is to be drawn only from the past. At this time, according to Patent Documents 1 to 3, the vector assigned to the “disappearing region” is the background motion vector detected last time.

  Since the background is accelerating at 2 blocks / frame, the background moves 2 blocks between the original image f (t-1) and the original image f (t). Therefore, 'D' and 'E' are displayed in the interpolated image f (t + 0.5). In the interpolated image f (t + 0.5), the background is “DEG”, and discontinuous points are generated between “EG”.

  Further, between the original image f (t + 1) and the original image f (t + 2), the pixel group of “M” in the object region and the background region is blended into the interpolated image f (t + 1.5) from both directions. The area between the object and 'M' is recognized as the “disappearing area”, and the image is to be drawn only from the past. At this time, according to Patent Documents 1 to 3, the vector assigned to the “disappearing region” is the motion vector of the background detected last time.

  Since the background is accelerating at 2 blocks / frame, the background motion between the original image f (t) and the original image f (t + 1) is 4. Therefore, “I”, “J”, and “K” are displayed in the interpolated image f (t + 1.5). In the interpolated image f (t + 1.5), the background is “IJKM”, and discontinuous points occur between “KM”. Between the original image f (t + 2) and the original image f (t + 3), 'U' of the object and the background is blended from both directions into the interpolated image f (t + 2.5), but the space between the object and 'U' disappears. It is recognized as “region” and tries to draw from the past. At this time, according to Patent Documents 1 to 3, the vector assigned to the “disappearing region” is the motion vector of the background detected last time.

  Since the background accelerates at 2 blocks / frame, the background between the original image f (t + 1) and the original image f (t + 2) moves by 6 blocks. Therefore, “P”, “Q”, “R”, and “S” are displayed in the interpolated image f (t + 2.5). In the interpolated image f (t + 2.5), the background becomes “PQRSU”, and discontinuous points occur between “SU”. Up to this point, the “disappearing region” has been described, but the same phenomenon occurs even though the description of the “appearing region” is omitted.

  As described above, when Patent Documents 1 to 3 are followed, a discontinuous point occurs along the contour of the object on the interpolation due to a subtle change in motion between frames, and a good quality interpolation cannot be obtained. Will occur.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to determine a highly reliable motion vector in order to prevent the interpolation image from being broken and to accurately generate the interpolation image. An object of the present invention is to provide a vector correction device, a vector correction method, an image interpolation device, a television receiver, a video reproduction device, a control program, and a computer-readable recording medium.

  The vector correction apparatus according to the present invention interpolates between a first original image corresponding to a first time and a second original image corresponding to a second time after the first time. A vector correction device that corrects a vector assigned to each block of an interpolated image to be executed, starting from each block of the first original image from the first original image and the second original image, Detecting means for detecting a motion vector whose end point is one of the blocks of the second original image; and for each block of the interpolated image, drawing from only the first original image, The drawing direction determining means for determining whether to draw only from the original image or from both the first original image and the second original image, and depending on the determination result of the drawing direction determining means, Detected by the detection means in the block of the inset image An analogy vector for allocating to the undetermined block to which the assigned motion vector is not allocated is a block within a predetermined range from the undetermined block in the third original image at the third time before the first time. In the block of the interpolated image according to the first analogizing means that infers from the appearance frequency of the motion vector whose starting point is one of the blocks of the first original image and the drawing direction determining means. An analog vector for allocating to an undetermined block to which a motion vector detected by the detecting means is not allocated is a predetermined range from the undetermined block in a fourth original image at a fourth time after the second time. Analogize from the appearance frequency of motion vectors starting from a block inside and ending at any block of the second original image Allocation means for allocating the motion vector detected by the second analogy means, the motion vector detected by the detection means, and the analogy vector estimated by the first analogy means and the second analogy means to each block of the interpolated image; Correction means for replacing the analogy vector assigned to the undetermined block with a motion vector closest to the analogy vector among the motion vectors assigned to blocks within a predetermined range from the undetermined block. It is a feature.

  According to said structure, the vector correction apparatus which concerns on this invention is the 1st original image corresponding to 1st time, and the 2nd original image corresponding to 2nd time after 1st time The interpolation vector assigned to each block of the interpolated image to be interpolated between is corrected.

  The detecting means is a motion vector starting from each block of the first original image and ending at any block of the second original image from the first original image and the second original image. Is detected.

  The drawing direction determining means draws each block of the interpolated image from only the first original image, draws only from the second original image, or draws the first original image and the first image. Whether to draw from both of the two original images is determined. The drawing direction determination means determines the drawing direction using a technique such as that disclosed in Patent Documents 1 to 3.

  The first analogizing means is an undetermined block to which the motion vector detected by the detecting means is not assigned in the block of the interpolated image, and the drawing direction determining means determines that the drawing is performed using only the first original image. As a vector to be assigned to the undetermined block, any one of the first original images starting from a block within a predetermined range from the undetermined block in the third original image at the third time before the first time. A vector having the highest appearance frequency is selected from motion vectors having the block as an end point.

  The second analogizing means is an undetermined block to which the motion vector detected by the detecting means is not assigned in the block of the interpolated image, and the drawing direction determining means determines that the drawing is performed using only the second original image. As a vector to be assigned to the undetermined block, any of the second original images starting from a block within a predetermined range from the undetermined block in the fourth original image at a fourth time after the second time is used. A vector having the highest appearance frequency is selected from motion vectors having the block as an end point.

  The assigning means assigns the motion vector detected by the detecting means and the vector estimated by the first analogizing means and the second analogizing means to each block of the interpolated image.

  Then, the correcting means converts the interpolation vector assigned to the undetermined block to the interpolation vector assigned to a block other than the undetermined block within the predetermined range from the undetermined block (that is, the periphery of the block to which the analogy vector is assigned). Among the interpolation vectors assigned to the block, the nearest interpolation vector of the interpolation vectors detected by the detection means) is replaced. The correction means takes the difference between the analogy vector and the interpolation vector assigned to the surrounding blocks, and replaces the analogy vector with a vector having the smallest difference from the analogy vector.

  Thereby, the vector correction apparatus according to the present invention calculates the motion vector between the third original image and the first original image and the motion vector between the second original image and the fourth original image, It is not used directly as an interpolation vector. That is, no analogy vector is used as the interpolation vector. Therefore, even if a change in motion occurs between frames, a motion vector detected between the first original image and the second original image is used as an interpolation vector, so that discontinuities occur in the background. Therefore, it is possible to obtain a good quality interpolated image.

  The vector correction method according to the present invention interpolates between a first original image corresponding to a first time and a second original image corresponding to a second time after the first time. A vector correction method for correcting a vector assigned to each block of an interpolated image to be executed, starting from each block of the first original image from the first original image and the second original image, A detection step of detecting a motion vector whose end point is one of the blocks of the second original image, and each block of the interpolated image is drawn from only the first original image, or the second According to the drawing direction determination step for determining whether to draw only from the original image or from both the first original image and the second original image, and according to the determination result of the drawing direction determination step, In the block of the inset image, the above detection scan An analogy vector for allocating to the undetermined block to which the motion vector detected by the mapping is not allocated is within a predetermined range from the undetermined block in the third original image at the third time before the first time. A first analogizing step that estimates from the appearance frequency of a motion vector having a block at the start point as a start point and any block of the first original image as an end point, and the interpolation according to the determination result of the drawing direction determination step In the fourth original image at the fourth time after the second time, the analog block for allocating the motion vector detected by the detection step in the image block to the undetermined block is not determined. Motion starting from a block within a predetermined range from and starting from one of the blocks of the second original image A second analogy step inferred from the appearance frequency of the kuttle, the motion vector detected in the detection step, and the analogical vector inferred by the first analogy step and the second analogy step are respectively represented in the interpolated image. An assigning step for assigning to a block, and a correction step for replacing the analogy vector assigned to the undetermined block with a motion vector closest to the analogy vector among the motion vectors assigned to blocks within a predetermined range from the undetermined block. It is characterized by including.

  According to said structure, there exists an effect similar to the vector correction apparatus which concerns on this invention.

  The vector correction apparatus according to the present invention interpolates between a first original image corresponding to a first time and a second original image corresponding to a second time after the first time. A vector correction apparatus that corrects a vector assigned to each block of an interpolated image to be executed, starting from each block of the second original image from the first original image and the second original image, Detection means for detecting a motion vector whose end point is one of the blocks of the first original image, and each block of the interpolated image is drawn only from the first original image, or the second The drawing direction determining means for determining whether to draw only from the original image or from both the first original image and the second original image, and depending on the determination result of the drawing direction determining means, Detected by the detection means in the block of the inset image An analogy vector for allocating to the undetermined block to which the assigned motion vector is not allocated is a block within a predetermined range from the undetermined block in the third original image at the third time before the first time. In the block of the interpolated image according to the first analogizing means that infers from the appearance frequency of the motion vector whose starting point is one of the blocks of the first original image and the drawing direction determining means. An analog vector for allocating to an undetermined block to which a motion vector detected by the detecting means is not allocated is a predetermined range from the undetermined block in a fourth original image at a fourth time after the second time. Analogize from the appearance frequency of motion vectors starting from a block inside and ending at any block of the second original image Allocation means for allocating the motion vector detected by the second analogy means, the motion vector detected by the detection means, and the analogy vector estimated by the first analogy means and the second analogy means to each block of the interpolated image; Correction means for replacing the analogy vector assigned to the undetermined block with a motion vector closest to the analogy vector among the motion vectors assigned to blocks within a predetermined range from the undetermined block. It is a feature.

  According to said structure, the vector correction apparatus which concerns on this invention is the 1st original image corresponding to 1st time, and the 2nd original image corresponding to 2nd time after 1st time The interpolation vector assigned to each block of the interpolated image to be interpolated between is corrected.

  The detection means is a motion vector starting from each block of the second original image and ending at any block of the first original image from the first original image and the second original image. Is detected.

  The drawing direction determining means draws each block of the interpolated image from only the first original image, draws only from the second original image, or draws the first original image and the first image. Whether to draw from both of the two original images is determined. The drawing direction determination means determines the drawing direction using a technique such as that disclosed in Patent Documents 1 to 3.

  The first analogizing means is an undetermined block to which the motion vector detected by the detecting means is not assigned in the block of the interpolated image, and the drawing direction determining means determines that the drawing is performed using only the first original image. As a vector to be assigned to the undetermined block, any one of the first original images starting from a block within a predetermined range from the undetermined block in the third original image at the third time before the first time. A vector having the highest appearance frequency is selected from motion vectors having the block as an end point.

  The second analogizing means is an undetermined block to which the motion vector detected by the detecting means is not assigned in the block of the interpolated image, and the drawing direction determining means determines that the drawing is performed using only the second original image. As a vector to be assigned to the undetermined block, any of the second original images starting from a block within a predetermined range from the undetermined block in the fourth original image at a fourth time after the second time is used. A vector having the highest appearance frequency is selected from motion vectors having the block as an end point.

  The assigning means assigns the motion vector detected by the detecting means and the vector estimated by the first analogizing means and the second analogizing means to each block of the interpolated image.

  Then, the correcting means converts the interpolation vector assigned to the undetermined block to the interpolation vector assigned to a block other than the undetermined block within the predetermined range from the undetermined block (that is, the periphery of the block to which the analogy vector is assigned). Among the interpolation vectors assigned to the block, the nearest interpolation vector of the interpolation vectors detected by the detection means) is replaced. The correction means takes the difference between the analogy vector and the interpolation vector assigned to the surrounding blocks, and replaces the analogy vector with a vector having the smallest difference from the analogy vector.

  Thereby, the vector correction apparatus according to the present invention calculates the motion vector between the third original image and the first original image and the motion vector between the second original image and the fourth original image, It is not used directly as an interpolation vector. That is, no analogy vector is used as the interpolation vector. Therefore, even if a change in motion occurs between frames, a motion vector detected between the first original image and the second original image is used as an interpolation vector, so that discontinuities occur in the background. Therefore, it is possible to obtain a good quality interpolated image.

  The vector correction method according to the present invention interpolates between a first original image corresponding to a first time and a second original image corresponding to a second time after the first time. A vector correction method for correcting a vector assigned to each block of an interpolated image, wherein each block of the second original image is set as a starting point from the first original image and the second original image. A detection step of detecting a motion vector whose end point is one of the blocks of the first original image, and each block of the interpolated image is drawn from only the first original image, or the second According to the drawing direction determination step for determining whether to draw only from the original image or from both the first original image and the second original image, and according to the determination result of the drawing direction determination step, In the block of the inset image, the above detection scan An analogy vector for allocating to the undetermined block to which the motion vector detected by the mapping is not allocated is within a predetermined range from the undetermined block in the third original image at the third time before the first time. A first analogizing step that estimates from the appearance frequency of a motion vector having a block at the start point as a start point and any block of the first original image as an end point, and the interpolation according to the determination result of the drawing direction determination step In the fourth original image at the fourth time after the second time, the analog block for allocating the motion vector detected by the detection step in the image block to the undetermined block is not determined. Motion starting from a block within a predetermined range from and starting from one of the blocks of the second original image A second analogy step inferred from the appearance frequency of the kuttle, the motion vector detected in the detection step, and the analogical vector inferred by the first analogy step and the second analogy step are each of the interpolated images. An assigning step for assigning to a block, and a correcting step for replacing the analogy vector assigned to the undetermined block with a motion vector closest to the analogy vector among the motion vectors assigned to blocks within a predetermined range from the undetermined block. It is characterized by including.

  According to said structure, there exists an effect similar to the vector correction apparatus which concerns on this invention.

  The vector correction apparatus according to the present invention interpolates between a first original image corresponding to a first time and a second original image corresponding to a second time after the first time. A vector correction device that corrects a vector assigned to each block of an interpolated image to be executed, starting from each block of the first original image from the first original image and the second original image, From the first detection means for detecting a forward motion vector whose end point is one of the blocks of the second original image, the first original image, and the second original image, the second original image Second detection means for detecting a backward motion vector starting from each block and ending at any block of the first original image; a starting point of the forward motion vector; and an end point of the forward motion vector The distance from the end point of the backward motion vector starting from A first determination unit that determines that the forward motion vector is not assigned to the block of the interpolated image when the threshold value is greater than a predetermined threshold; a starting point of the backward motion vector; and an end point of the backward motion vector as a starting point Second determination means for determining that the backward vector is not allocated to the block of the interpolated image when the interval from the end point of the forward motion vector to be performed is greater than a predetermined threshold, and each block of the interpolated image In contrast, it is determined whether to draw from only the first original image, to draw from only the second original image, or to draw from both the first original image and the second original image. The forward motion vector detected by the first detection means and the second detection means in the block of the interpolated image according to the determination result of the drawing direction determination means and the drawing direction determination means. Blocks that are within a predetermined range from the undetermined block in the third original image at a third time prior to the first time, the analogy vector for allocating to the undetermined block to which no backward motion vector is assigned And a block of the interpolated image according to the determination result of the drawing direction determination means, and first analogy means for estimating from the appearance frequency of a motion vector whose starting point is any block of the first original image In this case, an analogy vector for allocating to the undetermined block to which the forward motion vector and the backward motion vector detected by the first detection unit and the second detection unit are not allocated is set to a fourth time after the second time. Any one of the second original images starting from a block within a predetermined range from the undetermined block in the fourth original image at the time Second analogy means for inferring from the appearance frequency of the motion vector having the block as the end point, the motion vector detected by the detecting means, and the analogy vector estimated by the first analogy means and the second analogy means The assigning means for assigning each block of the interpolated image and the analogy vector assigned to the undetermined block are the most similar to the analogy vector among the motion vectors assigned to the blocks within a predetermined range from the undetermined block. It is characterized by comprising correction means for replacing with a near motion vector.

  According to said structure, the vector correction apparatus which concerns on this invention is the 1st original image corresponding to 1st time, and the 2nd original image corresponding to 2nd time after 1st time The interpolation vector assigned to each block of the interpolated image to be interpolated between is corrected.

  The first detection means starts each block of the first original image from the first original image and the second original image, and uses any block of the second original image as an end point. A forward motion vector is detected. Further, the second detection means starts each block of the second original image from the first original image and the second original image, and sets any block of the first original image as an end point. The backward motion vector is detected.

  When the interval between the start point of the forward motion vector and the end point of the backward motion vector starting from the end point of the forward motion vector is greater than a predetermined threshold, the first determination unit Is not assigned as the interpolation vector. Further, the second determination means, when the interval between the start point of the backward motion vector and the end point of the forward motion vector starting from the end point of the backward motion vector is larger than a predetermined threshold, Is not assigned as the interpolation vector.

  When the region in the first original image is also present in the second original image, the start point of the forward motion vector and the end point of the backward motion vector starting from the end block of the forward motion vector are substantially the same. On the other hand, if they do not substantially match, the region existing in the first original image does not exist in the second original image. That is, the first determination unit and the second determination unit detect “disappearing area” (that is, an area hidden behind a moving object or the like) in the first original image as the correction target block. In addition, as a predetermined threshold value, for example, the number of blocks may be preset according to the detection accuracy of the motion vector, and is not particularly limited.

  Further, the drawing direction determination means draws only the first original image, draws only the second original image, or draws the first original image for each block of the interpolated image. Whether to draw from both the second original image and the second original image is determined. The drawing direction determination means can be obtained by hit board technique or vector calculation as disclosed in Patent Documents 1 to 3. Also, an analogy vector may be allocated from the original image correction area and given to the interpolation coordinates.

  The first analogizing means is an undetermined block to which the motion vector detected by the detecting means is not assigned in the block of the interpolated image, and the drawing direction determining means determines that the drawing is performed using only the first original image. As a vector to be assigned to the undetermined block, any one of the first original images starting from a block within a predetermined range from the undetermined block in the third original image at the third time before the first time. A vector having the highest appearance frequency is selected from motion vectors having the block as an end point.

  The second analogizing means is an undetermined block to which the motion vector detected by the detecting means is not assigned in the block of the interpolated image, and the drawing direction determining means determines that the drawing is performed using only the second original image. As a vector to be assigned to the undetermined block, any of the second original images starting from a block within a predetermined range from the undetermined block in the fourth original image at a fourth time after the second time is used. A vector having the highest appearance frequency is selected from motion vectors having the block as an end point.

  For example, in the detection of the first original image and the third original image, the appearance frequency of the backward motion vector detection result from the first original image within the predetermined range centering on the undetermined block toward the third original image. The vector with the largest number is used as an analogy vector when drawing using only the first original image. In the detection of the second original image and the fourth original image, the appearance frequency is the highest in the forward motion vector detection result from the second original image within the predetermined range to the fourth original image with the undetermined block as the center. An analogy vector for drawing a large number of images using only the second original image is used.

  The assigning means assigns the motion vector detected by the detecting means and the vector estimated by the first analogizing means and the second analogizing means to each block of the interpolated image.

  Then, the correcting means converts the interpolation vector assigned to the undetermined block to the interpolation vector assigned to a block other than the undetermined block within the predetermined range from the undetermined block (that is, the periphery of the block to which the analogy vector is assigned). Among the interpolation vectors assigned to the block, the nearest interpolation vector of the interpolation vectors detected by the detection means) is replaced. The correction means takes the difference between the analogy vector and the interpolation vector assigned to the surrounding blocks, and replaces the analogy vector with a vector having the smallest difference from the analogy vector.

  Thereby, the vector correction apparatus according to the present invention calculates the motion vector between the third original image and the first original image and the motion vector between the second original image and the fourth original image, It is not used directly as an interpolation vector. That is, no analogy vector is used as the interpolation vector. Therefore, even if a change in motion occurs between frames, a motion vector detected between the first original image and the second original image is used as an interpolation vector, so that discontinuities occur in the background. Therefore, it is possible to obtain a good quality interpolated image.

  The vector correction method according to the present invention interpolates between a first original image corresponding to a first time and a second original image corresponding to a second time after the first time. A vector correction method for correcting a vector assigned to each block of an interpolated image to be executed, starting from each block of the first original image from the first original image and the second original image, From the first detection step of detecting a forward motion vector whose end point is one of the blocks of the second original image, and from the first original image and the second original image, A second detection step of detecting a backward motion vector starting from each block and ending at any block of the first original image; a starting point of the forward motion vector; and an end point of the forward motion vector The end point of the backward motion vector starting from A first determination step for determining that the forward motion vector is not allocated to the block of the interpolated image when the interval is larger than a predetermined threshold; a start point of the backward motion vector; and an end point of the backward motion vector A second determination step for determining that the backward vector is not assigned to the block of the interpolated image when the interval from the end point of the forward motion vector starting from the point is greater than a predetermined threshold; and the interpolated image Whether to draw only from the first original image, only from the second original image, or from both the first original image and the second original image In accordance with the determination result of the drawing direction determination step and the drawing direction determination step, the first detection step and the second detection step in the block of the interpolated image according to the determination result of the drawing direction determination step. An analogy vector for allocating to the undetermined block to which the detected forward motion vector and backward motion vector are not allocated is determined from the undetermined block in the third original image at the third time before the first time. According to the determination result of the first analogizing step that analogizes from the appearance frequency of the motion vector starting from a block within a predetermined range and starting from any block of the first original image, and the drawing direction determining step The analog vector for allocating to the undetermined block to which the forward motion vector and the backward motion vector detected by the first detection step and the second detection step are not allocated in the block of the interpolated image is the second time. Within a predetermined range from the undetermined block in the fourth original image at the fourth time after A second analogizing step for inferring from the appearance frequency of a motion vector having a block at the start point and any block of the second original image as an end point; the motion vector detected by the detecting step; A step of assigning the analogy vector estimated by the step and the second analogy step to each block of the interpolated image, and an analogy vector assigned to the undetermined block within a predetermined range from the undetermined block It includes a correction step of replacing the motion vector assigned to the block with a motion vector closest to the analogy vector.

  According to said structure, there exists an effect similar to the vector correction apparatus which concerns on this invention.

  The vector correction apparatus according to the present invention interpolates between a first original image corresponding to a first time and a second original image corresponding to a second time after the first time. A vector correction device that corrects a vector assigned to each block of an interpolated image to be executed, starting from each block of the first original image from the first original image and the second original image, From the first detection means for detecting a first forward motion vector whose end point is one of the blocks of the second original image, the first original image, and the second original image, the second original image is used. Second detection means for detecting a first backward motion vector starting from each block of the image and ending at any block of the first original image; a starting point of the first forward motion vector; First backward motion vector starting from the end point of the first forward motion vector A first determination means for determining that the first forward motion vector is not allocated to the block of the interpolated image when the interval between the first motion vector and the end point of the first backward motion vector is greater than a predetermined threshold; And the end point of the first forward motion vector starting from the end point of the first backward motion vector is larger than a predetermined threshold, the first backward vector is used as the block of the interpolated image. The second determination means for determining that the block is not allocated, and any block of the third original image corresponding to the third time before the first time, starting from each block of the first original image Is compared with the second backward motion vector whose starting point is each block of the third original image and whose second endpoint is any block of the first original image. Comparison means and the first source Among the blocks of the image, the distance from the end point of the first backward motion vector starting from the end point of the first forward motion vector is smaller than a predetermined threshold, and the end point of the second backward motion vector A first non-matching block specifying means for specifying, as a first non-matching block, a block whose interval from the end point of the second forward motion vector starting from the second threshold is greater than a predetermined threshold; and a block of the first original image Of which the interval between the end point of the first backward motion vector starting from the end point of the first forward motion vector is larger than a predetermined threshold and the end point of the second backward motion vector is the start point. Second non-matching block specifying means for specifying a block whose distance from the end point of the second forward motion vector to be smaller than a predetermined threshold as a second non-matching block, and the interpolated image Whether to draw only from the first original image, only from the second original image, or from both the first original image and the second original image The first forward motion vector detected by the first detection means and the second detection means in the block of the interpolated image according to the determination result of the drawing direction determination means, A first analogizing means for estimating an analogy vector for allocating an undetermined block to which the first backward motion vector is not allocated from the first forward motion vector starting from the first inconsistent block; and the drawing direction The first forward motion vector and the first forward motion vector detected by the first detection unit and the second detection unit in the block of the interpolated image according to the determination result of the determination unit. A second analogy means for analogizing from the second backward motion vector starting from the second inconsistent block an analogy vector for allocating to an undetermined block to which the first backward motion vector is not assigned; Allocating means for allocating the motion vector detected by the detecting means and the second detecting means and the analogy vector estimated by the first analogizing means and the second analogizing means to each block of the interpolated image; Correction means for replacing an analogy vector assigned to a block with a motion vector closest to the analogy vector among motion vectors assigned to a block within a predetermined range from the undetermined block, Yes.

  According to said structure, the vector correction apparatus which concerns on this invention is the 1st original image corresponding to 1st time, and the 2nd original image corresponding to 2nd time after 1st time The interpolation vector assigned to each block of the interpolated image to be interpolated between is corrected.

  The first detection means starts each block of the first original image from the first original image and the second original image, and uses any block of the second original image as an end point. A first forward motion vector is detected. Further, the second detection means starts each block of the second original image from the first original image and the second original image, and sets any block of the first original image as an end point. The first backward motion vector is detected.

  The first determination unit is configured such that an interval between the start point of the first forward motion vector and the end point of the first backward motion vector starting from the end point of the first forward motion vector is greater than a predetermined threshold. Then, it is determined that the first forward motion vector is not allocated as the interpolation vector. Further, the second determination means has an interval between the start point of the first backward motion vector and the end point of the first forward motion vector starting from the end point of the first backward motion vector greater than a predetermined threshold value. In this case, it is determined that the backward vector is not assigned as the interpolation vector.

  When the region in the first original image is also present in the second original image, the first backward motion vector starting from the start point of the first forward motion vector and the end block of the first forward motion vector. However, if they do not substantially match, the area existing in the first original image does not exist in the second original image. That is, the first determination unit and the second determination unit detect “disappearing area” (that is, an area hidden behind a moving object or the like) in the first original image as the correction target block. In addition, as a predetermined threshold value, for example, the number of blocks may be preset according to the detection accuracy of the motion vector, and is not particularly limited.

  Further, the comparing means starts with each block of the first original image as a starting point, and ends with any block of the third original image corresponding to a third time before the first time as an end point. The second backward motion vector is compared with the second forward motion vector starting from each block of the third original image and ending at any block of the first original image.

  Further, the first mismatch block specifying means is configured such that, in the first original image block, an interval from the end point of the first backward motion vector starting from the end point of the first forward motion vector is a predetermined threshold value. And a block having an interval from the end point of the second forward motion vector starting from the end point of the second backward motion vector to a start point is specified as a first mismatch block. Further, the second inconsistent block specifying means is configured such that, in the first original image block, an interval from the end point of the first forward motion vector starting from the end point of the first forward motion vector is a predetermined threshold value. A block that is larger than the end point of the second forward motion vector starting from the end point of the second backward motion vector and smaller than a predetermined threshold is specified as a second mismatched block.

  The first forward motion vector in the first non-matching block is a vector for the “appearing region”. Further, the second backward motion vector in the second non-matching block is a vector for “disappearing area”.

  Further, the drawing direction determination means draws only the first original image, draws only the second original image, or draws the first original image for each block of the interpolated image. Whether to draw from both the second original image and the second original image is determined. The drawing direction determination means can be obtained by hit board technique or vector calculation as disclosed in Patent Documents 1 to 3. Also, an analogy vector may be allocated from the original image correction area and given to the interpolation coordinates.

  Further, the first analogizing means is configured to detect the first forward motion vector detected by the first detecting means and the second detecting means in the block of the interpolated image according to the determination result of the drawing direction determining means and the first detecting means. An analogy vector for assigning to an undetermined block to which no first backward motion vector is assigned is inferred from the first forward motion vector starting from the first inconsistent block.

  Further, the second analogy means includes the first forward motion vector detected by the first detection means and the second detection means in the block of the interpolated image according to the determination result of the drawing direction determination means, and the An analogy vector for assignment to an undetermined block to which no first backward motion vector is assigned is inferred from the second backward motion vector starting from the second inconsistent block.

  For example, among the first forward motion vectors starting from the first non-matching block within a predetermined range centering on an undetermined block, a motion vector having a high appearance frequency or an average motion vector is used as an analogy vector. And Of the second backward motion vectors starting from the second non-matching block within a predetermined range centering on an undetermined block, a motion vector having a high appearance frequency or an average motion vector is used as an analogy vector. And

  The assigning means assigns the motion vector detected by the first detecting means and the second detecting means and the vector estimated by the first analogizing means and the second analogizing means to each block of the interpolated image.

  Then, the correcting means converts the interpolation vector assigned to the undetermined block to the interpolation vector assigned to a block other than the undetermined block within the predetermined range from the undetermined block (that is, the periphery of the block to which the analogy vector is assigned). Among the interpolation vectors assigned to the block, the nearest interpolation vector of the interpolation vectors detected by the detection means) is replaced. The correction means takes the difference between the analogy vector and the interpolation vector assigned to the surrounding blocks, and replaces the analogy vector with a vector having the smallest difference from the analogy vector.

  Thereby, the vector correction apparatus according to the present invention does not directly use the motion vector motion vector between the third original image and the first original image as the interpolation vector. That is, no analogy vector is used as the interpolation vector. Therefore, even if a change in motion occurs between frames, a motion vector detected between the first original image and the second original image is used as an interpolation vector, so that discontinuities occur in the background. Therefore, it is possible to obtain a good quality interpolated image. In addition, the fourth original image is not required, and an equivalent interpolated image can be created with the three original images of the first original image, the second original image, and the third original image. .

The vector correction method according to the present invention interpolates between a first original image corresponding to a first time and a second original image corresponding to a second time after the first time. A vector correction method for correcting a vector assigned to each block of an interpolated image,
A first forward motion vector starting from each block of the first original image and ending at any block of the second original image from the first original image and the second original image. A first detection step of detecting
From the first original image and the second original image, a first backward motion vector starting from each block of the second original image and ending at any block of the first original image A second detection step of detecting
If the interval between the start point of the first forward motion vector and the end point of the first backward motion vector starting from the end point of the first forward motion vector is greater than a predetermined threshold, the first forward direction A first determination step for determining not to allocate a motion vector to the block of the interpolated image;
If the interval between the start point of the first backward motion vector and the end point of the first forward motion vector starting from the end point of the first backward motion vector is greater than a predetermined threshold, the first backward vector A second determination step that determines not to allocate a block to the interpolated image block;
A second backward motion vector starting from each block of the first original image and ending at any block of the third original image corresponding to the third time before the first time; A comparison step of comparing a second forward motion vector starting from each block of the third original image and ending at any block of the first original image;
Among the blocks of the first original image, an interval from the end point of the first backward motion vector starting from the end point of the first forward motion vector is smaller than a predetermined threshold, and the second rear image A first non-matching block specifying step for specifying a block having an interval from the end point of the second forward motion vector starting from the end point of the directional motion vector as a first non-matching block, which is larger than a predetermined threshold;
Among the blocks of the first original image, an interval from the end point of the first backward motion vector starting from the end point of the first forward motion vector is larger than a predetermined threshold, and the second rear image A second non-matching block specifying step for specifying, as a second non-matching block, a block whose interval from the end point of the second forward motion vector starting from the end point of the direction motion vector is smaller than a predetermined threshold;
For each block of the interpolated image, draw only from the first original image, draw only from the second original image, or both the first original image and the second original image A drawing direction determination step for determining whether to draw from,
The first forward motion vector and the first backward motion vector detected by the first detection step and the second detection step in the block of the interpolated image are allocated according to the determination result of the drawing direction determination step. A first analogizing step for analogizing an analogy vector to be assigned to an undetermined block from the first forward motion vector starting from the first inconsistent block;
The first forward motion vector and the first backward motion vector detected by the first detection step and the second detection step in the block of the interpolated image are allocated according to the determination result of the drawing direction determination step. A second analogizing step for analogizing an analogy vector to be assigned to an undetermined block from the second backward motion vector starting from the second inconsistent block;
An allocation step for allocating the motion vectors detected by the first detection step and the second detection step and the analogy vectors estimated by the first analogy step and the second analogy step to each block of the interpolated image; ,
A correction step of replacing the analogy vector assigned to the undetermined block with a motion vector closest to the analogy vector among the motion vectors assigned to the blocks within a predetermined range from the undetermined block. Feature vector correction method.

  According to said structure, there exists an effect similar to the vector correction apparatus which concerns on this invention.

  An image interpolation device according to the present invention is an image interpolation device including a vector correction device, and includes an interpolation image generation unit that generates the interpolation image based on the vector corrected by the correction unit. Preferably it is.

  According to said structure, the image interpolation apparatus which concerns on this invention can draw an interpolation image based on the correct | amended interpolation vector. Thereby, a good quality interpolated image can be obtained.

  Moreover, it is preferable that the television receiver according to the present invention includes the image interpolation device.

  Moreover, it is preferable that the video reproduction apparatus according to the present invention includes the image interpolation apparatus.

  Note that the image interpolation device may be realized by a computer. In this case, a control program for realizing the image interpolation device in the computer by operating the computer as each of the above means and a computer-readable recording medium recording the control program also fall within the scope of the present invention.

  The vector correction apparatus according to the present invention interpolates between a first original image corresponding to a first time and a second original image corresponding to a second time after the first time. A vector correction device that corrects a vector assigned to each block of an interpolated image to be executed, starting from each block of the first original image from the first original image and the second original image, Detecting means for detecting a motion vector whose end point is one of the blocks of the second original image; and for each block of the interpolated image, drawing from only the first original image, The drawing direction determining means for determining whether to draw only from the original image or from both the first original image and the second original image, and depending on the determination result of the drawing direction determining means, Detected by the detection means in the block of the inset image An analogy vector for allocating to the undetermined block to which the assigned motion vector is not allocated is a block within a predetermined range from the undetermined block in the third original image at the third time before the first time. In the block of the interpolated image according to the first analogizing means for inferring from the appearance frequency of the motion vector whose starting point is one of the blocks of the first original image and the drawing direction determining means. An analog vector for allocating to an undetermined block to which a motion vector detected by the detecting means is not allocated is a predetermined range from the undetermined block in a fourth original image at a fourth time after the second time. Analogize from the appearance frequency of motion vectors starting from a block inside and ending at any block of the second original image Allocation means for allocating the motion vector detected by the second analogy means, the motion vector detected by the detection means, and the analogy vector estimated by the first analogy means and the second analogy means to each block of the interpolated image; Correction means for replacing the analogy vector assigned to the undetermined block with a motion vector closest to the analogy vector among the motion vectors assigned to blocks within a predetermined range from the undetermined block. It is a feature.

  Moreover, the vector correction apparatus according to the present invention includes a first original image corresponding to a first time and a second original image corresponding to a second time after the first time. A vector correction apparatus for correcting a vector assigned to each block of an interpolated image to be interpolated, wherein each block of the second original image is started from the first original image and the second original image. Detection means for detecting a motion vector whose end point is one of the blocks of the first original image, and each block of the interpolated image is drawn from only the first original image, or the first The drawing direction determining means for determining whether to draw only from the two original images or from both the first original image and the second original image, and according to the determination result of the drawing direction determining means, By the detection means in the block of the interpolated image. A block that is within a predetermined range from the undetermined block in the third original image at a third time before the first time, the analogy vector for allocating the detected motion vector to the undetermined block that is not allocated And a block of the interpolated image according to the determination result of the drawing direction determination means, and first analogy means for estimating from the appearance frequency of a motion vector whose starting point is any block of the first original image In the fourth original image at a fourth time after the second time, a presumed vector for allocating to the undetermined block to which the motion vector detected by the detecting means is not assigned in From the appearance frequency of motion vectors starting from a block within the range and ending at any block of the second original image Allocating means for allocating the second analogy means to be estimated, the motion vector detected by the detection means, and the analogy vector estimated by the first analogy means and the second analogy means to each block of the interpolated image And a correction unit that replaces the analogy vector assigned to the undetermined block with a motion vector closest to the analogy vector among the motion vectors assigned to blocks within a predetermined range from the undetermined block. It is characterized by that.

  Moreover, the vector correction apparatus according to the present invention includes a first original image corresponding to a first time and a second original image corresponding to a second time after the first time. A vector correction apparatus for correcting a vector assigned to each block of an interpolated image to be interpolated, wherein each block of the first original image is started from the first original image and the second original image. From the first original image and the second original image, the first original detecting means for detecting a forward motion vector whose end point is one of the blocks of the second original image, and the second original image. Second detection means for detecting a backward motion vector starting from each block of the image and ending at any block of the first original image; a starting point of the forward motion vector; and the forward motion vector With the end point of the backward motion vector starting from the end point of A first determination unit that determines that the forward motion vector is not assigned to the block of the interpolated image, and a start point of the backward motion vector and an end point of the backward motion vector when the interval is greater than a predetermined threshold Second determination means for determining that the backward vector is not allocated to the block of the interpolated image when the interval from the end point of the forward motion vector starting from the point is greater than a predetermined threshold; and the interpolated image Whether to draw only from the first original image, only from the second original image, or from both the first original image and the second original image And a forward motion vector detected by the first detection means and the second detection means in the block of the interpolated image according to the determination result of the drawing direction determination means. Analog vectors for assignment to undecided blocks to which no motion vector or backward motion vector is assigned are within a predetermined range from the undetermined block in the third original image at a third time before the first time. The first analogizing means for inferring from the appearance frequency of the motion vector having the block as the start point and any block of the first original image as the end point, and the interpolation direction of the interpolated image according to the determination result of the drawing direction determining means An analogy vector for allocating to the undetermined block to which the forward motion vector and the backward motion vector detected by the first detection means and the second detection means in the block are not assigned is the fourth vector after the second time. In the fourth original image at the time, a block within a predetermined range from the undetermined block is used as a starting point of the second original image. A second analogy means for inferring from the appearance frequency of a motion vector having any block as an end point, a motion vector detected by the detecting means, and an analogy vector inferred by the first analogy means and the second analogy means Are assigned to each block of the interpolated image, and the analogy vector assigned to the undetermined block is selected from among the motion vectors assigned to the blocks within a predetermined range from the undetermined block. And a correction means for replacing the motion vector closest to.

  Moreover, the vector correction apparatus according to the present invention includes a first original image corresponding to a first time and a second original image corresponding to a second time after the first time. A vector correction apparatus for correcting a vector assigned to each block of an interpolated image to be interpolated, wherein each block of the first original image is started from the first original image and the second original image. From the first detection means for detecting a first forward motion vector whose end point is one of the blocks of the second original image, the second original image, and the second original image, the second Second detection means for detecting a first backward motion vector starting from each block of the original image and ending at any block of the first original image; and a start point of the first forward motion vector; The first backward motion starting from the end point of the first forward motion vector First determination means for determining that the first forward motion vector is not allocated to the block of the interpolated image when the distance from the end point of the vector is larger than a predetermined threshold, and the start point of the first backward motion vector And the end point of the first forward motion vector starting from the end point of the first backward motion vector is larger than a predetermined threshold, the first backward vector is used as the block of the interpolated image. The second determination means for determining that the block is not allocated, and any block of the third original image corresponding to the third time before the first time, starting from each block of the first original image Is compared with the second backward motion vector whose starting point is each block of the third original image and whose second endpoint is any block of the first original image. The comparison means and the first Among the blocks of the original image, the distance from the end point of the first backward motion vector starting from the end point of the first forward motion vector is smaller than a predetermined threshold, and the second backward motion vector A first non-matching block specifying means for specifying, as a first non-matching block, a block whose distance from the end point of the second forward motion vector starting from the end point of the second pre-motion vector is larger than a predetermined threshold, and the first original image Among the blocks, the distance from the end point of the first backward motion vector starting from the end point of the first forward motion vector is larger than a predetermined threshold, and the end point of the second backward motion vector is A second non-matching block specifying means for specifying, as a second non-matching block, a block whose interval from the end point of the second forward motion vector as the start point is smaller than a predetermined threshold; For each block of the inset image, drawing from only the first original image, drawing from only the second original image, or drawing from both the first original image and the second original image The first forward motion detected by the first detection means and the second detection means in the block of the interpolated image according to the determination result of the drawing direction determination means and the drawing direction determination means. A first analogizing means for estimating an analogy vector for assigning a vector and an undetermined block to which the first backward motion vector is not assigned from the first forward motion vector starting from the first inconsistent block; The first forward motion vector detected by the first detection means and the second detection means in the block of the interpolated image according to the determination result of the drawing direction determination means. And second analogy means for analogizing from the second backward motion vector starting from the second inconsistent block an analogy vector for allocating to an undetermined block to which the first backward motion vector is not assigned, Allocating means for allocating the motion vectors detected by the first detecting means and the second detecting means and the analogy vectors estimated by the first analogizing means and the second analogizing means to each block of the interpolated image; Correction means for replacing an analogy vector assigned to an undetermined block with a motion vector closest to the analogy vector among motion vectors assigned to a block within a predetermined range from the undetermined block. It is said.

  Therefore, the analogy vector assigned as the interpolation vector is replaced with a motion vector obtained as a result of motion detection between the original images sandwiching the interpolation image. Therefore, the interpolation vector becomes a vector corresponding to a change in motion, and it is possible to generate a good quality interpolation without generating discontinuous points in the interpolation drawn using the corrected interpolation vector. It becomes possible.

It is a block diagram which shows the structure of the image interpolation apparatus 1 which concerns on this invention. It is a figure which shows the original image of the image | video which a background moves, (a) is a figure which shows the past original image f (t-1) temporally with respect to an interpolation image, (b) It is a figure which shows the future original image f (t) temporally with respect to an insertion image. It is a figure for demonstrating the drawing process of an interpolation image, Comprising: It is the figure which took the cross section of the movement direction of the background, and displayed the one-dimensional display of the original image and interpolation image which are shown to Fig.2 (a) and (b). is there. It is a figure for demonstrating the state to which the amount of motion of a background changes, Comprising: The original image f (t-1) in the time t-1, the original image f (t) in the time t, The original image f (t + 1) in the time t + 1 ), An example of an original image f (t + 2) at time t + 2 and an original image f (t + 3) at time t + 3. It is a figure for explaining a state in which the amount of motion of the background changes, and is a figure explaining a problem that occurs in the interpolation when drawing FIG. 4, and an interpolation is created at an intermediate point of the original image It is a figure, the original image f (t-0.5) in the time t-0.5, the original image f (t + 0.5) in the time t, the original image f (t + 1.5) in the time t + 1.5, and the time t + 2 5 is a diagram illustrating an example of an original image f (t + 2.5) in .5. It is a figure for demonstrating the state from which the amount of motions of a background changes, Comprising: It is a figure which shows that the problem of the interpolation image of FIG. 5 can be improved by this invention. It is a block diagram which shows the structure of the image interpolation apparatus which concerns on this invention. It is a figure explaining the method of a bidirectional | two-way vector comparison, and is a figure which shows the coordinate to compare. It is a figure explaining the method of a bidirectional | two-way vector comparison, and is a figure explaining a matching condition. It is a figure explaining operation | movement of the statistical data process part which is a modification of the image interpolation apparatus which concerns on this invention. It is a figure which shows the relationship between the 60Hz original image and 120Hz output image which are input. It is a figure which shows the example which divides | segments a reference | standard image into a block, in order to calculate a motion vector. It is a figure which shows the example of allocation of the vector to an interpolation coordinate.

[Embodiment 1]
(Outline of the image interpolation device 1)
FIG. 1 is a block diagram showing a configuration of an image interpolation apparatus 1 according to the present invention. An image interpolation apparatus 1 according to the present invention includes a delay unit 2 (video data delay unit), a vector detection unit 3 (detection unit), a drawing direction determination unit 4 (drawing direction determination unit), and a delay unit 5 (vector data delay unit). , Vector allocation unit 6 (allocation unit), vector correction unit 7 (correction unit), statistical processing unit 8 (first analog estimation unit, second analog estimation unit), delay unit 9 (primary replacement vector delay unit), and delay unit 10 (Secondary replacement vector delay unit), drawing unit 11 (interpolated image generating unit), and delay unit 12 (drawing delay unit).

  In the present embodiment, the motion vector correction apparatus 100 includes a delay unit 2 (video data delay unit), a motion vector detection unit 3, a drawing direction determination unit 4, a delay unit 5 (vector data delay unit), and a vector allocation unit. 6, a vector correction unit 7, a statistical processing unit 8, a delay unit 9 (primary replacement vector delay unit), and a delay unit 10 (secondary replacement vector delay unit). The motion vector correction apparatus 100 is provided integrally with the image interpolation apparatus 1 and forms the image interpolation apparatus 1 together with the drawing unit 11 and the delay unit 12 (drawing delay unit).

  The image interpolation apparatus 1 includes an original image f (t + 1) (second original image, future image) at time t + 1 included in a video signal input from the outside, and an original image f (t) at time t before one. An interpolated image is generated from (first original image, past image). In the image interpolation device 1, an external video signal is input to the delay unit 2 and the vector detection unit 3. Hereinafter, processing when a video signal representing the original image f (t + 1) at time t + 1 is input from the outside will be described.

  The delay unit 2 delays the input external video signal and supplies the delayed video signal to the vector detection unit 3, the drawing unit 11, and the delay unit 12. At a timing when a video signal representing the original image f (t + 1) is input from the outside, the delay unit 2 outputs a video signal representing the original image f (t). The delay unit 2 is configured by a memory such as DDR, for example.

  The motion vector detection unit 3 calculates a motion vector using the original image f (t) as a reference image. In addition, as shown in FIG. 2, the motion vector detection unit 3 divides the original image into a plurality of blocks including a plurality of pixels, and detects a motion vector in units of blocks. FIG. 2 is a diagram showing an original image of a video whose background moves, and (a) is a diagram showing a temporally original image f (t−1) with respect to the interpolated image. ) Is a diagram showing an original image f (t) of the future in time with respect to the interpolated image. The motion vector detection unit 3 calculates a motion vector by block matching or a gradient method, but the motion vector may be calculated by another method. The motion vector detection unit 3 outputs the detected motion vector to the drawing direction determination unit 4 and the statistical processing unit 8. This vector is expressed on the basis of the starting time as V (t).

  The drawing direction determination unit 4 specifies a region that should be expressed only by the past image and a region that should be expressed only by the future image by a method (hitboard technique) as shown in Patent Document 1. The region-specified vector is sent to the delay unit 5 in order to match the temporal phase with the vector to be replaced, and delayed by one input.

  The statistical processing unit 8 processes the motion vector information from the motion vector detection unit 3 as statistical data (for example, a histogram), and extracts an optimum replacement vector (analogue vector) for each predetermined coordinate. For example, the vector given to the “disappearing area” is a vector whose end point is the coordinates of the “disappearing area” in the detection at the time of input one frame before, and the vector given to the “appearing area” is one frame after This is a vector whose starting point is the coordinates of the “appearing region” in detection at the time of input. By performing statistical processing, it is possible to reduce the amount of data to be held by calculating a representative replacement vector for each predetermined area, instead of calculating replacement vectors individually for all coordinates. Based on the processing result, a vector fV (t) to be applied to the “appearing region” and a vector pV (t) to be applied to the “disappearing region” are output. fV (t) is output to the vector allocation unit 6, and pV (t) is output to the delay unit 9 in order to adjust the time axis to be used.

  The vector allocating unit 6 detects the detection vector V (t−1) that has been temporally adjusted by the delay unit 5, the delay unit 9, and the delay unit 10, and information on the replacement area (analog vectors without using the detection vector). Information on a replacement area), an analogy vector fV (t) to be replaced when only the future image is to be expressed, and an analogy vector pV (t-2) to be replaced when only the past image is to be expressed Based on the information, an interpolation vector Vi (t−1) corresponding to the interpolation time is created and output to the vector correction unit 7. In other words, the information on the replacement area is an area in which, for example, a detection vector with low reliability or a detection vector erroneously detected is allocated in the interpolation coordinates (block of the interpolation image). This is information on a region that needs to be replaced with a more accurate analogy vector. Then, the vector allocating unit 6 performs processing for replacing the interpolation vector to be allocated to the interpolated image from the detection vector to the analogy vector based on the information on the replacement area. At this time, it is preferable to add information serving as a flag to the interpolation vector so that the replaced region can be recognized by the vector correction unit 7. It is also preferable to add information on an unallocated area.

  The vector correction unit 7 calculates the vector of the block adjacent to the block on the interpolated image and the replacement vector for the region to which the replacement vector (analogue vector) is allocated among the interpolation vectors Vi (t−1). And compare. Among the vectors of the adjacent blocks, the one having the smallest difference between the replacement vector and the vector is calculated for each block. When the calculated vector is mV (t−1), the block interpolation vector value fV (t) or pV (t−2) is replaced with mV (t−1). The adjacent block to be compared is set in advance according to the required accuracy, and is not particularly limited.

  When a non-allocated area also has a replacement area in an adjacent block, the non-allocated area is treated as a replacement area, and the value of fV (t) or pV (t−2) is handled by adapting the value on the adjacent block. If the vector difference with any adjacent block is larger than the threshold value, it is preferable not to perform the replacement process. When vectors similar to different fV (t) or pV (t−2) exist in a plurality of adjacent blocks, an average value of similar vectors may be taken.

  Then, based on the motion vector allocation information from the vector allocation unit 9, the drawing unit 11 reads the video signal at each coordinate of the interpolated image from the video signal of the original image, and generates an interpolated image.

(Image interpolation processing)
First, of the image interpolation processing in the image interpolation apparatus 1 according to the present invention, processing contents similar to general image interpolation processing will be described.

(1) Detection of Motion Vector FIG. 2 is a diagram illustrating an original image of a video in which the camera follows the object A, and (a) is a diagram illustrating an original image f (t−1) at time t−1. (B) is a figure which shows the original image f (t) in the time t. In the example illustrated in FIG. 2, an interpolated image between time t and time t−1 is generated. Time t-1 is the time immediately before time t, the original image f (t-1) is a past image with respect to the interpolated image, and the original image f (t) is with respect to the interpolated image. It will be a future image.

  As shown in FIG. 2 (a), in the original image f (t-1), a portion surrounded by a thin line like the part 2-1 represents one pixel (pixel), and is 8 × 8 pixels. Detection blocks 2-2 to 2-11 are configured for each pixel group. Also in the original image f (t), detection blocks 2-13 to 2-22 are configured for each pixel group of 8 × 8 pixels. A white pixel group in the original images f (t−1) and f (t) represents the object A. In FIGS. 2A and 2B, a region indicated by diagonal lines represents the background region of the object A. In FIGS. 2A and 2B, the difference depending on the line type is a texture added to visually show the movement of the background, and does not indicate the same information. An arrow 2-12 in FIG. 2A indicates a motion vector in the background area.

  FIG. 3 is a diagram for explaining the drawing process of the interpolated image. The original image and the interpolated image shown in FIGS. FIG. In FIG. 3, the past image f (t−1) and the future image f (t) are shown associated with each other by a motion vector. The motion vector is detected for each detection block between f (t−1) and f (t). As a method for obtaining the motion vector, a general method such as block matching or a gradient method is used.

  In motion vector detection, “disappearing areas” and “appearing areas” are problematic. For example, the image of the background area shown in the detection block 3-6 is a “disappearing area” that is hidden by an object in the future image. That is, the background image shown in the detection block 3-6 of the original image f (t-1) is hidden behind the object in the original image f (t). Therefore, a region having information only in one direction in time, such as 3-13 and 3-14, is generated between blocks (object and background region) correctly detected in the interpolation. Accordingly, this area is obtained by analogy because no detection information can exist.

(2) Analogy Vector The analogy vector starts with specifying an area corresponding to the detection block 3-6 or 3-8 of the original image area in FIG. First, in the area of the block 3-6, the vector obtained by motion vector detection competes with other detection vectors at any coordinate of the original image f (t). The starting point of the competing vector in the original image f (t) is determined as the “disappearing area”. In the area of the block 3-8, an area in which no vector is allocated in the coordinates of f (t) is determined as an “appearing area”.

  The motion vector of the “disappearing region” is inferred that the past motion is continued. Accordingly, a vector whose end point is the area of the block 3-6 in the detection between the original image f (t-2) (fourth original image) and the original image f (t-1) (second original image) is It is predicted to continue, and the analog vector of block 3-6 is used. The motion vector of the “appearing region” is inferred from future motion. Therefore, the vector starting from the area of the block 3-8 in the detection between the original image f (t) (first original image) and the original image f (t + 1) (third original image) continues the past. The reverse vector is assumed to be an analog vector from the block 3-8 of the original image f (t).

(3) Allocation of motion vectors Hereinafter, allocation of motion vectors to interpolation coordinates will be described. As described above, the image interpolation apparatus 1 according to the present invention corrects the motion vectors detected in the “disappearing region” and the “appearing region” based on the statistical data, and then assigns the motion vector to the interpolation coordinates. Process.

  FIG. 3 shows a state in which motion vectors detected for the original image f (t) with respect to the original image f (t−1) are assigned to the interpolation coordinates. In the example shown in FIG. 3, the motion vector is obtained from the past image f (t−1) to the future image f (t), and is shown as ideally obtained. In FIG. 3, the solid arrow indicates the motion vector detected in the original image and indicates the motion vector allocated in the interpolation block, and the broken arrow indicates the analogy vector.

  In FIG. 3, the original image f (t-1) includes detection blocks 3-1 to 3-6, and the original image f (t) includes detection blocks 3-7 to 3-12. The detection blocks 2-2 to 2-6 in FIG. 2A correspond to the detection blocks 3-2 to 3-6 in FIG. 3, and the detection blocks 2-13 to 2-17 in FIG. This corresponds to the detection blocks 3-8 to 3-12 in FIG.

  In the example illustrated in FIG. 3, the interpolated image fi (t−1) includes interpolation blocks 3-13 to 3-24, and the original image f (t−1) and the original image f (t) are Created at midpoint. A motion vector and an analogy vector detected in the reference original image are assigned to the interpolation block. The interpolation block is an area smaller than the detection block, and includes fewer pixels than the detection block. Hereinafter, the motion vector assigned to the interpolation block is referred to as an interpolation vector.

In FIG. 3, the detection block 3-1 detects the motion of the background as a motion vector and performs allocation. (Not shown)
Also, a vector indicating the background motion is detected toward 3-7 as the motion vector in the detection block 3-2. The vectors detected in 3-2 are assigned to interpolation blocks 3-14 and 3-15.

  In addition, as a motion vector in the detection block 3-3, a vector indicating the motion of the object is detected toward 3-9. Then, the motion vector detected in the detection block 3-3 is allocated to the interpolation blocks 3-17 and 3-18.

  In addition, as a motion vector in the detection block 3-4, a vector indicating the motion of the object is detected toward 3-10. Then, the motion vector detected in the detection block 3-4 is allocated to the interpolation blocks 3-19 and 3-20.

  In addition, as a motion vector in the detection block 3-5, a vector indicating the motion of the object is detected toward 3-11. Then, the motion vector detected in the detection block 3-5 is allocated to the interpolation blocks 3-21 and 3-22.

  The motion vector in the detection block 3-6 is detected as the closest detection result toward a similar point in the vicinity (not shown). The detected vector conflicts with the correct vector on the original image f (t). As a result, the analogy vector from the past is used together with the correct correct vector. In FIG. 3, a background motion vector 3-26 is generated and assigned from the detection block 3-6 to the interpolation block 3-23.

  On the other hand, the result is that there is no vector to be allocated to the block 3-8 on f (f). As a result, an analogy vector from the future is used. In FIG. 3, a background motion vector 3-25 is generated from the future position of the block 3-8 toward the interpolation block 3-16 and assigned. It is done.

(4) Drawing of Interpolated Image Next, a method of drawing an interpolated image using the interpolation vector will be described. In FIG. 2A, the upper left coordinate of the drawing is (0, 0), the right direction of the drawing is the positive direction of the x coordinate, and the lower direction of the drawing is the positive direction of the y coordinate. The pixel value of coordinates (x, y) is represented as Pp (x, y), and the pixel value of coordinates (x, y) in the future image f (t) is represented as Pf (x, y). In addition, in the interpolated image fi (t−1), the motion vector assigned to the coordinate (x, y) is (Vx, Vy), and the pixel value of the coordinate (x, y) is P (x, y). . Here, it is assumed that the interpolated image fi (t-1) is drawn in the middle (1/2 position) between the past image f (t-1) and the future image f (t).

At this time, the interpolated image fi (t−1) to be drawn is
P (x, y) = α × Pp (x−Vx × 1/2, y−Vy × 1/2) + (1−α) × Pf (x + Vx × 1/2, y + Vy × 1/2)
It becomes.

  Here, Pp (x−Vx × 1/2, y−Vy × 1/2), Pf (x + Vx × 1/2, y + Vy × 1/2) does not indicate a single pixel, but a past image, It means a value that represents the future. For example, when the vector indicates a decimal value, it means a value obtained by linear interpolation of the original image, a value filtered by a Gaussian filter, or the like. Α is a mixing ratio of the past image and the future image, and is called a blend ratio. The value of α may be fixed to ½, or may be varied depending on the ratio of temporal internal dividing points of the interpolation.

(Background speed change problem)
In the above description, the background speed is assumed to be constant between frames for simplification. However, in an actual image, acceleration motion and deceleration motion occur. This case will be described with reference to FIG. FIG. 4 is a diagram for explaining a state in which the amount of motion of the background changes. The original image f (t−1) at time t−1, the original image f (t) at time t, and the original image at time t + 1. It is an example which shows image f (t + 1), original image f (t + 2) in time t + 2, and original image f (t + 3) in time t + 3. In FIG. 4, a small square indicated by 4-1 represents one pixel and means the background. In FIG. 4, it is assumed that the motion vector is detected for each pixel. In addition, alphabets are written to distinguish each pixel. The shaded area indicated by 4-2 means an object. Actually, it is assumed that the object is accelerating, and the camera behaves so as to catch the object in the center. Therefore, it is assumed that the image is captured so that the background relatively accelerates.

  In the original image f (t-1) to the original image f (t), a motion vector 4-3 indicating that the object is stationary is detected. In the background, 4-4 motion vectors indicating the motion of two pixels in the left direction are detected. From the original image f (t) to the original image f (t + 1), the object detects a motion vector 4-5 indicating a stationary motion, and the background is a motion vector 4-6 indicating a motion of four pixels in the left direction. To detect.

  From the original image f (t + 1) to the original image f (t + 2), a motion vector 4-7 indicating that the object is stationary is detected. As the background, 4-8 motion vectors indicating the motion of 6 pixels in the left direction are detected.

  From the original image f (t + 2) to the original image f (t + 3), a motion vector 4-9 indicating that the object is stationary is detected. In the background, 4-10 motion vectors indicating the motion of 8 pixels in the left direction are detected.

  The background area between the object and the background indicating the background motion vectors 4-4, 4-6, 4-8, 4-10 is similar to the similar area because there is no corresponding point in the original image on the future side. To generate a vector. Since the generated vector competes with another vector at the end point, it is replaced with an analogy vector and assigned.

  The interpolation created by using FIG. 5 will be described. FIG. 5 is a diagram for explaining a state in which the amount of movement of the background changes, and is a diagram for explaining a problem that occurs in the interpolation when drawing FIG. 4. The interpolation is performed at an intermediate time point of the original image. The original image f (t-0.5) at time t-0.5, the original image f (t + 0.5) at time t, and the original image f (t + 1.5) at time t + 1.5 ) And an example of the original image f (t + 2.5) at time t + 2.5. In FIG. 5, the arrow toward the interpolation indicates that the original image is referred to by the allocation vector. Vectors indicated by reference numerals 5-3A, 5-5A, 5-7A, and 5-9A indicate that the object region of the interpolated image refers to the original image on the past side by the motion vector illustrated in FIG. Vectors indicated by reference numerals 5-3B, 5-5B, 5-7B, and 5-9B indicate that the object region of the interpolated image refers to the original image on the future side by the motion vector illustrated in FIG. Vectors indicated by reference numerals 5-4A, 5-6A, 5-8A, and 5-10A indicate that the background area of the interpolated image refers to the original image on the past side by the motion vector illustrated in FIG. The vectors indicated by reference numerals 5-4B, 5-6B, 5-8B, and 5-10B indicate that the background area of the interpolated image refers to the future original image by the motion vector illustrated in FIG. The interpolated image portion indicated by the arrows of both the past image and the future image so far is displayed as a final interpolated image by the above-described α blending. The arrows indicated by reference numerals 5-11 to 5-14 indicate that the disappearing area is drawn only from the past image by the analogy vector.

In the example of FIG. 5, in the motion vector detection in the original image f (t−1) and the original image f (t), the background A and the background B are replaced with an analogy vector because there is no corresponding point in the future image. In this case, it is inferred that the past movement is continued, and a vector indicated by reference numeral 5-11 is generated as an analogy vector. (Here, f (t-2) is not shown, but it is assumed that the object is stationary.) Since the analogy vector from the background A does not compete with others, it is assigned to the interpolation point. The analogy vector from B disappears because it competes with others. (The analogy vector generally has a lower priority at the time of competition. Also, the pixel correlation value tends to increase, and even if the priority at the competition point is determined by SAD, DFD, etc., it is difficult to select.)
In the vector detection between the original image f (t) and the original image f (t + 1), since the background C, the background D, the background E, and the background F have no corresponding points in the future image, the detection vectors from the backgrounds C to F are It replaces the analogy vector. As an analogy vector, since past motion is presumed to continue, a vector 5-12 corresponding to the reverse direction of the vector 5-4B is generated. The analogy vectors from the background D and the background E are allocated to the interpolation point because they do not compete with others, but the analogy vectors from the background C and the background F disappear because they compete with each other. Note that the undetermined block in the claims corresponds to, for example, D to E of the interpolated image f (t + 0.5).

  In the vector detection between the original image f (t + 1) and the original image f (t + 2), the background G, the background H, the background I, the background J, the background K, and the background L have no corresponding points in the future image. The detection vector from ˜K replaces the analogy vector. As an analogy vector, since it is presumed that past motion will continue, a vector 5-13 corresponding to the reverse direction of the vector 5-6B is generated. The background I, background J, and analogy vectors from the background are assigned to the interpolation point because they do not compete with others, but the analogy vectors from the background G, background H, and background L disappear because they compete with others. The undetermined block in the claims corresponds to, for example, I to M of the interpolated image f (t + 1.5).

  In vector detection between the original image f (t + 2) and the original image f (t + 3), the background M, the background N, the background O, the background P, the background Q, the background R, the background S, and the background T correspond to the future images. Therefore, the detection vector from the background M to S is replaced with an analogy vector. As an analogy vector, since it is presumed that the past movement is continued, a vector 5-14 corresponding to the reverse direction of the vector 5-8B is generated. The analogy vectors from the background P, background Q, background R, and background S do not conflict with each other, so they are assigned to the interpolation points, but the analogy vectors from the background M, background N, background O, and background T compete with others. To disappear. Note that the undetermined block in the claims corresponds to P to U of the interpolated image f (t + 2.5), for example.

Here, when focusing on the interpolation, in the interpolation f (t−0.5), a discontinuous point is present between the background C drawn using the detected vector and the background A drawn using the analogy vector. Has occurred. (Originally, background B should be present.)
Further, in the interpolated image f (t + 0.5), a discontinuous point occurs between the background G drawn using the detected vector and the background E drawn using the analogy vector. (Originally, there should be a background F.)
In addition, in the interpolated image f (t + 1.5), discontinuous points are generated between the background M drawn using the detected vector and the background K drawn using the analogy vector. (Originally, the background L should be present.)
In addition, in the interpolated image f (t + 2.5), discontinuous points are generated between the background U drawn using the detected vector and the background S drawn using the analogy vector. (Originally, there should be a background T.)
Since discontinuous points are generated along the contour of the object in this way, the interpolated image is lowered.

(Correction of interpolation vector)
In the image interpolation apparatus 1 according to the present invention, the interpolation vector is corrected by the vector correction unit 7 in FIG. In FIG. 1, the vector correction unit 7 receives an interpolation vector, a future-side analogy vector fV (t), a past-side analogy vector pV (t-2), and unallocated area information from the vector allocation unit 6. Get. At this time, fV (t) and pV (t−2) may be transmitted with actual data or with a flag or the like assigned to the assigned vector.

  The vector correction unit 7 performs correction processing in the horizontal direction when the upper left coordinate of the interpolated image is (0, 0). When the horizontal component of the interpolation vector is Vix (x, y) and the vertical component is Viy (x, y) when the interpolation coordinates to be corrected are (x, y), the interpolated image The vector component in the horizontal direction to the left of each is expressed as Vix (x-1, y), and the vector component in the vertical direction is expressed as Viy (x-1, y).

  The correction processing in the vector correction unit 7 occurs when the processing coordinates are an analogy vector. The correction content is performed by comparison with the vector of the adjacent block. When the analogy vector is (Vix (x, y), Viy (x, y)), for example, | Vix (x, y) −Vix (x ± 1, y ± 1) | ^ 2 + | Viy (x , Y) −Viy (x ± 1, y ± 1) | ^ 2 is obtained individually and replaced with the vector having the smallest size. For example, if (Vix (x-1, y), Viy (x-1, y)) shows the smallest value, the vector of the interpolation coordinates (x, y) is (Vix (x-1, y). ), Viy (x-1, y)), and the flag to which the analogy vector is assigned is canceled. When inspecting a difference from an adjacent vector, it is desirable to delete an interpolated coordinate vector to which an analogy vector is assigned from the inspection target.

  By this correction processing, the background motion vector can be propagated from the left or above, and the background vector of the original image can be selected with the interpolation time interposed therebetween, and discontinuous points can be improved. In the above description, the target coordinates (± 1, ± 1) are targeted, but this range may be expanded as necessary. It can also be reduced to reduce processing. For example, in the case of a general image, the background also exists in the upward direction as shown in FIG. Further, since propagation in the horizontal direction is also required, the background vector can be propagated by referring only to the coordinates on the correction target block and the left coordinates.

  Interpolation improved by correction will be described with reference to FIG. FIG. 6 is a diagram for explaining how the amount of background motion changes, and shows that the problem of the interpolated image of FIG. 5 can be improved by the present invention. The process up to the extraction of the analogy vector is the same as that described in FIG.

  As a result of vector detection between the original image f (t-1) and the original image f (t), a vector 5-11 shown in FIG. 5 is an analogy vector. In this case, in the correction process, the object vector is selected by chance because it is coincident with the object vector. In this case, the object vector cannot be improved, and the interpolation vector 6-11 in FIG. 6 is determined.

As a result of vector detection between the original image f (t) and the original image f (t-1), a vector 5-12 shown in FIG. 5 is an analogy vector. In this case, in the correction process, the difference between the object vector and the background vector happens to be the same, and either the vector 6-12A or 6-12B in FIG. 6 is selected, and this cannot necessarily be improved. (In this case, it is possible to give priority to a motion that continues the past motion, or to select more vectors in the surroundings.)
As a result of vector detection between the original image f (t + 1) and the original image f (t + 2), the vector 5-13 shown in FIG. 5 becomes an analogy vector. The vector correction unit 7 compares the detected vector of a block within a predetermined range adjacent to the interpolation block to which the analogy vector 5-13 is assigned with the analogy vector 5-13, and determines the analogy vector 5-13. Replace with the detection vector closest to the analogy vector 5-13. In this case, in the correction process, the analogy vector 5-13 is determined to be close to the background vector 5-8A as a result of comparison with the detection vector of the adjacent block, and is corrected to the vector 6-13 in FIG. This improves the accuracy of the interpolation. The predetermined range can be arbitrarily set according to the required accuracy.

  As a result of vector detection between the original image f (t + 2) and the original image f (t + 3), the vector 5-14 shown in FIG. 5 becomes an analogy vector. The vector correction unit 7 compares the detected vector of a block within a predetermined range adjacent to the interpolation block to which the analogy vector 5-14 is assigned with the analogy vector 5-14, and determines the analogy vector 5-14. Replace with the detection vector closest to the analogy vector 5-14. In this case, in the correction process, the analogy vector 5-14 is determined to be close to the background vector 5-10A as a result of comparison with the detection vector of the adjacent block, and is corrected to the vector 6-14 in FIG. This improves the accuracy of the interpolation.

  As described above, when the difference between the object vector and the background vector is larger than the amount of change in motion, discontinuous points in the background of the interpolation can be deleted, and a high-quality interpolation can be obtained.

  Note that the description has been made so far on the basis of the past original image in terms of time, but a configuration in which detection is performed toward the past original image on the basis of the temporally future original image may be used. That is, in the above description, for example, the motion vector for the original image f (t) is detected from the original image f (t−1) and assigned to the interpolated image. The configuration may be such that a motion vector for the original image f (t−1) is detected and assigned to the interpolated image. Since only the past and the future are considered to be the opposite, the explanation is omitted here.

[Embodiment 2]
(Outline of the image interpolation device 3)
FIG. 7 is a block diagram showing the configuration of the image interpolation apparatus 3 according to the present invention. The image interpolation apparatus 3 according to the present invention includes a delay unit 13 (video data delay unit), a motion vector detection unit 14 (forward motion vector detection unit, first detection unit), and a motion vector detection unit 15 (backward motion vector detection). Section, second detection means), bidirectional vector comparison section 16 (first determination means, second determination means), delay section 17 (vector delay means), vector allocation section 18 (allocation means), and statistical processing section 19 (first 1 analogy means, 2 analogy means), delay part 20 (past side analogy vector delay part), delay part 21 (future side analogy vector delay part), vector correction part 22 (correction means) and drawing part 23 (interpolated image) Generating means) and a delay unit 24 (drawing data delay unit).

  The image interpolation device 3 includes an original image f (t + 1) (second original image, future image) at time t + 1 included in a video signal input from the outside, and an original image f (t) at the previous time t. An interpolated image is generated from (first original image, past image). In the image interpolating apparatus 3, the video signal from the outside receives a delay unit 13 (video data delay unit), a motion vector detection unit 14 (forward motion vector detection unit), and a motion vector detection unit 15 (backward motion vector detection unit). Is input. Hereinafter, processing when a video signal representing the original image f (t + 1) at time t + 1 is input from the outside will be described.

  The delay unit 13 (video data delay means) delays the input external video signal, and the motion vector detection unit 14 (forward motion vector detection unit), motion vector detection unit 15, drawing unit 23, and delay unit 24 (drawing data delay unit). At a timing when a video signal representing the original image f (t + 1) is input from the outside, the delay unit 13 outputs a video signal representing the original image f (t). The delay unit 13 is configured by a memory such as DDR, for example.

  The motion vector detection units 14 and 15 calculate bidirectional motion vectors using the original image f (t) and the original image f (t−1) as reference images, respectively. Further, as shown in FIG. 2, the motion vector detection units 14 and 15 divide the original image into a plurality of blocks including a plurality of pixels, and detect a motion vector in units of blocks. The motion vector detection units 14 and 15 calculate a motion vector by block matching or a gradient method, but the motion vector may be calculated by another method. The motion vector detection units 14 and 15 output the detected motion vector to the bidirectional vector comparison unit 16 and the statistical data processing unit 19.

  The bidirectional vector comparison unit 16 compares the motion vectors detected by the motion vector detection units 14 and 15, and identifies “appearing area” and “disappearing area” (correction target block) based on the comparison result. Then, the bidirectional vector comparison unit 16 sends information specifying the “appearing region” and “disappearing region” together with the motion vectors detected by the motion vector detection units 14 and 15 to the delay unit 17 (motion vector delay unit) and the statistics. A vector assigned to the data processing unit 19 is output. The vector to be allocated may be a forward motion vector or a backward motion vector, but it is preferable to use a vector that is detected on the basis of the temporally closer side of the interpolation.

  The delay unit 17 (motion vector delay unit) receives the vector to be allocated, delays it until calculation of the analogical vector on the future side is completed, and outputs it to the vector allocation unit 18.

  The statistical data processing unit 19 performs statistical processing on the information from the motion vector information from the motion vector detection units 14 and 15, calculates motion vector statistical data (for example, a histogram), and outputs an analogy vector. In general, the “disappearing area” and the “appearing area” have a certain size. By statistically processing, it is possible to make an analogy for each region, and it is not necessary to have an analogy vector with the size of the total number of pixels and the total number of blocks, and the scale can be reduced. The statistical data processing unit 18 outputs the future analogy vector fV (t) to the vector allocation unit 17 and outputs the past analogy vector pV (t) to the delay unit 19 (analogous vector delay unit).

  The delay units 20 and 21 exist to match the time difference because the past analogy vector needs to be obtained from the detection result immediately before the assignment. The past analogy vector with the temporal phase matched is output to the vector allocation unit 18.

  When the vector allocating unit 18 should represent only the detection vector V (t−1) that has been temporally adjusted by the delay unit 17, the delay unit 20, and the delay unit 21, information on the replacement area, and the future image. Based on the vector value fV (t) to be replaced and information on the vector pV (t-2) to be replaced when it should be expressed only by the past image, the interpolation vector Vi (t-1) corresponding to the interpolation time ) And output to the vector correction unit 22. At this time, it is preferable to add information as a flag to the interpolation vector so that the vector correction unit 22 can recognize the replaced region. It is also preferable to add information on an unallocated area.

  The vector correction unit 22 compares the replacement vector with the vector of the block adjacent to the block on the interpolation image with respect to the region to which the replacement vector is assigned by the interpolation vector Vi (t−1). Among the vectors of the adjacent blocks, the one having the smallest difference between the replacement vector and the vector is calculated for each block. When the calculated vector is mV (t−1), the block interpolation vector value fV (t) or pV (t−2) is replaced with mV (t−1). When a non-allocated area also has a replacement area in an adjacent block, the non-allocated area is treated as a replacement area, and the value of fV (t) or pV (t−2) is handled by adapting the value on the adjacent block. If the vector difference with any adjacent block is larger than the threshold value, it is preferable not to perform the replacement process. When vectors similar to different fV (t) or pV (t−2) exist in a plurality of adjacent blocks, an average value of similar vectors may be taken.

  Then, based on the motion vector allocation information from the vector allocation unit 21, the drawing unit 23 reads the video signal at each coordinate of the interpolated image from the video signal of the original image, and generates an interpolated image.

(Image interpolation device 1 and image interpolation device 3)
In the image interpolation device 1 and the image interpolation device 3, there are motion vector detection units 14 and 15 and a bidirectional vector comparison unit 16 for extracting “disappearing region” and “appearing region” in order to obtain vectors in both directions. The bidirectional vector comparison unit 16 will be described. The other parts are the same as those of the image interpolation device 1 and are therefore omitted.

  The bidirectional vector comparison unit 16 uses the original image f (), which is the end point of each forward motion vector, for the forward motion vector starting from each detection block of the original image f (t) detected by the motion vector detection unit 14. It is compared with the backward motion vector starting from the detection block at t + 1), and it is determined whether or not the opposing forward motion vector and backward motion vector are opposite vectors having the same magnitude. That is, it is determined whether or not the end point of the backward motion vector starting from the end block of the forward motion vector matches the start point of the forward motion vector.

An example of comparison processing in the bidirectional vector comparison unit 16 will be described with reference to FIG. FIG. 8 is a diagram for explaining a bidirectional vector comparison method and is a diagram showing coordinates to be compared. Assume that when detecting f (t) from the original image f (t-1), the vector detection result of the forward detection of a certain block on f (t-1) is (Vxft1, Vyft1). In FIG. 8, 8-1 to 8-4 are fixed blocks on f (t) that perform backward detection from f (t) to f (t−1). 8-5 is an area allocated on f (t) when the vector detection result of the forward detection of a block on f (t-1) indicates (Vxft1, Vyft1). Further, when the backward vectors obtained in 8-1 to 8-4 are (Vxbt1, Vybt1) to (Vxbt4, Vybt4), the conditions for determining that they are equal in the comparison process performed by the comparison unit 16 are as follows. If Vxth and Vyth are judgment values that can be changed by setting, | Vxft1 + Vxbt1 | <Vxth
｜ Vxft1 + Vxbt2 | <Vxth
｜ Vxft1 + Vxbt3 | <Vxth
｜ Vxft1 + Vxbt4 | <Vxth
Is true, and | Vyft1 + Vybt1 | <Vyth
| Vyft1 + Vybt2 | <Vyth
｜ Vyft1 + Vybt3 | <Vyth
｜ Vyft1 + Vybt4 | <Vyth
This refers to the case where one of the above holds. In other words, when the difference between the back direction vector of some block around the end point indicated by the front direction detection vector and a certain value or less Is ideal).

  Here, the bidirectional vector comparison block is a 2 × 2 area, but depending on the calculation accuracy, a 3 × 3 area or a single block may be used.

  More specifically, the bidirectional vector comparison unit 16 (in the following description, the comparison block is described as a single block for simplification), and the detection blocks 9-14 to 9-26 shown in FIG. The forward motion vector to be compared is compared with the backward motion vector starting from the detection block of the end point of each forward motion vector. FIG. 9 is a diagram for explaining a bidirectional vector comparison method and for explaining the matching condition.

  For example, the end point of the forward motion vector 9-40 starting from the detection block 9-25 of the original image f (t-1) is the detection block 9-38 of the original image f (t). The unit 15 compares the backward motion vector 9-41 starting from the detection block 9-38 with the forward motion vector 9-40. In this case, the bidirectional vector comparison unit 15 is both a zero vector for the forward motion vector 9-40 and the backward motion vector 9-41, and the start point of the forward motion vector and the forward motion vector It is determined that the end point of the backward motion vector starting from the end point is equal.

  Further, for example, the end point of the forward motion vector 9-42 starting from the detection block 9-24 of the original image f (t-1) is the detection block 9-38 of the original image f (t). The vector comparison unit 16 compares the backward motion vector 9-41 starting from the detection block 9-38 with the forward motion vector 9-42. In this case, the bidirectional vector comparison unit 16 determines that the start point of the forward motion vector 9-24 is different from the end point of the backward motion vector starting from the end point of the forward motion vector.

  The bidirectional vector comparison unit 16 performs the same determination process for all detection blocks of the original image f (t−1). In the example illustrated in FIG. 9, the bidirectional vector comparison unit 16 determines the forward motion vector start point and the forward motion vector end point for the forward motion vector starting from the detection blocks 5-21 to 5-24. It is determined that the end point of the backward motion vector as the start point is different.

  When the start point of the forward motion vector and the end point of the backward motion vector starting from the end point of the forward motion vector are the same, the detection block of the original image f (t) indicated by the forward motion vector, The detection block of the original image f (t−1) indicated by the directional motion vector is the same object region or the same background region.

  On the contrary, when the start point of the forward motion vector is different from the end point of the backward motion vector starting from the end point of the forward motion vector, the original image f (t−1) that is the start point of the forward motion vector This means that the object or background area displayed in the detection block is not displayed in the original image f (t).

  That is, the forward motion vector starting from the detection block in the original image f (t−1), and the backward motion vector starting from the detection block in the original image f (t) that is the end point of the forward motion vector. To determine whether the forward motion vector and the backward motion vector are opposite in direction and equal in magnitude, thereby obtaining the original image f (t−1) at the previous time in time. Is displayed, but the “disappearing area” that is not displayed in the original image f (t) at a later time in time can be detected. In the example illustrated in FIG. 9, the detection blocks 9-21 to 9-24 in the original image f (t−1) are identified as “disappearing areas”.

  Note that the above-described determination processing in the bidirectional vector comparison unit 16 may have a width in consideration of motion vector detection accuracy. That is, whether the interval between the start point of the forward motion vector and the end point of the backward motion vector starting from the end block of the forward motion vector is greater than a predetermined threshold (within a predetermined range) It is also possible to adopt a configuration in which it is determined whether or not the area disappears when the threshold value is larger than a predetermined threshold. Here, the predetermined threshold (predetermined range) may be configured such that the number of blocks corresponding to the motion vector detection accuracy can be set, and is not particularly limited.

  In the video interpolating apparatus 1, the determination is made with the forward vector 9-40 and the forward direction 9-42 competing in the block 9-38. In this case, it is impossible to determine which of the 9-24 and 9-25 vectors is on the “disappearing area” side, and both 9-24 and 9-25 are set as correction target areas. The video interpolation device 3 can clearly determine the region.

(Modification)
In the image interpolating apparatus 1 and the image interpolating apparatus 3, when drawing an interpolation between the original image f (t) and the original image f (t + 1) in order to extract an analogy vector, the original image f (t-1) And the detection of the original image f (t) and the detection results of the original image f (t + 1) and the original image f (t + 2) are required. In the image interpolation device 3, the number of detections necessary for one interpolation process required can be reduced by changing the processing in the statistical data processing unit 19 of FIG.

  The statistical data processing unit 19 refers to the result of the bidirectional vector comparison unit on the coordinates of the original image f (t) in the detection of the original image f (t) and the original image f (t + 1). In the detection of the original image f (t-1) and the original image f (t), the detection of the original image f (t) and the original image f (t + 1) The motion vector of the “appearing region” is defined as the forward motion vector of the detection of the original image f (t) and the original image f (t + 1) in the region where the results of the bidirectional vector comparison match. In the detection of the original image f (t-1) and the original image f (t), in the region where the result of the bidirectional vector comparison is the same, the detection of the original image f (t) and the original image f (t + 1) is bidirectional. The backward motion vector detection of the original image f (t−1) and the original image f (t) in the region where the vector comparison results do not match is set as the “disappearing region” motion vector.

The coordinates of “disappearing area” and “appearing area” exist at neighboring points continuously between frames. For example, in FIG. 9, “disappearing areas” are 9-4 to 9-7 on the original image f (t−2), and 9-21 to 9-24 on the original image f (t−1). It becomes. In the original image f (t), it is 9-38 to 9-41. When an area disappearing by overlapping adjacent frames is viewed, the coordinates of the “disappearing area” when the original image f (t−2) and the original image f (t−1) are overlapped are the original image f (t− It becomes 9-4 to 9-11 on the coordinates of 2). The same applies to the “appearing area”. (In this example, “disappearing areas” are adjacent between frames, but when the background moves, a shift occurs, but it exists in a neighboring area.)
The statistical data processing unit 19 performs a statistical process such as a histogram on the occurrence frequency of the vector detected as the motion vector of the “disappearing region” and the motion vector of the “appearing region” using the bidirectional vector comparison. An analogy vector for “disappearing area” and an analogy vector of “appearing area” are extracted.

  This will be described more specifically with reference to FIG. FIG. 10 is a diagram for explaining the operation of a statistical data processing unit 19 (comparison means, first mismatch block specifying means, second mismatch block specifying means), which is a modification of the image interpolation device 3 according to the present invention. A case where an original image f (t + 0.5) between the original image f (t) and the original image f (t + 1) is created will be described. At this time, attention is paid to the coincidence / non-coincidence of the bidirectional vectors of the detection block on the original image f (t) and the original image f (t + 1). Also, the detection result of the previous frame (the detection result between the original image f (t−1) and the original image f (t)), and the coincidence of the bidirectional vectors in the detection block on f (t) / Use mismatch results as reference values.

  First, the analogy vector of the “appearing region” is a bi-directional vector (second forward motion vector and second rearward vector) in vector detection between the original image f (t−1) and the original image f (t). Directional motion vectors) are mismatched, and bidirectional vectors (first forward motion vector and first backward motion vector) are detected in vector detection between the original image f (t) and the original image f (t + 1). ) Is a vector in a block on the original image f (t) (in FIG. 10, the block 10-14 corresponds to the first non-matching block in the claims), and the original image f (t) And the vector obtained from the original image f (t) obtained at the time of vector detection between the original image f (t + 1) and the original image f (t + 1).

  In other words, the analogy vector of “appearing area” is an area where the bidirectional vector does not match (corresponding to “X” in FIG. 10) in the past detection on the original image, and the bidirectional vector is not detected in the current detection. This is a region indicating a coincidence (corresponding to “◯” in FIG. 10), that is, a future-oriented vector 10-41 of the current detection in the block 10-14.

  A block 10-14 is a block that does not match as a result of the bidirectional vector comparison in the detection between the original image f (t-1) and the original image f (t), and is on the future side. Therefore, there is an “appearing region” between the original image f (t−1) and the original image f (t). The motion vector of the original image f (t) and the original image f (t + 1) of this block is detected as 10-41 between the original image f (t) and the original image f (t + 1), and bidirectional vector comparison is performed. The results also agree.

  Therefore, it can be expected that the motion vector of the “appearing region” in the vicinity (predetermined range) indicates the motion of the block 10-41. This vector is output (in the reverse direction) as an analogy vector of blocks 10-28 to 10-31 detected as “appearing regions” in the original image f (t) and the original image f (t + 1). In addition, the said predetermined range is set according to a specification, for example.

  On the other hand, the analogy vector of the “disappearing region” is the same as that of the original image f (t) in the vector detection between the original image f (t−1) and the original image f (t). Blocks on the original image f (t) in which bidirectional vectors do not match in vector detection with the original image f (t + 1) (in FIG. 10, blocks 10-21 to 10-24, and claims) To the original image f (t) obtained at the time of vector detection between the original image f (t-1) and the original image f (t). It is a vector (reverse direction) obtained toward t-1).

In other words, the analogy vector of “disappearing area” is the area where the bidirectional vector coincides with the previous detection on the original image (corresponding to “◯” in FIG. 10), and the bidirectional vector does not coincide with the current detection. This is an area indicating (corresponding to “x” in FIG. 10), that is, a past direction vector 10-40 of past detection in the blocks 10-21 to 10-24. (The vector 10-40 represents a value representing the past direction vector of the blocks 10-21 to 10-24.)
Blocks 10-15 to 10-20 perform bidirectional vector comparison both when detecting the original image f (t-1) and the original image f (t) and when detecting the original image f (t) and the original image f (t + 1). Therefore, it is not the target of the analogy vector.

  Blocks 10-21 to 10-24 match in the comparison of bidirectional vectors in detection between the original image f (t-1) and the original image f (t), and the original image f (t) and the original image f (t) match. In the detection with the image f (t + 1), the bidirectional vector comparison does not match.

  Therefore, the motion vector of the “disappearing region” in the vicinity (predetermined range) can be expected to move 10-40. This vector is output (in the reverse direction) as an analogy vector of 10-21 to 10-24 detected as “disappearing area” in the original image f (t) and the original image f (t + 1). In addition, the said predetermined range is set according to a specification, for example.

  Here, in drawing an interpolated image, it is determined whether the original image f (t) or the original image f (t + 1) is used for an unallocated area. The determination is based on the original image coordinates of the original image f (t) and the original image f (t + 1) indicated by the zero vector from the unassigned interpolation points, and counting the number of matching / mismatching blocks of both vectors in a certain area. To do.

  In the interpolated image f (t + 0.5), the unallocated areas are the interpolation blocks 10-43, 10-44, 10-51, and 10-52. Here, the match / mismatch of the bidirectional vectors in the past and future original image areas indicated by the zero vector from the interpolation block 10-43 is compared.

  In FIG. 10, at the interpolation points corresponding to the blocks 10-15 to 10-18 and the blocks 10-28 to 10-31 (and their peripheral points), the mismatch point of the bidirectional vector is the original image f (t + 1). Exists only in That is, there are too many blocks only on the future side, and it is determined that this unallocated interpolation point is an “appearing region”.

  That is, the bidirectional vectors around the block 10-15 of the original image indicate a match, and the bidirectional vectors around the block 10-28 of the original image indicate a mismatch. Since there is a surplus area on the future side, it is determined that 10-43 is an “appearing area”. Therefore, an analogy vector 10-41 of “region to appear” is applied to the interpolation block 10-43, and it is determined to draw only from the future. The same applies to the interpolation block 10-44.

  On the other hand, at the interpolation points corresponding to the blocks 10-21 to 10-24 and the blocks 10-34 to 10-37 (and their peripheral points), the mismatch point of the bidirectional vector exists only in the original image f (t). To do. That is, there are too many blocks only on the past side, and it is determined that this unallocated interpolation point is a “disappearing area”.

  In other words, the match / mismatch of the bidirectional vectors in the past and future original image areas indicated by the zero vector from the interpolation block 10-51 is compared. A bidirectional vector around the block 10-23 of the original image indicates a mismatch, and a bidirectional vector around the block 10-36 of the original image indicates a match. Since there is a surplus area on the past side, it is determined that the block 10-51 of the interpolated image is an “erasing area”. Therefore, the analogy vector 10-40 of “disappearing area” is applied to the block 10-51, and it is determined to draw only from the past. The same applies to the block 10-52 of the interpolated image.

  The analogy vector obtained above is applied to the “disappearing area” and “appearing area” determined by the interpolation point. Further, similar to the image interpolating apparatus 1, it is compared with a vector of peripheral points, and a replacement process for acceleration is performed.

  The analogy vector is a vector of “regions that appear” from the detection results of the original image f (t−1) and the original image f (t) and the detection results of the original image f (t) and the original image f (t + 1) for each fixed region. An average value or a high frequency of vectors determined to correspond to the “disappearing region” vector is used as an analogy vector of the region. Each of “appearing area” and “disappearing area” may be individually provided or may have one representative value.

  In this description, the drawing direction is determined by comparing the number of inconsistencies of bidirectional vectors with respect to the unallocated area, centering on the zero vector, but the original image f (t) and the original image f ( It is also possible to use a method of comparing the number of inconsistencies in bidirectional vectors centered on the area of t + 1).

  As described above, when the vectors of the original image f (t) and the original image f (t + 1) are detected, the vector detection between the original image f (t−1) and the original image f (t) is performed one frame before. Is completed, and the result is used. In that sense, a delay unit is required for the vector detection result. In this case, since the detection result of the original image f (t + 1) and the original image f (t + 2) is not necessary in creating the interpolated image f (t + 0.5), it was detected in the past (one frame before). The result may be used, eliminating the need for vector detection again.

  Therefore, in the modification of the present invention, it is possible to create an interpolated image without using future detection results. In FIG. 7, the delay unit 17 and the delay unit 21 are not necessary, and an equivalent result can be obtained in a smaller format.

  In addition, the image interpolation devices 1 and 3 according to the present invention may be configured, for example, in a television receiver to interpolate an image to be displayed based on a video signal included in a received broadcast signal. . Alternatively, the image interpolation device 1 according to the present invention is provided in a video playback device such as a DVD playback device or a video playback device, and interpolates an image to be played back based on video data recorded on a recording medium. There may be.

  The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

  Finally, each block of the image interpolation apparatus 1 may be configured by hardware logic, or may be realized by software using a CPU as follows.

  In other words, the image interpolation apparatuses 1 and 3 include a central processing unit (CPU) that executes instructions of a control program that realizes each function, a read only memory (ROM) that stores the program, and a random access memory (RAM) that expands the program. memory), a storage device (recording medium) such as a memory for storing the program and various data. An object of the present invention is to provide a recording medium on which a program code (execution format program, intermediate code program, source program) of a control program of the image interpolation apparatus 1 which is software for realizing the above-described functions is recorded so as to be readable by a computer. This can also be achieved by supplying the image interpolation apparatus 1 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

  Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy (registered trademark) disks / hard disks, and disks including optical disks such as CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

  Further, the image interpolation apparatus 1 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Also, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The image interpolating apparatus according to the present invention can generate an interpolated image from an original image included in a video signal with high accuracy, and thus can be suitably used in a television receiver, a DVD reproducing apparatus, and the like.

DESCRIPTION OF SYMBOLS 1 Image interpolation apparatus 2 Delay part (Video data delay means)
3 Motion vector detection unit (detection means)
4 drawing direction determination unit (drawing direction determination means)
5 Delay part (vector data delay means)
6 Vector assignment part (assignment means)
7 Vector correction unit (correction means)
8 Statistical processing part (first analogy means, second analogy means)
9 Delay part (primary replacement vector delay part)
10 Delay part (second-order replacement vector delay part)
11 Drawing part (interpolated image generating means)
12 Delay part (delay means for drawing)
100 Vector Correction Device 13 Delay Unit (Video Data Delay Unit)
14 motion vector detection unit (forward motion vector detection unit, first detection means)
15 motion vector detection unit (backward motion vector detection unit, second detection means)
16 Bidirectional vector comparison unit (first determination means, second determination means)
17 Delay part (vector delay means)
18 Vector assignment section (assignment means)
19 Statistical processing unit (first analogy inference means, second analogy inference means, comparison means, first inconsistent block specifying means, second inconsistent block specifying means)
20 Delay part (Past side analogy vector delay part)
21 Delay part (Future side analogy vector delay part)
22 Vector correction unit (correction means)
23 drawing part (interpolated image generating means)
24 delay part (drawing data delay part)

Claims (13)

Assigned to each block of the interpolated image to be interpolated between the first original image corresponding to the first time and the second original image corresponding to the second time after the first time A vector correction device for correcting a vector,
Detection for detecting a motion vector starting from each block of the first original image and ending at any block of the second original image from the first original image and the second original image Means,
For each block of the interpolated image, draw only from the first original image, draw only from the second original image, or both the first original image and the second original image Drawing direction determination means for determining whether to draw from,
In accordance with the determination result of the drawing direction determination unit, an analogy vector for allocating to the undetermined block to which the motion vector detected by the detection unit is not allocated in the block of the interpolated image is before the first time. In a third original image at the third time, a first analogy is made from the appearance frequency of motion vectors starting from a block within a predetermined range from the undetermined block and ending at any block of the first original image. 1 analogy means;
According to the determination result of the drawing direction determination means, an analogy vector for allocating to the undetermined block to which the motion vector detected by the detection means is not allocated in the block of the interpolated image is later than the second time. In the fourth original image at the fourth time, an analogy is made from the appearance frequency of motion vectors starting from a block within a predetermined range from the undetermined block and ending at any block of the second original image. Two analogy means;
Allocating means for allocating the motion vector detected by the detecting means and the analogy vector estimated by the first analogizing means and the second analogizing means to each block of the interpolated image;
Correction means for replacing the analogy vector assigned to the undetermined block with a motion vector closest to the analogy vector among the motion vectors assigned to blocks within a predetermined range from the undetermined block. Feature vector correction device. Assigned to each block of the interpolated image to be interpolated between the first original image corresponding to the first time and the second original image corresponding to the second time after the first time A vector correction device for correcting a vector,
Detection for detecting a motion vector starting from each block of the second original image and ending at any block of the first original image from the first original image and the second original image Means,
For each block of the interpolated image, draw only from the first original image, draw only from the second original image, or both the first original image and the second original image Drawing direction determination means for determining whether to draw from,
In accordance with the determination result of the drawing direction determination unit, an analogy vector for allocating to the undetermined block to which the motion vector detected by the detection unit is not allocated in the block of the interpolated image is before the first time. In a third original image at the third time, a first analogy is made from the appearance frequency of motion vectors starting from a block within a predetermined range from the undetermined block and ending at any block of the first original image. 1 analogy means;
According to the determination result of the drawing direction determination means, an analogy vector for allocating to the undetermined block to which the motion vector detected by the detection means is not allocated in the block of the interpolated image is later than the second time. In the fourth original image at the fourth time, an analogy is made from the appearance frequency of motion vectors starting from a block within a predetermined range from the undetermined block and ending at any block of the second original image. Two analogy means;
Allocating means for allocating the motion vector detected by the detecting means and the analogy vector estimated by the first analogizing means and the second analogizing means to each block of the interpolated image;
Correction means for replacing the analogy vector assigned to the undetermined block with a motion vector closest to the analogy vector among the motion vectors assigned to blocks within a predetermined range from the undetermined block. Feature vector correction device. Assigned to each block of the interpolated image to be interpolated between the first original image corresponding to the first time and the second original image corresponding to the second time after the first time A vector correction device for correcting a vector,
A forward motion vector starting from each block of the first original image and ending at any block of the second original image is detected from the first original image and the second original image. First detecting means for
A backward motion vector starting from each block of the second original image and ending at any block of the first original image is detected from the first original image and the second original image. Second detecting means for
When the interval between the start point of the forward motion vector and the end point of the backward motion vector starting from the end point of the forward motion vector is larger than a predetermined threshold, the forward motion vector is First determination means for determining that the block is not allocated;
When the interval between the start point of the backward motion vector and the end point of the forward motion vector starting from the end point of the backward motion vector is larger than a predetermined threshold, the backward vector is added to the block of the interpolated image. Second determination means for determining that no assignment is made;
For each block of the interpolated image, draw only from the first original image, draw only from the second original image, or both the first original image and the second original image Drawing direction determination means for determining whether to draw from,
In accordance with the determination result of the drawing direction determination means, the forward motion vector and the backward motion vector detected by the first detection means and the second detection means in the block of the interpolated image are assigned to an undetermined block. An analogy vector for the first original image starting from a block within a predetermined range from the undetermined block in the third original image at a third time prior to the first time. A first analogy means for inferring from the appearance frequency of a motion vector whose end point is a block;
In accordance with the determination result of the drawing direction determination means, the forward motion vector and the backward motion vector detected by the first detection means and the second detection means in the block of the interpolated image are assigned to an undetermined block. For the analogy vector for this, any of the second original images starts from a block within a predetermined range from the undetermined block in the fourth original image at a fourth time after the second time. A second analogy means for inferring from the appearance frequency of a motion vector whose end point is a block;
Allocating means for allocating the motion vectors detected by the first detecting means and the second detecting means and the analogy vectors estimated by the first analogizing means and the second analogizing means to each block of the interpolated image; ,
Correction means for replacing the analogy vector assigned to the undetermined block with a motion vector closest to the analogy vector among the motion vectors assigned to blocks within a predetermined range from the undetermined block. Feature vector correction device. Assigned to each block of the interpolated image to be interpolated between the first original image corresponding to the first time and the second original image corresponding to the second time after the first time A vector correction device for correcting a vector,
A first forward motion vector starting from each block of the first original image and ending at any block of the second original image from the first original image and the second original image. First detecting means for detecting
From the first original image and the second original image, a first backward motion vector starting from each block of the second original image and ending at any block of the first original image Second detecting means for detecting
If the interval between the start point of the first forward motion vector and the end point of the first backward motion vector starting from the end point of the first forward motion vector is greater than a predetermined threshold, the first forward direction First determination means for determining that a motion vector is not allocated to a block of the interpolated image;
If the interval between the start point of the first backward motion vector and the end point of the first forward motion vector starting from the end point of the first backward motion vector is greater than a predetermined threshold, the first backward vector Second determination means for determining that the block is not allocated to the block of the interpolated image,
A second backward motion vector starting from each block of the first original image and ending at any block of the third original image corresponding to the third time before the first time; Comparing means for comparing a second forward motion vector starting from each block of the third original image and ending at any block of the first original image;
Among the blocks of the first original image, an interval from the end point of the first backward motion vector starting from the end point of the first forward motion vector is smaller than a predetermined threshold, and the second rear image First mismatch block specifying means for specifying, as a first mismatch block, a block whose interval from the end point of the second forward motion vector starting from the end point of the direction motion vector is larger than a predetermined threshold;
Among the blocks of the first original image, an interval from the end point of the first backward motion vector starting from the end point of the first forward motion vector is larger than a predetermined threshold, and the second rear image A second non-matching block specifying means for specifying a block whose interval from the end point of the second forward motion vector starting from the end point of the direction motion vector is smaller than a predetermined threshold as a second non-matching block;
For each block of the interpolated image, draw only from the first original image, draw only from the second original image, or both the first original image and the second original image Drawing direction determination means for determining whether to draw from,
The first forward motion vector and the first backward motion vector detected by the first detection means and the second detection means in the block of the interpolated image are allocated according to the determination result of the drawing direction determination means. First analogy means for analogizing from the first forward motion vector starting from the first inconsistent block, an analogy vector for allocating to an undetermined block that is not possible;
The first forward motion vector and the first backward motion vector detected by the first detection means and the second detection means in the block of the interpolated image are allocated according to the determination result of the drawing direction determination means. Second analogy means for inferring an analogy vector for allocating to an undetermined block from the second backward motion vector starting from the second inconsistent block;
Allocating means for allocating the motion vector detected by the first detecting means and the second detecting means and the analogy vector estimated by the first analogizing means and the second analogizing means to each block of the interpolated image; ,
Correction means for replacing the analogy vector assigned to the undetermined block with a motion vector closest to the analogy vector among the motion vectors assigned to the blocks within a predetermined range from the undetermined block. Feature vector correction device. An image interpolation apparatus comprising the vector correction apparatus according to any one of claims 1 to 4,
An image interpolation apparatus comprising: an interpolated image generating unit that generates the interpolated image based on the vector corrected by the correcting unit.   A television receiver comprising the image interpolation device according to claim 5.   A video reproduction apparatus comprising the image interpolation apparatus according to claim 5. Assigned to each block of the interpolated image to be interpolated between the first original image corresponding to the first time and the second original image corresponding to the second time after the first time A vector correction method for correcting a vector,
Detection for detecting a motion vector starting from each block of the first original image and ending at any block of the second original image from the first original image and the second original image Steps,
For each block of the interpolated image, draw only from the first original image, draw only from the second original image, or both the first original image and the second original image A drawing direction determination step for determining whether to draw from,
In accordance with the determination result of the drawing direction determination step, an analogy vector for allocating an undetermined block to which the motion vector detected by the detection step is not allocated in the block of the interpolated image is before the first time. In a third original image at the third time, a first analogy is made from the appearance frequency of motion vectors starting from a block within a predetermined range from the undetermined block and ending at any block of the first original image. 1 analogy step;
According to the determination result of the drawing direction determination step, an analogy vector for allocating to the undetermined block to which the motion vector detected by the detection step is not allocated in the block of the interpolated image is later than the second time. In the fourth original image at the fourth time, an analogy is made from the appearance frequency of motion vectors starting from a block within a predetermined range from the undetermined block and ending at any block of the second original image. Two analogy steps;
An allocation step for allocating the motion vector detected by the detection step and the analogy vector estimated by the first analogy step and the second analogy step to each block of the interpolated image;
A correction step of replacing the analogy vector assigned to the undetermined block with a motion vector closest to the analogy vector among the motion vectors assigned to the blocks within a predetermined range from the undetermined block. Feature vector correction method. Assigned to each block of the interpolated image to be interpolated between the first original image corresponding to the first time and the second original image corresponding to the second time after the first time A vector correction method for correcting a vector,
Detection for detecting a motion vector starting from each block of the second original image and ending at any block of the first original image from the first original image and the second original image Steps,
For each block of the interpolated image, draw only from the first original image, draw only from the second original image, or both the first original image and the second original image A drawing direction determination step for determining whether to draw from,
In accordance with the determination result of the drawing direction determination step, an analogy vector for allocating an undetermined block to which the motion vector detected by the detection step is not allocated in the block of the interpolated image is before the first time. In a third original image at the third time, a first analogy is made from the appearance frequency of motion vectors starting from a block within a predetermined range from the undetermined block and ending at any block of the first original image. 1 analogy step;
According to the determination result of the drawing direction determination step, an analogy vector for allocating to the undetermined block to which the motion vector detected by the detection step is not allocated in the block of the interpolated image is later than the second time. In the fourth original image at the fourth time, an analogy is made from the appearance frequency of motion vectors starting from a block within a predetermined range from the undetermined block and ending at any block of the second original image. Two analogy steps;
An allocation step for allocating the motion vector detected by the detection step and the analogy vector estimated by the first analogy step and the second analogy step to each block of the interpolated image;
A correction step of replacing the analogy vector assigned to the undetermined block with a motion vector closest to the analogy vector among the motion vectors assigned to the blocks within a predetermined range from the undetermined block. Feature vector correction method. Assigned to each block of the interpolated image to be interpolated between the first original image corresponding to the first time and the second original image corresponding to the second time after the first time A vector correction method for correcting a vector,
A forward motion vector starting from each block of the first original image and ending at any block of the second original image is detected from the first original image and the second original image. First detecting step,
A backward motion vector starting from each block of the second original image and ending at any block of the first original image is detected from the first original image and the second original image. A second detecting step,
When the interval between the start point of the forward motion vector and the end point of the backward motion vector starting from the end point of the forward motion vector is larger than a predetermined threshold, the forward motion vector is A first determination step for determining that the block is not allocated;
When the interval between the start point of the backward motion vector and the end point of the forward motion vector starting from the end point of the backward motion vector is larger than a predetermined threshold, the backward vector is added to the block of the interpolated image. A second determination step for determining not to allocate;
For each block of the interpolated image, draw only from the first original image, draw only from the second original image, or both the first original image and the second original image A drawing direction determination step for determining whether to draw from,
According to the determination result of the drawing direction determination step, the forward motion vector and the backward motion vector detected by the first detection step and the second detection step in the block of the interpolated image are allocated to an undetermined block to which no allocation is made. An analogy vector for the first original image starting from a block within a predetermined range from the undetermined block in the third original image at a third time prior to the first time. A first analogizing step for analogizing from the appearance frequency of motion vectors having a block as an end point;
According to the determination result of the drawing direction determination step, the forward motion vector and the backward motion vector detected by the first detection step and the second detection step in the block of the interpolated image are allocated to an undetermined block to which no allocation is made. For the analogy vector for this, any of the second original images starts from a block within a predetermined range from the undetermined block in the fourth original image at a fourth time after the second time. A second analogy step for inferring from the appearance frequency of the motion vector whose end point is the block;
An allocation step for allocating the motion vector detected by the detection step and the analogy vector estimated by the first analogy step and the second analogy step to each block of the interpolated image;
A correction step of replacing the analogy vector assigned to the undetermined block with a motion vector closest to the analogy vector among the motion vectors assigned to the blocks within a predetermined range from the undetermined block. Feature vector correction method. Assigned to each block of the interpolated image to be interpolated between the first original image corresponding to the first time and the second original image corresponding to the second time after the first time A vector correction method for correcting a vector,
A first forward motion vector starting from each block of the first original image and ending at any block of the second original image from the first original image and the second original image. A first detection step of detecting
From the first original image and the second original image, a first backward motion vector starting from each block of the second original image and ending at any block of the first original image A second detection step of detecting
If the interval between the start point of the first forward motion vector and the end point of the first backward motion vector starting from the end point of the first forward motion vector is greater than a predetermined threshold, the first forward direction A first determination step for determining not to allocate a motion vector to the block of the interpolated image;
If the interval between the start point of the first backward motion vector and the end point of the first forward motion vector starting from the end point of the first backward motion vector is greater than a predetermined threshold, the first backward vector A second determination step that determines not to allocate a block to the interpolated image block;
A second backward motion vector starting from each block of the first original image and ending at any block of the third original image corresponding to the third time before the first time; A comparison step of comparing a second forward motion vector starting from each block of the third original image and ending at any block of the first original image;
Among the blocks of the first original image, an interval from the end point of the first backward motion vector starting from the end point of the first forward motion vector is smaller than a predetermined threshold, and the second rear image A first non-matching block specifying step for specifying a block having an interval from the end point of the second forward motion vector starting from the end point of the directional motion vector as a first non-matching block, which is larger than a predetermined threshold;
Among the blocks of the first original image, an interval from the end point of the first backward motion vector starting from the end point of the first forward motion vector is larger than a predetermined threshold, and the second rear image A second non-matching block specifying step for specifying, as a second non-matching block, a block whose interval from the end point of the second forward motion vector starting from the end point of the direction motion vector is smaller than a predetermined threshold;
For each block of the interpolated image, draw only from the first original image, draw only from the second original image, or both the first original image and the second original image A drawing direction determination step for determining whether to draw from,
The first forward motion vector and the first backward motion vector detected by the first detection step and the second detection step in the block of the interpolated image are allocated according to the determination result of the drawing direction determination step. A first analogizing step for analogizing an analogy vector to be assigned to an undetermined block from the first forward motion vector starting from the first inconsistent block;
The first forward motion vector and the first backward motion vector detected by the first detection step and the second detection step in the block of the interpolated image are allocated according to the determination result of the drawing direction determination step. A second analogizing step for analogizing an analogy vector to be assigned to an undetermined block from the second backward motion vector starting from the second inconsistent block;
An allocation step for allocating the motion vectors detected by the first detection step and the second detection step and the analogy vectors estimated by the first analogy step and the second analogy step to each block of the interpolated image; ,
A correction step of replacing the analogy vector assigned to the undetermined block with a motion vector closest to the analogy vector among the motion vectors assigned to the blocks within a predetermined range from the undetermined block. Feature vector correction method.   A control program for operating the vector correction apparatus according to claim 1, wherein the computer functions as each of the means.   A computer-readable recording medium in which the control program according to claim 12 is recorded.

Images