JP2008193462A - Frame interpolation apparatus and frame interpolation method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a frame interpolation apparatus for generating an image of interpolation frame between two frames without causing blur in motion picture and judder. <P>SOLUTION: The frame interpolation apparatus is provided with a similarity calculator for calculating the similarity for each pixel block pair comprising a pixel block BLa of a first reference frame a point-symmetrical with respect to an interpolation pixel in an interpolation frame and a pixel block BLb of a second reference frame; a reference block selector for selecting a pixel block pair having the greatest similarity as a candidate block pair, and selecting a pixel block pair having the greatest similarity from among the pixel block pairs other than a pixel block having the positional relation with respect to the interpolation pixel similar to that of the candidate block as an evaluation block pair, and deciding whether the center position of the candidate block pair is selected as a reference pixel position or the reference pixel position selected one frame before is selected on the basis of the similarity between the candidate block pair and a block pair for evaluation. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a frame interpolation apparatus and a frame interpolation method for generating an image of an interpolated frame existing between two frames using two frames of images included in a video signal.

  A hold-type display such as a liquid crystal display continues to display the same image for one frame period, and when the object in the image moves, the tracking of the human eye to the moving object moves continuously, There is a problem that the edge portion looks blurry because the object moves discontinuously in units of one frame. On the other hand, it is conceivable to increase the number of display frames by interpolating frames to make the object move smoothly.

  In addition, for materials in which film images such as movies are converted into television signals, two (2 fields) or 3 (fields) are created from the same frame because of the difference in frame frequency between the two (film images and television signals). If the image is displayed as it is, there is a problem that the motion is blurred or judder with a jerky motion occurs.

  Similarly, a material obtained by converting computer-processed video into a TV signal is an image in which two (2 fields) are made from the same frame.

  The conventional frame interpolation apparatus and frame interpolation method is a zero hold method in which interpolation is performed with the same image as the previous frame with respect to the interpolated frame, or between the previous frame image and the next frame image with respect to the interpolated frame. Although there is an average value interpolation method for interpolating with an average image, the zero hold method does not move smoothly with respect to an image moving in a fixed direction, so the problem of blurring of the hold type display is still not solved. Further, the average value interpolation method has a problem that a moving image becomes a double image.

  As an improvement measure, an image is divided into a plurality of pixel blocks, a frame one frame before the interpolation frame is used as a first reference frame, and a frame one frame after the interpolation frame is used as a second reference frame. Find a pixel block having the highest correlation with the pixel block in the first reference frame from the second reference frame, and interpolate an image of the pixel block of the interpolation frame from the found pixel block pair (for example, Patent Reference 1). In this method, since an interpolation frame is generated in units of pixel blocks, data read / write in a memory (mainly SDRAM is used) necessary for processing becomes complicated, and hardware implementation is difficult. There is.

  As another improvement measure, from the pixel having the largest correlation between the pixel on the first reference frame and the pixel on the second reference frame that is point-symmetric with respect to the interpolation pixel of the interpolation frame. Some generate an interpolation pixel of an interpolation frame (see, for example, Patent Document 2). In this method, because correlation is detected in units of pixels, it may be detected that the correlation between pixels is large even though the contents of the image are different, and an interpolated image may not be obtained correctly.

JP 2004-357215 A (page 9, FIG. 16) Japanese Patent Laying-Open No. 2006-129181 (page 8, FIG. 3)

  The conventional frame interpolating apparatus is configured as described above, and there is a problem that motion is blurred or judder in which motion is jerky occurs. In addition, in the method of generating an interpolation frame in units of blocks, there is a problem that data read / write with respect to the memory becomes complicated and hardware implementation is difficult. In addition, there is a problem that the correlation cannot be detected correctly in the method of detecting the correlation in pixel units.

The present invention
A frame interpolation device for generating a pixel value of an interpolation pixel in an interpolation frame between a first reference frame and a second reference frame,
A pixel block pair composed of a pixel block of the first reference frame and a pixel block of the second reference frame which are in point symmetry with respect to the position of the interpolation pixel, and one of the pixel block pair Extraction means for extracting a plurality of one or both of the pixel block pairs in which the pixel block is replaced with a pixel block shifted by one pixel;
For each of a plurality of pixel block pairs extracted by the extraction means, similarity calculation means for calculating the similarity between two pixel blocks based on the pixel values of the pixels in the two pixel blocks;
Among the plurality of block pairs extracted by the extracting unit, the similarity calculated by the similarity calculating unit or a pixel block pair having the largest correction is selected as a candidate block pair, and Among the plurality of block pairs extracted by the extracting means, the relative positional relationship with respect to the interpolation pixel is other than the relative positional relationship with respect to the interpolation pixel of the candidate block pair, Block pair selection means for selecting a pixel block pair having the largest similarity or a corrected one as a block pair for evaluation;
Similarity of the candidate block pair or a correction thereof (hereinafter referred to as “candidate similarity”) and similarity of the evaluation block pair or a correction thereof (hereinafter referred to as “evaluation similarity”) ) Based on the size of the candidate block pair, a reliability evaluation means for evaluating the reliability of the candidate block pair;
When the reliability evaluation unit determines that there is reliability, the center position of each pixel block constituting the candidate block pair is selected as a reference pixel position, and when it is determined that there is no reliability, the interpolation pixel Reference pixel position selection means for selecting a reference pixel position previously selected as a reference pixel position of
There is provided a frame interpolation device comprising: interpolation means for generating the interpolation pixel based on a pixel value at a reference pixel position selected by the reference pixel position selection means.

  According to the present invention, the position of the pixel block is located at the center of each of the pair of pixel blocks having the highest degree of similarity and thus having a mutual distance closest to the amount of image motion between the first reference frame and the second reference frame. Since the pixel value of the interpolation pixel can be determined based on the pixel value of the pixel to be interpolated, the interpolation can be performed accurately.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
Embodiment 1 FIG.
FIG. 1 is a block diagram showing a frame interpolation apparatus according to Embodiment 1 of the present invention. The illustrated frame interpolation apparatus includes an input terminal 1, a delay unit 2, an extraction unit 3, a similarity calculation unit 4, a block pair selection unit 40, a reliability evaluation unit 41, and a reference pixel position selection unit 5. , Delay means 42, symmetrical position interpolation means 6, and output terminal 7.

The video signal input from the input terminal 1 is input to the delay unit 2, delayed by one frame period, and input to one of the extraction units 3 as a first reference frame. The video signal input to the input terminal 1 is input to the other input of the extraction unit 3 as a second reference frame. The extraction means 3 is located substantially symmetrical with respect to the interpolated pixel from the first reference frame and the second reference frame, and extracts a plurality of pairs of pixel blocks made up of the same number of pixels to calculate similarity. Output to 4.
The extraction unit 3 also outputs the pixel value at the center position of each pixel block to the symmetric position interpolation unit 6.

  As shown in FIG. 2, the pixels in each frame are arranged in a matrix in the frame F (on the screen constituting the frame). That is, they are aligned with each other in the horizontal direction (main scanning direction) and the vertical direction (sub-scanning direction). The position of each pixel in the frame F is represented by (h, v) by a horizontal coordinate value h and a vertical coordinate value v. The coordinate origin is at the leftmost and uppermost position of the frame, the horizontal coordinate value h increases rightward by one for each pixel interval, and the vertical coordinate value v decreases downward. Assume that the value increases by 1 for each interval.

  When generating an interpolation pixel at an arbitrary position (h, v) in the interpolation frame, as the pixel block BLa in the first reference frame and the pixel block BLb in the second reference frame, for example, the center position thereof is in the horizontal direction. Are extracted in the range from (h-2) to (h + 2) and in the vertical direction are extracted from (v-1) to (v + 1). There are 15 pixel blocks satisfying such conditions in each of the first reference frame and the second reference frame. Each pixel block is composed of, for example, pixels of 3 rows and 5 columns.

  3A to 3D and FIGS. 4A to 4D illustrate some of such pixel blocks. Among them, what are shown in FIGS. 3A to 3D are pixel blocks BLa in the first reference frame, and those shown in FIGS. 4A to 4D are those in the second reference frame. This is a pixel block BLb.

  The pixel block BLa (1) shown in FIG. 3 (a) and the pixel block BLb (1) shown in FIG. 4 (a) constitute a pixel block pair composed of pixel blocks located at point symmetry with respect to the interpolation pixel. is doing. Similarly, a pixel block consisting of a pixel block in which the pixel block BLa (2) shown in FIG. 3B and the pixel block BLb (2) shown in FIG. Make up a pair. Similarly, a pixel block consisting of a pixel block in which the pixel block BLa (3) shown in FIG. 3C and the pixel block BLb (3) shown in FIG. 4C are symmetric with respect to the interpolation pixel. Make up a pair. Similarly, a pixel block composed of pixel blocks BLa (15) shown in FIG. 3D and pixel blocks BLb (15) shown in FIG. Make up a pair.

  In this way, the extraction unit 3 sequentially extracts pixel block pairs composed of two pixel blocks located at point-symmetric positions with the interpolation pixel as the center.

  The extraction means 3 further includes not only a pair of pixel blocks located at point symmetry with respect to the interpolated pixel as shown in FIGS. 3A to 3D but also one of them to the left of one pixel. Also extracted are those replaced with pixel blocks shifted to. For example, the pixel block BLb (1) shown in FIG. 4A is shifted to the left by one pixel as shown in FIG. 5A, and the pixel block BLb (1) shown in FIG. A pixel block pair consisting of a block BLa (1) and a pixel block BLb in which the pixel block BLb (2) shown in FIG. 4B is shifted to the left by one pixel as shown in FIG. 5B. FIG. 5C shows a pixel block pair composed of (2) and the pixel block BLa (2) shown in FIG. 3B. Similarly, the pixel block BLb (3) shown in FIG. As shown in FIG. 4, a pixel block pair composed of the pixel block BLb (3) shifted to the left by one pixel and the pixel block BLa (3) shown in FIG. The pixel block BLb (15) shown is shown in FIG. A pixel block BLb (15) which is shifted by one pixel left so that, to extract the composed pixel block pair out with pixel block BLa (15) shown in Figure 3 (d).

  The pixel blocks illustrated in FIGS. 3A to 3D and the pixel blocks illustrated in FIGS. 4A to 4D are horizontally aligned with each other (between the pixels corresponding to each other inside them). There is a difference (distance) in the position of an even number of pixels, that is, 4 pixels, 2 pixels, and 0 pixels, but the pixel blocks shown in FIGS. 3A to 3D and FIGS. The pixel blocks shown in (1) have an odd number of pixels in the horizontal direction, that is, a difference (distance) of 3 pixels and 1 pixel.

  The extraction unit 3 further includes pixels constituting a pair of pixel blocks located at point-symmetric positions with the interpolation pixel as the center, as illustrated in FIGS. 3A to 3D and FIGS. 4A to 4D. A block obtained by replacing one of the blocks with a pixel block shifted upward by one pixel is also extracted. For example, a pixel block constituted by a pixel block obtained by shifting the pixel block illustrated in FIGS. 4A to 4D by one pixel and a pixel block illustrated in FIGS. 3A to 3D. Extract pairs.

  The extraction unit 3 further includes pixels constituting a pair of pixel blocks located at point-symmetric positions with the interpolation pixel as the center, as illustrated in FIGS. 3A to 3D and FIGS. 4A to 4D. One obtained by replacing one of the blocks with a pixel block shifted to the left by one pixel and upward by one pixel (that is, obliquely to the upper left) is also extracted. For example, a pixel constituted by a pixel block in which the pixel block illustrated in FIGS. 4A to 4D is shifted one pixel diagonally to the upper left and a pixel block illustrated in FIGS. 3A to 3D. Extract block pairs.

  The pixel block pair extracted by the extraction unit 3 is input to the similarity calculation unit 4 and the similarity between the pixel blocks is calculated for each pixel block pair. For example, when calculating the similarity between the pixel block BLa in FIG. 3A and the pixel block BLb in FIG. 4A, the similarity calculation unit 4 calculates the pixels a1 to 1 in the pixel block BLa in FIG. The sum of absolute differences (hereinafter referred to as “a15”) and the pixels b1 to b15 (pixels at the positions corresponding to the pixels a1 to a15 (the same positions in the pixel block)) in the pixel block BLb of FIG. (Referred to as SAD) is calculated as the similarity according to the following equation.

Similarly, the sum SAD of absolute difference values is calculated for other pixel block pairs, and the calculation result is output to the block pair selecting unit 40 and the reliability evaluating unit 41 as the similarity of each pixel block pair. Here, the smaller the sum SAD value, the greater the similarity, and the greater the sum SAD, the smaller the similarity.
In this way, the sum SAD of absolute difference values (index indicating similarity) is calculated for all block pairs extracted by the extraction means 3. In this example, since 15 block pairs are extracted for each of the above four types, the sum SAD of absolute difference values is calculated for a total of 60 block pairs.

The block pair selection unit 40 selects a pixel block pair having the highest similarity calculated by the similarity calculation unit 4 as a candidate block pair from among the plurality of block pairs extracted by the extraction unit 3.
The block pair selection means 40 is also configured such that, among the plurality of block pairs extracted by the extraction means 3, the relative positional relationship with respect to the interpolation pixel is other than the relative positional relationship with respect to the interpolation pixel of the candidate block pair. Among them, the pixel block pair having the highest similarity is selected as the evaluation block pair.
The “relative positional relationship” is defined, for example, by the direction of a line connecting the central pixels of the pixel blocks of each block pair.
For example, when the block pairs are arranged in a matrix in accordance with the “relative positional relationship”, the blocks other than the block pair located in the vicinity of the candidate block pair are treated as having a relatively distant relative relationship.

FIG. 6 shows an example of the matrix arrangement of the relative positional relationship described above.
In this example (examined example), the extraction means extracts 60 block pairs, so the matrix of FIG. 6 has 60 element positions arranged in 6 rows and 10 columns. These 60 element positions respectively correspond to the relative positional relationship between the block pairs, and therefore it can be said that each element corresponds to the block pair.
The position of each element in the matrix M is represented by (i, j) by a horizontal coordinate value (column number) i and a vertical coordinate value (row number) j. The coordinate origin is at the leftmost and uppermost position of the matrix, the horizontal coordinate value i increases to the right by one for each element interval, the vertical coordinate value j decreases downward, Assume that the value increases by 1 for each interval.

The element at the leftmost and uppermost position (i = 1, j = 1) of the matrix is composed of the pixel block shown in FIG. 3 (a) and the pixel block shown in FIG. 4 (a). Corresponds to a block pair.
The element at the position of i = 3 and j = 1 corresponds to a block pair composed of the pixel block shown in FIG. 3B and the pixel block shown in FIG.
The element at the position of i = 5 and j = 1 corresponds to a block pair composed of the pixel block shown in FIG. 3C and the pixel block shown in FIG.
The element at the position of i = 9 and j = 5 corresponds to a block pair composed of the pixel block shown in FIG. 3D and the pixel block shown in FIG.
Generally speaking, when f = 0, 1, 2, 3 or 4, g = 0, 1 or 2, the position of i = 1 + (2f), j = 1 + (2g) (in FIG. The elements in (indicated by “”) are point-symmetrical with respect to the interpolated pixel as illustrated in FIGS. 3A to 3D and FIGS. 4A to 4D. This corresponds to a pixel block pair composed of pixel blocks.

The element at the position of i = 2 and j = 1 corresponds to a block pair composed of the pixel block shown in FIG. 3A and the pixel block shown in FIG.
The element at the position of i = 4 and j = 1 corresponds to a block pair composed of the pixel block shown in FIG. 3B and the pixel block shown in FIG.
The element at the position of i = 6 and j = 1 corresponds to a block pair composed of the pixel block shown in FIG. 3C and the pixel block shown in FIG.
The element at the position of i = 10 and j = 5 corresponds to a block pair composed of the pixel block shown in FIG. 3D and the pixel block shown in FIG.
Generally speaking, when f = 0, 1, 2, 3 or 4, g = 0, 1 or 2, the position of i = 2 + (2f), j = 1 + (2g) (“Δ” in FIG. As shown in FIGS. 3A to 3D and FIGS. 5A to 5D, the element in FIG. 3A is a pixel block in a point-symmetric position with respect to the interpolation pixel. Corresponds to a pixel block pair in which the pixel block of the second reference frame is replaced with a pixel block shifted to the left by one pixel.

Similarly, when f = 0, 1, 2, 3 or 4, g = 0, 1 or 2, the position of i = 2 + (2f), j = 1 + (2g) (indicated by “□” in FIG. 6). The element in (shown) is replaced with a pixel block in which the pixel block of the second reference frame is shifted up by one pixel with respect to a pixel block pair consisting of pixel blocks located in point symmetry with respect to the interpolation pixel. Pixel block pair (a pixel block composed of a pixel block obtained by shifting the pixel block shown in FIGS. 4A to 4D by one pixel and a pixel block shown in FIGS. 3A to 3D) Corresponding to).
Similarly, when f = 0, 1, 2, 3 or 4, g = 0, 1 or 2, the position of i = 2 + (2f), j = 2 + (2g) (indicated by “x” in FIG. 6). The element in (shown) is replaced with a pixel block in which the pixel block of the second reference frame is shifted one pixel diagonally to the upper left with respect to a pixel block pair consisting of pixel blocks located symmetrically with respect to the interpolation pixel. The pixel block pair (the pixel block shown in FIGS. 4A to 4D is shifted by one pixel diagonally to the upper left and the pixel block shown in FIGS. 3A to 3D). Corresponds to a pixel block pair).

  In the matrix of FIG. 6, the block pairs adjacent in the upper, lower, left, right, diagonally upper left, diagonally upper right, diagonally lower left, and diagonally lower right directions of the candidate block pairs are treated as “neighboring block pairs”, otherwise Are treated as a block pair whose relative positional relationship is relatively far. In the example shown in FIG. 6, it is assumed that an element at a position of i = 3 and j = 3 (marked with “*”) is selected as a candidate block pair. Lower, left, right, diagonally upper left, diagonally upper right, diagonally lower left, diagonally lower right, that is, a block pair within the range surrounded by the thick line NB is treated as “a block pair located in the vicinity”.

  As described above, “a block pair adjacent in the upper, lower, left, right, diagonal upper left, diagonal upper right, diagonal lower left, and diagonal lower right directions of the candidate block pair” In other words, the pair of blocks whose horizontal and vertical distances are 1 or less (the distance between adjacent elements is defined as 1) is less than the pair. Generally speaking, “a block pair whose horizontal and vertical distances are R or less (R is an integer of 1 or more) relative to a candidate block pair” is treated as a “block pair located in the vicinity”. It is also good.

The reliability evaluation unit 41 calculates the average value of all the pixel block pairs extracted by the extraction unit 3 (average value of the sum SAD of the absolute differences), and the similarity average value and the evaluation similarity level The reliability of the candidate similarity is evaluated from the candidate similarity. For example, if the following inequality (2a) holds, it is determined that there is reliability, and if it does not hold, it is determined that there is no reliability.
(Se−Sc) × B> (Sa−Se) (2a)
(Se is a value indicating similarity for evaluation (total sum SAD of difference absolute values of evaluation block pair),
Sc is a value indicating the candidate similarity (the sum SAD of absolute differences of the block pairs for evaluation),
Sa is a value indicating the average value of the similarity (the sum SAD of absolute differences of the block pairs for evaluation),
B is a coefficient larger than 1, for example 2)
Rewriting equation (2a)
{(B + 1) / B} × (Sa−Se) <(Sa−Sc) (2b)
It becomes.
The reliability evaluation unit 41 outputs the determination result to the reference pixel position selection unit 5.

The fact that formula (2a) is satisfied means that B times the difference between the candidate similarity and the evaluation similarity (B> 1)
Meaning that the difference between the similarity for evaluation and the average value of the similarity is larger and satisfying the expression (2b) means that the difference between the similarity of the candidate block pair and the average value of the similarity is an evaluation block. Meaning that the difference between the pair similarity and the average value of the similarities is greater than a predetermined magnification (B + 1) / B (the numerical value of the similarity is small), that is, either of the formulas (2a) and (2b) This also means that the similarity of the candidate block pair is remarkably large (the peak including the maximum value of the similarity is prominent).

Hereinafter, the significance of using Expression (2a) or Expression (2b) for determining whether the similarity between the candidate block pairs is remarkably large will be further described.
A pixel block pair whose relative positional relationship with respect to the interpolation pixel is close to the candidate block pair often has a high degree of similarity according to the candidate block pair. For example, the relative positional relationship of the block pair with respect to the interpolated pixel is actually discrete, but similar when assuming that the block pair can be generated continuously (while the relative positional relationship is continuously changed). The relative positional relationship having the maximum degree is a candidate block pair selected from the pixel block pair having the discrete positional relationship as described above, and a block pair adjacent to the candidate block pair (on the matrix in FIG. 6). Between the candidate blocks and the similarities of adjacent block pairs may be extremely small.
If the pixel block pair with the highest similarity is selected from those other than those having a close relative positional relationship to the interpolation pixel, the similarity is close to the maximum value because the positional relationship is close to the candidate block pair as described above. A pixel block pair having the highest similarity among the pixel block pairs can be extracted as an evaluation block pair.

  Hereinafter, this point will be described with reference to FIGS. 7 and 8, the horizontal axis represents the relative positional relationship of the pixel block pair with respect to the interpolation pixel, and corresponds to the position in the horizontal direction in the row including the candidate block pair in FIG. 6, for example. The vertical axis represents the sum SAD of absolute differences and the average value thereof.

In the example of FIG. 7, there is a candidate block pair at the fifth position (i = 5) from the left, and in the example shown in FIG. 8, there is a candidate block pair at the third position (i = 3) from the left. . The position of the candidate block pair and the position of the adjacent pixel block pair are “the range NB in which the relative positional relationship with respect to the interpolation pixel is close to the candidate block.
The evaluation block pair is the pixel block pair having the highest similarity outside the above range NB. In FIG. 7 and FIG. 8, it is assumed that the evaluation block pair exists in the same row as the candidate block pair for simplicity.

The evaluation block pair extracted in this manner is continuous with the maximum value portion (peak) when the change in similarity in the vicinity of the candidate block pair is moderate as shown in FIG. A pair of pixel blocks located in a “bottom” portion (a portion sufficiently away from the peak, that is, a portion other than a portion having a value according to the maximum value because it is close to the maximum value),
When the change in similarity in the vicinity of the candidate block is steep as shown in FIG. 8, the pixel block pair is located at the second highest similarity peak.
(Note that in FIGS. 7 and 8, the sum SAD of the absolute differences is shown as an index representing similarity, so the peak of similarity corresponds to the bottom of the sum SAD)
Compared with the pixel block pair located at the “bottom” continuous to the maximum value portion (peak) and the pixel block pair located at the second largest peak (maximum value portion), the average value of the similarity If the difference is larger than a predetermined magnification, it means that the similarity of the candidate block pair is remarkably large.
In the present invention, as described above, only when the similarity between the candidate block pairs is remarkably large, it is determined that the selection of the candidate block pair is reliable, and interpolation is performed using the center position of the candidate block pair as the reference pixel position. On the other hand, if the similarity is the maximum, but the difference in similarity is not so large compared to the others, the center position of the candidate block pair is not set as the reference pixel position, so that the interpolation is further performed. This is done accurately, and this can increase the accuracy of interpolation.

When the reliability evaluation unit 41 determines that the reference pixel position selection unit 5 is reliable, the reference pixel position selection unit 5 selects the center pixel position of the block pair, which is the output of the block pair selection unit 40, as the reference pixel position. If it is determined that there is no reference pixel position, a reference pixel position selected previously, for example, the previous reference pixel position delayed by one frame by the one-frame delay means 42 is selected, and information indicating the reference pixel position is set as a symmetric position. This is supplied to the interpolation means 6.
Since the selection of the reference pixel position is also the selection of the direction of the reference pixel with respect to the interpolation pixel, it also has the meaning of selecting the interpolation direction.

Here, the pixel located at the center of each pixel block (center pixel) refers to a pixel located in a central row and a central column constituting each pixel block. For example, when the pixel block is composed of pixels of 3 rows and 5 columns, it means the pixels of the second row and the third column. In general, each pixel block is composed of pixels of p rows and q columns, and when p and q are odd numbers, the pixels of the (p + 1) / 2th row and the (q + 1) / 2th column are referred to.
When p and q are even numbers, the pixel in the p / 2 row or the (p / 2) +1 row, the q / 2 column, or the (q / 2) +1 column is handled as the central pixel. That is, the center of the pixel block is not limited to the accurate center position, but may be a position near the accurate center position.
In order to uniquely determine the center position, it is desirable that p and q are odd numbers.

The symmetric position interpolation unit 6 calculates an average value of the pixel values at the reference pixel position selected by the reference pixel position selection unit 5 among the pixel values at the center position of each pixel block supplied from the extraction unit 3. The result is output to the output terminal 7 as the pixel value of the interpolation pixel.
By performing the above operation on all the interpolation pixels of the interpolation frame, an interpolation image of the interpolation frame is obtained.

In the above example, the extracting unit 3 uses the pixel block pair (a) that is point-symmetric with respect to the interpolation pixel and one reference frame such as the second reference frame for such a pixel block pair. A pixel block pair (b) in which the inner pixel block is shifted to the left by one pixel (b), a pixel block pair (c) that is shifted by one pixel, and a pixel block pair (d) that is shifted by one pixel diagonally to the upper left are extracted. However, the present invention is not limited to this, and in order to reduce the scale of hardware, in addition to the pixel block pair (a), a pixel block pair (b) in which the pixel block in one reference frame is shifted to the left by one pixel. ) Only, or only the pixel block pair (c) shifted up by one pixel, or only the pixel block pair (d) shifted by one pixel diagonally to the upper left may be extracted.
Also, instead of “one pixel left”, one shifted to “one pixel right” may be used, and instead of “one pixel up”, one shifted “one pixel down” may be used, Instead of “one pixel left diagonally upper”, one shifted “one pixel right diagonally downward”, “one pixel left diagonally downward”, or “one pixel right diagonally upward” may be used.
Furthermore, a pixel block pair in which the pixel block in the first reference frame is shifted may be used instead of the pixel block pair in which the pixel block in the second reference frame is shifted.
Further, in addition to the pixel block pair composed of pixel blocks that are point-symmetric with respect to the interpolation pixel, one pixel block of the pixel block pair composed of pixel blocks that are point-symmetric with respect to the interpolation pixel is one pixel. Instead of using a pixel block pair replaced with a shifted one, a pixel block in a point-symmetrical position with respect to the interpolation pixel is used instead of a pixel block pair consisting of a pixel block in a point-symmetrical position with respect to the interpolation pixel. It is also possible to use only a pixel block pair in which one pixel block of the pixel block pair consisting of is replaced by one pixel block shifted.

  As described above, when extracting a block pair in which one pixel block of a pixel block pair composed of pixel blocks located symmetrically with respect to the interpolation pixel is shifted to one pixel, the first reference is made. Even when there is an odd number of pixels in the shifted direction between the frame and the second reference frame (the image of the subject has moved by an odd number of pixels), a pair of distances that are closest to the amount of movement The center position of each pixel block can be selected as a reference pixel position, and there is an effect that interpolation can be performed more accurately.

  For example, when extracting a block pair in which one pixel block of a pixel block pair composed of pixel blocks located symmetrically with respect to the interpolation pixel is shifted to the left by one pixel or to the right by one pixel, Even when there is an odd number of pixels in the horizontal direction between one reference frame and the second reference frame (the subject image has moved by an odd number of pixels), a pair of distances that are closest to the amount of movement The center position of each pixel block can be selected as a reference pixel position, and there is an effect that interpolation can be performed more accurately.

  Similarly, when a block pair in which one block of a pixel block pair composed of pixel blocks located symmetrically with respect to the interpolation pixel is shifted one pixel up or one pixel down is extracted, A pair having a mutual distance closest to the amount of movement even when there is an odd number of pixels in the vertical direction between the first reference frame and the second reference frame (the subject image has moved by an odd number of pixels). The center position of each pixel block can be selected as the reference pixel position, and there is an effect that interpolation can be performed more accurately.

  Similarly, one block of a pixel block pair consisting of pixel blocks that are point-symmetric with respect to the interpolation pixel is shifted one block diagonally to the upper left or one pixel diagonally to the lower right, or one pixel diagonally to the lower left. When a block pair shifted one pixel diagonally to the upper right is extracted, there is an odd pixel movement in the diagonal direction between the first reference frame and the second reference frame (the subject Even when the image is moved by an odd number of pixels), the center position of each of the pair of pixel blocks having the mutual distance closest to the amount of movement can be selected as the reference pixel position, and interpolation can be performed more accurately. There is an effect that can be done.

  On the other hand, when a pixel block pair composed of pixel blocks that are point-symmetric with respect to the interpolation pixel is extracted, there is even-numbered pixel movement between the first reference frame and the second reference frame. When the subject image is moved by an even number of pixels, the center position of each of the pair of pixel blocks having the mutual distance closest to the amount of movement can be selected as the reference pixel position, and interpolation can be performed more accurately. There is an effect that can be done.

  In addition to the pixel block pair consisting of pixel blocks that are point-symmetric with respect to the interpolation pixel, one pixel block of the pixel block pair consisting of pixel blocks that are point-symmetric with respect to the interpolation pixel is shifted to one pixel. When the block pair is extracted, even if the movement in the shifted direction between the first reference frame and the second reference frame is for odd pixels or even pixels. Even if there is even-numbered pixel movement, the center position of each of the pair of pixel blocks having the mutual distance closest to the amount of movement can be selected as the reference pixel position, and interpolation can be performed more accurately.

  When it is determined that a block pair obtained by shifting one pixel block of a pixel block pair consisting of pixel blocks located symmetrically with respect to the interpolation pixel to one pixel has the maximum similarity, Not only the position but also a position shifted by one pixel may be selected as the reference pixel position. For example, when it is determined that a block pair obtained by shifting one pixel block of a pixel block pair composed of pixel blocks located symmetrically with respect to an interpolation pixel to the left by one pixel has the maximum similarity, Not only the center position, but also the pixel position adjacent to the right side of the center position is selected as the reference pixel position, and the average of the pixel values of the four selected reference pixel positions is obtained. May be the pixel value of the interpolated pixel.

  The symmetrical position interpolating means 6 uses the average value of the pixels located at the center of the pixel blocks constituting the pixel block pair as the pixel value of the interpolated pixel, but is located at the center of either one of the pixel block pairs. The pixel value of the pixel to be used may be used as the pixel value of the interpolation pixel.

  Further, as shown in FIG. 2, the similarity calculation unit 4 calculates the total SAD for all the pixels in the pixel block (from a1 to a15 and from b1 to b15), but in order to reduce the hardware scale. In addition, the sum of absolute values of the differences between the pixel values of the pixels after performing equal thinning processing on the pixels in one pixel block, for example, the pixel block BLa, and the pixels at corresponding positions in the other pixel block BLb It is good to ask for. For example, in the block BLa, pixels (r is an integer of 2 or more) are extracted in the horizontal direction, and pixels (r is an integer of 2 or more) are extracted in the vertical direction. It is also possible to extract each pixel (one for every s), obtain the absolute difference value of the pixel values of these pixels and the corresponding pixels in the block BLb, and obtain the sum thereof. An example of the thinning process when r and s are 2 is shown in FIGS. In the example shown in the figure, every other pixel in the vertical direction and the horizontal direction, and only the pixels arranged in a checkered pattern (shown by hatching) are extracted, and the absolute difference value is obtained only for these pixels, Find the sum SAD of them.

  As described above, in the first embodiment, a pixel block pair including the pixel block BLa of the first reference frame and the pixel block BLb of the second reference frame that is substantially point-symmetrical with respect to the position of the interpolation pixel. A plurality of extraction means 3 for extracting, and a similarity calculation means 4 for calculating the similarity of two pixel blocks based on the pixel values of the pixels in the two pixel blocks, for each of a plurality of extracted pixel block pairs; Pixels other than pixel block pairs whose similarity (candidate similarity) of the pixel block (candidate block pair) having the highest similarity calculated by the similarity calculation means 4 and the relative positional relationship with respect to the interpolation pixel are close to the candidate block pair Reliability evaluation that evaluates reliability based on the similarity (evaluation similarity) of the pixel block (evaluation block pair) having the highest similarity among the block pairs The reference pixel for selecting either the central pixel position of the current candidate block pair or the reference pixel position one frame before delayed by the delay means 42 according to the result evaluated by the means 41 and the reliability evaluation means 41 Since the position selection means 5 and the symmetrical position interpolation means 6 for generating an interpolation pixel using the pixel value of the reference pixel position selected by the reference pixel position selection means 5 are provided, the degree of similarity is the highest, and the others A pair of pixel blocks having a remarkably high degree of similarity with respect to a pair of pixel blocks, and thus having a mutual distance closest to the amount of image motion between the first reference frame and the second reference frame, The pixel value of the interpolated pixel can be determined based on the pixel value of the pixel located at the center of each of the pair of pixel blocks in which the close distance stands out compared to the other pixel block pairs. Therefore, it is possible to perform interpolation accuracy.

  Further, a sum of absolute values of differences between pixel values of at least some of the pixels in one pixel block constituting the pixel block pair and corresponding pixels in the other pixel block is output as similarity. Therefore, there is an effect that the similarity can be obtained easily and accurately, and a more accurate interpolation image can be obtained.

  In this case, the sum of the absolute values of the differences between the pixel values of one pixel block constituting the pixel block pair and the pixel value at the corresponding position in the other pixel block is output as the similarity. For example, it is possible to obtain the similarity more accurately and to obtain a more accurate interpolated image.

  On the other hand, the sum of the absolute values of the difference between the pixel values of the pixels after the pixels of one pixel block constituting the pixel block pair and the pixels at the corresponding positions in the other pixel block are obtained. Is output as the similarity, there is an effect that the calculation of the similarity can be realized with a relatively small hardware scale.

Embodiment 2. FIG.
In the first embodiment, the similarity calculation result of the similarity calculation unit 4 is similarly evaluated for any pixel block pair and used to select the reference pixel position. However, the pixel block pair far from the interpolation pixel position is used. Since the reliability tends to be lower, there is a possibility that an interpolation pixel is erroneously generated. In the second embodiment, the degree of similarity is corrected to a smaller value for a pixel block pair farther from the position of the interpolated pixel (correction is performed by multiplying the sum SAD of absolute difference values by a larger coefficient). By using the corrected similarity as a selection index, the reference pixel position is more appropriately selected, and the reliability of generating the pixel value of the interpolation pixel is increased.

  FIG. 11 shows a block diagram of a frame interpolation apparatus according to the second embodiment. The illustrated frame interpolating apparatus is generally the same as the frame interpolating apparatus of FIG. 1, but differs in that similarity correction means 13 is added. Components having the same reference numerals as those in FIG. 1 operate in the same manner as in the first embodiment.

  The similarity correction unit 13 sets the distance between the interpolation pixel and the pixel (center pixel) located at the center of the pixel block with respect to the sum SAD of the absolute difference values of each pixel block pair calculated by the similarity calculation unit 4. Accordingly, SAD × K is calculated for the sum SAD using the correction coefficient K as shown in FIG. 12, and the calculation result is output to the block pair selection means 40 as the selection index SI. Since the larger the sum SAD means that the similarity is smaller, the larger the coefficient K applied to the sum SAD is, the larger the selection index SI is, and the similarity is corrected to a smaller value (the corrected similarity is more It means that it is a small value).

  For example, when the distance between the interpolated pixel and the central pixel of each of the pair of pixel blocks used for similarity calculation is 1 pixel or less, K is 1, and for 2 pixels or more, K is 1.5. In the range from 1 pixel to 2 pixels, K is gradually increased from 1 to 1.5.

  As the distance between the interpolated pixel and the central pixel of each of the pair of pixel blocks used for similarity calculation, for example, the longer distance in the horizontal direction and the longer distance in the vertical direction are used. In this case, in the example described with reference to FIGS. 3A to 3D and FIGS. 4A to 4D, the horizontal distance between the interpolated pixel and the center pixel of the farthest pixel block is 2. When the pixel block (the distance in the vertical direction is one pixel) and the two pixel blocks constituting the pixel block pair are “farthest pixel blocks”, the similarity of the pixel block pair used for calculating the similarity The coefficient K with respect to is 1.5.

  When the distance from the center pixel to the interpolated pixel is different between one pixel block and the other pixel block of the pixel block pair used for calculating the position of the interpolated pixel and the similarity, the average is used. For example, as shown in FIGS. 3A and 5A, the horizontal distance from the center pixel of one pixel block of the pixel block pair used for calculating the similarity to the interpolation pixel is 2 pixels ( When the distance in the vertical direction is 1 pixel) and the horizontal distance from the center pixel of the other pixel block to the interpolation pixel is 1 pixel (the vertical distance is also 1 pixel), the average of the distance is 1.5 pixels and the correction coefficient K is 1.25.

  Based on the output of the similarity correction unit 13, the block pair selection unit 40 has the highest corrected similarity for each interpolation pixel (the value of the selection index SI = SAD × K is the smallest). The position is selected as a reference pixel position of candidate similarity, and is output to the reference pixel position selection means 5. Thereafter, the pixel value of the interpolation pixel is calculated as in the first embodiment, and an image of the interpolation frame is obtained.

  As described above, in the second embodiment shown in FIG. 11, the selection index SI used to select the reference pixel position in the reference pixel position selection means 5 is generated by the similarity calculation means 4 and the similarity correction means 13. The selection index generation means 32 is configured. On the other hand, in the first embodiment shown in FIG. 1, it can be seen that the output itself of the similarity calculation means 4 is the selection index SI, and that the selection index generation means is constituted only by the similarity selection means 4. it can.

In the above example, the longer of the horizontal distance and the vertical distance is used for determining the correction coefficient as the distance between the center pixel and the interpolation pixel of the pixel block. Instead, the horizontal distance and the vertical distance are used instead. The average of the distance in the direction may be used as the distance between the pixel blocks, and the square root of the sum of the square of the horizontal distance and the square of the vertical distance, that is,
{(Horizontal distance) ² + (vertical distance) ² } ^1/2
May be used as the distance between the center pixel of the pixel block and the interpolated pixel.
If the distance from the center pixel to the interpolated pixel is different between one pixel block and the other pixel block of the pixel block pair used for calculating the position of the interpolated pixel and the similarity, the average is used. However, the longer one may be used instead, and the shorter one may be used.

  As in the above embodiment, the pixel block pair whose center position is far from the interpolated pixel is corrected so that the similarity is smaller, and the corrected similarity is used as a selection index. Therefore, there is an effect that a more accurate interpolation image can be obtained.

Embodiment 3 FIG.
The similarity calculation unit 4 according to the first embodiment is a flat image (an image with little change in density between pixels over a wide range of the screen) or an image with a change in the calculation process of the similarity between pixel block pairs. Although the evaluation was performed in the same way, the flatness of the flat image tends to increase the similarity, and therefore the selection of the reference pixel position may not be appropriate. In the third embodiment, a pixel block pair including a flat image is evaluated with a lower similarity, and a pixel block pair including a changed image is evaluated with a higher similarity. Therefore, each pixel block is determined according to the degree of image variation. The reliability of the selection of the reference pixel position is improved by correcting the pair of similarities and using the corrected similarity as a selection index.

FIG. 13 shows a frame interpolation apparatus according to the third embodiment. The illustrated interpolation apparatus is generally the same as that shown in FIG. 1 except that a correction control means 27 and a similarity correction means 28 are added. Components having the same reference numerals as those in FIG. 1 operate in the same manner as in the first embodiment.
In the embodiment of FIG. 13, the selection index generation unit 32 is configured by the similarity calculation unit 4, the correction control unit 27, and the similarity correction unit 28.

  The correction control unit 27 includes a horizontal direction variation degree calculation unit 27a, a vertical direction variation degree calculation unit 27b, an addition unit 27c, and a correction coefficient generation unit 27d.

  The variation degree calculation means 27a calculates the variation degree for each pixel block pair. A method of calculating the degree of variation will be described by taking the pixel block pair in FIGS. 3A and 4A as an example. As the degree of variation in the horizontal direction, calculation is performed according to the following equation (3).

The above equation is a function indicating a larger value as the change in the horizontal direction is larger, and a smaller value as the change in the horizontal direction is smaller.
The vertical variation degree calculation means 27b further calculates the vertical variation degree by the following equation (4).

  The above equation is a function indicating a larger value as the change in the vertical direction is larger, and a smaller value as the change in the vertical direction is smaller.

The adding means 27c adds the horizontal variation degree calculation result CH and the vertical variation degree calculation result CV, and outputs the addition result as a total variation degree CT.
The correction coefficient generation unit 27d outputs a correction coefficient D according to the total variation degree CT. The relationship between the overall variation degree CT and the correction coefficient D is as shown in FIG. 14, for example. Above the first predetermined value Tha, the correction coefficient D is 1, and the second is smaller than the first predetermined value Tha. The correction coefficient D is 2 below the predetermined value Thb, and in the range between the first predetermined value Tha and the second predetermined value Thb, the correction coefficient D decreases from 1 to 2 as the overall variation degree CT decreases. Gradually increase.
The correction coefficient generation means 27d may be constituted by a conversion table, for example, or may be constituted by a calculation means.
The correction coefficient D output from the correction coefficient generation unit 27 d is supplied to the similarity correction unit 28 as an output of the correction control unit 27.

The similarity correction unit 28 calculates SAD × D for the sum SAD output from the similarity calculation unit 4 using the correction coefficient D, and outputs the calculation result as a selection index SI.
As a result of such correction, when the total variation degree CT is equal to or greater than the first predetermined value Tha, the coefficient for the total SAD is 1, so that the amount of similarity correction is zero, but the total variation degree CT is Below the second predetermined value Thb, the total SAD is doubled, the similarity is corrected to a smaller value (the corrected similarity is set to a smaller value), and the overall variation degree CT is the first Between the predetermined value Tha and the second predetermined value Thb, as the total variation degree CT decreases, the coefficient D for the total SAD is increased and the similarity is corrected to a smaller value (corrected similarity) Is a smaller value).

  Based on the output of the similarity correction unit 28, the block pair selection unit 40 has the center of the pair of pixel blocks having the highest corrected similarity for each interpolation pixel (the value of the selection index SI = SAD × D is the smallest). The position is selected as a reference pixel position of candidate similarity, and is output to the reference pixel position selection means 5. Thereafter, the pixel value of the interpolation pixel is calculated as in the first embodiment, and an image of the interpolation frame is obtained.

  With the above configuration, the degree of variation of the pixel values of the pixels in the pixel block is calculated, and when the degree of variation is small, the degree of similarity is corrected so as to be smaller. This has the effect of solving the problem that a correct interpolated image cannot be obtained.

Embodiment 4 FIG.
In the first to third embodiments, the similarity calculation unit 4 obtains the sum of absolute values of differences between a pair of pixel blocks located in point symmetry with respect to the interpolation pixel, and uses it as an index indicating the similarity. The center position of the pixel block pair having the highest degree of similarity is selected as the reference pixel position, and the result of selection of the reference pixel position for each interpolation pixel (interpolation pixel of interest) is obtained for the interpolation pixels around the interpolation pixel of interest. If the determined reference pixel position is changed using the approximate direction (interpolation direction) with respect to the interpolation pixel, an interpolation image of a better interpolation frame can be obtained.

  FIG. 15 shows a frame interpolation apparatus according to the fourth embodiment. The illustrated interpolation apparatus is generally the same as the interpolation apparatus shown in FIG. 1, but differs in that reference pixel position changing means 16 and delay compensation means 17 and 18 are added. Components having the same reference numerals as those in FIG. 1 operate in the same manner as in the first embodiment.

  The reference pixel position changing unit 16 is aligned in the horizontal direction and is continuously subjected to interpolation processing, and for example, for each of a plurality of, for example, seven interpolation pixels, the reference pixel position selection unit 5 selects the reference for each interpolation pixel. The direction of the pixel position with respect to the interpolation pixel is stored. For example, the reference pixel position changing unit 16 changes the approximate direction (interpolation direction) with respect to the interpolation pixel of the reference pixel position sequentially selected for each interpolation pixel by the reference pixel position selection unit 5 in the horizontal direction by a predetermined number of interpolations. The number of pixels, for example, 7 pixels is held. Specifically, information indicating a region to which the reference pixel position selected by the reference pixel position selection unit 5 for each interpolation pixel belongs is stored for a predetermined number of interpolation pixels, for example, seven pixels as information indicating the general interpolation direction. .

  For example, when the center pixel of the pixel block of the first reference frame in the pixel block pair determined by the reference pixel position selection unit 5 is in the region UL of FIG. 16A, the first flag “FUL” , Set the second flag “FDL” if it is in the region DL of FIG. 16B, set the third flag “FDR” if it is in the region DR of FIG. If it is in the area UR of FIG. 16D, the fourth flag “FUR” is set. In FIG. 16, in the third row of the first to fourth columns, both the first flag “FUL” and the second flag “FDL” are set (in the first area and in the second area When the third row of the sixth to ninth columns is in the third row, the third flag “FDR” and the fourth flag “FUR” are both set (in the third region, 1st flag “FUL” and 4th flag “FUR” are both set (in the first region), when in the fifth column of the first to second rows. And the second flag “FDL” and the third flag “FDR” are both set (when the fourth flag is in the fifth column of the fourth to fifth rows). Means that it is in the third area, and is in the fifth column of the third row. Flags “FUL”, “FDL”, “FDR”, “FUR” are all set (in the first area, in the second area, in the third area, and in the fourth area Means that

  The reference pixel position changing unit 16 has three interpolation pixels (n) whose approximate interpolation direction for each target interpolation pixel (nth interpolation pixel) is located on the left side of the target interpolation pixel and is continuous to the target interpolation pixel. The flags set for each of the predetermined number of interpolation pixels present in the left direction of the target interpolation pixel, all of which coincide with the rough interpolation direction for each of the (-3, n-2, n-1th interpolation pixels). At least one of the flags set for the target interpolation pixel), or three interpolation pixels (n + 1, n + 2, n + 3) located on the right side of the target interpolation pixel and continuing to the target interpolation pixel At least one flag set for each of the predetermined number of interpolation pixels present in the right direction of the target interpolation pixel. Is coincident with any of the flags set for the target interpolation pixel), the reference pixel position selected by the reference pixel position selection unit 5 for the interpolation pixel is directly used as the reference pixel position to the symmetrical position interpolation unit 6. Output. On the other hand, if none of the above, instead of the reference pixel position selected by the reference pixel position selection means 5, information indicating the same position as the interpolation pixel in the first and second reference frames is used as the reference pixel. The information indicating the position is output to the symmetrical position interpolation means 6.

  In other words, the reference pixel position changing unit 16 has a rough interpolation direction for each target interpolation pixel (n-th interpolation pixel) located on the left side of the target interpolation pixel and continuous to the target interpolation pixel. Different from the rough interpolation direction for any of the three interpolation pixels (n-3, n-2, n-1th interpolation pixel) (the left-hand direction of the interpolation pixel of interest is the predetermined number of interpolation pixels). None of the flags set for any of the flags set for the target interpolation pixel), and three interpolation pixels that are located to the right of the target interpolation pixel and are continuous to the target interpolation pixel (N + 1, n + 2, n + 3th interpolation pixel) is different from the approximate interpolation direction (the flag set for any of the predetermined number of interpolation pixels existing in the right direction of the interpolation pixel of interest) If the deviation does not match any of the flags set for the target interpolation pixel), the interpolation in the first and second reference frames is used instead of the reference pixel position selected by the reference pixel position selection means 5. Information indicating the same position as the pixel is output to the symmetrical position interpolation means 6 as information indicating the reference pixel position. Otherwise, information indicating the reference pixel position selected by the reference pixel position selection unit 5 is output to the symmetrical position interpolation unit 6 as it is.

  The delay compensation means 17 and 18 compensate the delay time required for correcting the interpolation direction of the reference pixel position changing means 16 with respect to the pixel value at the center position of each pixel block output from the extracting means 3, and the symmetrical position interpolation means 6. Output to. The symmetric position interpolation means 6 is a pixel value at a position specified by the information indicating the reference pixel position output from the reference pixel position changing means 16 (any one of the pixel values at the center position of each block supplied from the extraction means 3). ) Similar to the first embodiment, the pixel value of the interpolation pixel is generated.

  According to the above configuration, the interpolation of the reference pixel position selected by the reference pixel position selection unit 5 for each of the plurality of interpolation pixels that are aligned in the horizontal direction and are successively subjected to interpolation processing is performed. A reference pixel position in which a direction with respect to a pixel is stored, and a reference pixel position with respect to each interpolation pixel of interest is selected with respect to any of a predetermined number of interpolation pixels in which the direction with respect to the interpolation pixel exists in the left direction of the interpolation pixel of interest If the reference pixel position selected for any of the predetermined number of interpolation pixels existing in the right direction of the target interpolation pixel is different from the direction with respect to the interpolation pixel, the reference pixel Instead of the reference pixel position selected by the position selection means 5, a reference in which the same position as the target interpolation pixel in the first and second reference frames is set as the reference pixel position. Since with the element position changing means 16, which produces the effect of a user can select a reference pixel position better.

  As an application example of the present invention, it can be applied to frame frequency conversion for television and frame frequency conversion for commercial monitors. Even if the frame frequency is not converted, it can be applied to frame conversion in which one redundant frame (a frame having the same content as the immediately preceding frame) is removed and a new frame is generated.

It is a block diagram which shows the interpolation apparatus of Embodiment 1 of this invention. FIG. 3 is a diagram showing a pair of pixel blocks that are sequentially selected for selection of a reference pixel position in the first embodiment. (A)-(d) is a figure which shows the pixel block in the 1st reference frame selected sequentially for selection of the reference pixel position in Embodiment 1. FIG. (A)-(d) is a figure which shows the pixel block in the 2nd reference frame selected sequentially for selection of the reference pixel position in Embodiment 1. FIG. (A)-(d) is a figure which shows the pixel block in the 2nd reference frame selected sequentially for selection of the reference pixel position in Embodiment 1. FIG. It is a figure which shows the relative positional relationship with respect to the interpolation pixel of the some pixel block pair extracted by the extraction means 3 with a matrix. It is a figure which shows an example of the change of similarity. It is a figure which shows the other example of the change of similarity. 6 is a diagram showing pixel blocks in a first reference frame that are sequentially selected for selection of a reference pixel position in the first embodiment. FIG. FIG. 6 is a diagram illustrating pixel blocks in a second reference frame that are sequentially selected for selection of a reference pixel position in the first embodiment. It is a block diagram which shows the interpolation apparatus of Embodiment 2 of this invention. 10 is a diagram illustrating a relationship between a distance between a reference pixel and an interpolated pixel and a correction coefficient K in Embodiment 2. FIG. It is a block diagram which shows the interpolation apparatus of Embodiment 3 of this invention. FIG. 10 is a diagram illustrating a relationship between a total variation degree CT and a correction coefficient D in the third embodiment. It is a block diagram which shows the interpolation apparatus of Embodiment 4 of this invention. (A)-(d) is a figure which shows each area | region used for determination of the general direction with respect to the interpolation pixel of the reference pixel position selected by the reference pixel position selection means 5 in Embodiment 4. FIG.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Input terminal, 2 Delay means, 3 Extraction means, 4 Similarity calculation means, 5 Reference pixel position selection means, 6 Symmetric position interpolation means, 7 Output terminal, 13 Similarity correction means, 16 Reference pixel position change means, 17 Delay Compensation means, 18 delay compensation means, 27 correction control means, 28 similarity correction means, 32 selection index generation means, 40 block pair selection means, 41 reliability evaluation means, 42 delay means.

Claims (18)

A frame interpolation device for generating a pixel value of an interpolation pixel in an interpolation frame between a first reference frame and a second reference frame,
A pixel block pair composed of a pixel block of the first reference frame and a pixel block of the second reference frame which are in point symmetry with respect to the position of the interpolation pixel, and one of the pixel block pair Extraction means for extracting a plurality of one or both of the pixel block pairs in which the pixel block is replaced with a pixel block shifted by one pixel;
For each of a plurality of pixel block pairs extracted by the extraction means, similarity calculation means for calculating the similarity between two pixel blocks based on the pixel values of the pixels in the two pixel blocks;
Among the plurality of block pairs extracted by the extracting unit, the similarity calculated by the similarity calculating unit or a pixel block pair having the largest correction is selected as a candidate block pair, and Among the plurality of block pairs extracted by the extracting means, the relative positional relationship with respect to the interpolation pixel is other than the relative positional relationship with respect to the interpolation pixel of the candidate block pair, Block pair selection means for selecting a pixel block pair having the largest similarity or a corrected one as a block pair for evaluation;
Similarity of the candidate block pair or a correction thereof (hereinafter referred to as “candidate similarity”) and similarity of the evaluation block pair or a correction thereof (hereinafter referred to as “evaluation similarity”) ) Based on the size of the candidate block pair, a reliability evaluation means for evaluating the reliability of the candidate block pair;
When the reliability evaluation unit determines that there is reliability, the center position of each pixel block constituting the candidate block pair is selected as a reference pixel position, and when it is determined that there is no reliability, the interpolation pixel Reference pixel position selection means for selecting a reference pixel position previously selected as a reference pixel position of
A frame interpolation device comprising: interpolation means for generating the interpolation pixel based on a pixel value at a reference pixel position selected by the reference pixel position selection means.   The frame interpolation apparatus according to claim 1, wherein the relative positional relationship is defined by a line connecting center pixels of pixel blocks of each block pair.   3. The block pair positioned in the vicinity of the candidate block pair is treated as having a close relative positional relationship when the block pairs are arranged in a matrix according to the relative positional relationship. Frame interpolator.   4. The frame interpolating apparatus according to claim 3, wherein on the matrix arrangement, those adjacent in the upward, downward, left, right, diagonally upward, and diagonally downward directions are treated as a block pair located in the vicinity.   The apparatus further comprises frame delay means for delaying the selection result of the reference pixel position selection means by one frame, and the reference pixel position selection means uses the output of the frame delay means as the previously selected reference pixel position. The frame interpolation apparatus according to claim 1.   The extracting means sets one pixel block to one pixel block pair consisting of a pixel block at a point-symmetrical position with respect to the interpolation pixel and a pixel block pair consisting of a pixel block at the point-symmetrical position. 2. The frame interpolation device according to claim 1, wherein a pixel block pair replaced with a pixel block shifted to the left of the pixel or to the right of one pixel is extracted.   The extracting means sets one pixel block to one pixel block pair consisting of a pixel block at a point-symmetrical position with respect to the interpolation pixel and a pixel block pair consisting of a pixel block at the point-symmetrical position. The frame interpolating apparatus according to claim 1, wherein a pixel block pair replaced with a pixel block shifted on one pixel or one pixel below is extracted.   The extracting means sets one pixel block to one pixel block pair consisting of a pixel block at a point-symmetrical position with respect to the interpolation pixel and a pixel block pair consisting of a pixel block at the point-symmetrical position. 2. The pixel block pair replaced by a pixel block shifted to the upper right pixel, the lower right pixel, the upper left pixel, or the lower left pixel pixel is extracted. Frame interpolator.   The similarity calculation means obtains the sum of absolute values of differences between pixel values of at least some of the pixels in one pixel block constituting the pixel block pair and pixels at corresponding positions in the other pixel block, 9. The frame interpolating apparatus according to claim 1, wherein the sum is output as the similarity.   The similarity calculation means obtains a sum of absolute values of differences between pixel values of all pixels in one pixel block constituting the pixel block pair and pixels at corresponding positions of the other pixel block. The frame interpolation apparatus according to claim 9.   The similarity calculation means calculates a difference between a pixel value of a pixel after performing equal thinning processing on pixels in one pixel block constituting a pixel block pair and a pixel at a corresponding position in the other pixel block. The frame interpolation apparatus according to claim 9, wherein a sum of absolute values of the frame interpolation is obtained.   The reliability evaluation unit evaluates reliability using an average value of similarities of all pixel block pairs extracted by the extraction unit, the candidate similarity, and the similarity for evaluation. Item 12. The frame interpolation device according to any one of Items 1 to 11. The reliability evaluation means includes the following inequality:
{(Se-Sc)} × B> (Sa-Se)
(Se represents the similarity for evaluation,
Sc represents the candidate similarity,
Sa represents the average value of similarity,
B represents a coefficient of 1 or more)
The frame interpolating apparatus according to claim 12, wherein when it is satisfied, it is determined that there is reliability. Similarity correction that outputs a corrected similarity that is smaller than the similarity of the pixel block pair as the distance from the interpolation pixel of the pixel located at the center of each pixel block constituting the pixel block pair is longer Further comprising means,
14. The frame interpolating apparatus according to claim 1, wherein the block pair selecting unit calculates a candidate similarity and an evaluation similarity based on the corrected similarity. Correction control means for calculating the degree of variation of the pixel values of the pixels in the pixel block and detecting the calculated degree of variation;
The degree of variation detected by the correction control means further includes a similarity correction means for outputting a corrected similarity that is smaller than the similarity output by the similarity calculation means,
14. The frame interpolation device according to claim 1, wherein the block pair selection unit selects the candidate block pair and the evaluation block pair based on the corrected similarity. Alignment in the horizontal direction, storing the direction of the reference pixel position selected by the reference pixel position selection means for each of the plurality of interpolation pixels that are continuously subjected to interpolation processing, with respect to the interpolation pixel,
The direction of the reference pixel position of the target interpolation pixel with respect to the interpolation pixel is the direction of the reference pixel position selected for any of a predetermined number of interpolation pixels existing in the left direction of the target interpolation pixel. If the reference pixel position selected for any of the predetermined number of interpolation pixels existing in the right direction of the target interpolation pixel is different from the direction with respect to the interpolation pixel, the reference pixel position selection unit selects the reference pixel position. 14. A reference pixel position changing unit that uses the same position as the target interpolation pixel in the first and second reference frames as a reference pixel position instead of the reference pixel position. The frame interpolation device according to any one of the above. The interpolation means uses the average value of the pixel values at the reference pixel position selected by the reference pixel position selection means as the pixel value of the interpolation pixel, or sets one pixel value at the reference pixel position as the pixel value of the interpolation pixel. The frame interpolation device according to claim 1, wherein the frame interpolation device is a frame interpolation device. A frame interpolation method for generating a pixel value of an interpolation pixel in an interpolation frame between a first reference frame and a second reference frame,
A pixel block pair composed of a pixel block of the first reference frame and a pixel block of the second reference frame which are in point symmetry with respect to the position of the interpolation pixel, and one of the pixel block pair An extraction step of extracting a plurality of one or both of the pixel block pairs in which the pixel block is replaced with a pixel block shifted by one pixel;
A similarity calculation step of calculating the similarity of two pixel blocks based on the pixel values of the pixels in the two pixel blocks for each of a plurality of pixel block pairs extracted by the extraction step;
Among the plurality of block pairs extracted in the extraction step, select a pixel block pair having the largest similarity calculated by the similarity calculation step or a corrected one as a candidate block pair, and Among the plurality of block pairs extracted in the extraction step, the relative positional relationship with respect to the interpolation pixel is other than the relative positional relationship with respect to the interpolation pixel of the candidate block pair. A block pair selection step of selecting a pixel block pair having the largest similarity or a corrected one as a block pair for evaluation;
Similarity of the candidate block pair or a correction thereof (hereinafter referred to as “candidate similarity”) and similarity of the evaluation block pair or a correction thereof (hereinafter referred to as “evaluation similarity”) ) To evaluate the reliability of the candidate block pair based on the magnitude of
When the reliability evaluation step determines that there is reliability, the center position of each pixel block constituting the candidate block pair is selected as a reference pixel position, and when it is determined that there is no reliability, the interpolation pixel A reference pixel position selection step for selecting a reference pixel position previously selected as the reference pixel position of
An interpolating step for generating the interpolated pixel based on a pixel value at the reference pixel position selected in the reference pixel position selecting step.

Images