JP2011223086A - Resolution converting device and method, scanning line interpolating device and method, and video display device and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a resolution conversion method in which a high-quality high-resolution image is obtained by reducing block matching errors in motion compensation.SOLUTION: A future image segmentation unit 101 and a past image segmentation unit 102 perform segmentation to produce two blocks each larger than a searching range at positions that are point-symmetrical to a pixel of interest. A non-similarity calculating unit 103 calculates a non-similarity between both blocks. A motion estimating unit 104 estimates motion vectors on the basis of non-similarities of candidate motion vectors. A motion compensation interpolation processing unit 106 generates a motion compensation interpolation image from the past and future images on the basis of the motion vectors. A combining unit 107 combines a current image and the motion compensation interpolation image to generate a high resolution image. A searching range determining unit 105 determines the next searching range on the basis of the current motion vectors.

Description

  The present invention relates to a resolution conversion apparatus and method, a scanning line interpolation apparatus and method used in a display device such as a television or a display, and a video display apparatus and method including the resolution conversion apparatus or method.

When displaying low-resolution video content on a display device having a high-resolution display performance, the display device generally converts the resolution of the video content into a resolution that matches the performance of the display device and displays it.
As a conventional technique for resolution conversion, resolution conversion by motion compensation is known. In this method, the motion of a pixel between a plurality of adjacent images on the time axis is estimated, and pixel information is acquired from different images based on this motion information to generate an interpolation pixel. At this time, the image quality of the high-resolution image obtained as an output result changes depending on the accuracy of motion estimation.

  An example of motion estimation is motion estimation by block matching. Motion estimation by block matching is a method for estimating motion between two images. As a general method, first, a block composed of a plurality of pixels is set on an image (hereinafter referred to as a reference image) to be subjected to motion estimation, and then a block similar to this block is set as the other image (hereinafter referred to as a reference image). , Find out from the reference image). At this time, the relative position of the two associated blocks is estimated as a motion estimation result. As described above, motion estimation by general block matching is a method of estimating a motion by finding a block similar to the block set on the reference image from the reference image. When there are a lot of similar blocks on the reference image, there is a case that the block is not correctly associated and proper motion estimation cannot be performed.

  Such an error in block correspondence can be reduced by assuming that the motion of each pixel is a uniform linear motion. Since the time interval is generally short between adjacent images on the time axis in the multiple images that make up the video, the change in the movement of each pixel between these images is small, and the movement of each pixel is a uniform linear motion Can be approximated. For this reason, by introducing the above assumption into the motion estimation, an unnatural association deviating from the uniform linear motion is not performed, and an error in the association of blocks is reduced.

  When performing motion estimation assuming constant velocity linear motion, a technique used in frame rate conversion can be used. For example, in Patent Document 1, when an intermediate frame is generated from a current frame and a frame immediately before the current frame (hereinafter referred to as a past frame), a point relating to a pixel (hereinafter referred to as a target pixel) that is a target of motion estimation in the intermediate frame. The pixel of the current frame and the pixel of the past frame that are symmetrical are paired, and the most similar pair is found. For this reason, the movement of the pixel pair from the pixel on the past frame to the target pixel is the same as the movement of the pixel of interest from the pixel pair to the pixel on the current frame, and the three frames have a uniform linear motion. Here, the intermediate frame is set as a reference image to be subjected to motion estimation, the current frame is an image immediately after the reference image (hereinafter, future image), and the past frame is an image immediately before the reference image (hereinafter, past image). ), It is possible to realize motion estimation assuming constant-velocity linear motion among the three images of the past image, the reference image, and the future image. At this time, if a block pair is found instead of a pixel pair, the same estimation can be performed by block matching. Hereinafter, such a method for estimating motion from a pair of pixels or blocks having a point-symmetric positional relationship with respect to the pixel of interest is referred to as linear motion estimation.

Japanese Patent No. 4359223

  However, in the linear motion estimation, when there is a region where the change in luminance between adjacent pixels is small (hereinafter referred to as a flat region), there is a possibility that the block correspondence in the vicinity fails and an erroneous motion vector is estimated. is there. For this reason, resolution conversion using linear motion estimation has a problem in that image quality deteriorates due to failure caused by acquiring pixel information based on an erroneous motion vector in a part of an output image.

The resolution converter of the present invention
In a plurality of continuous images, a reference image to be increased in resolution, a future image positioned one time later on the time axis with respect to the reference image, and one time previous on the time axis with respect to the reference image Use the past image located as the input image,
A search range determination unit for determining a search range of a motion vector;
Based on the motion vector candidate within the search range determined by the search range determination unit and the coordinates of the pixel to be interpolated on the reference image, a block composed of a plurality of pixels from the future image is larger than the width of the search range A future image cutout unit that cuts out with a predetermined size set to
A past image cutout unit that cuts out a block that is point-symmetric with a block cut out by the future image cutout unit with respect to the pixel to be interpolated on the reference image;
A dissimilarity calculation unit that calculates a dissimilarity between two blocks cut out by the future image cutout unit and the past image cutout unit;
A motion estimator that outputs a motion vector that minimizes a motion vector evaluation value obtained based on the dissimilarity calculated by the dissimilarity calculator;
A motion compensation interpolation processing unit that obtains pixel information from a future image and a past image based on a motion vector output from the motion estimation unit, and generates an interpolation image by generating an interpolation pixel from the pixel information;
An integrated processing unit that obtains an output image by integrating a reference image and an interpolation image generated by the motion compensation interpolation processing unit or a synthetic interpolation image generated based on the interpolation image;
The search range determination unit determines a search range in the high resolution processing when the next image is a reference image based on a motion vector output from the motion estimation unit when each image is a reference image. It is characterized by.

  According to the present invention, it is possible to prevent erroneous block association by linear motion estimation in the vicinity of a flat region. Thereby, appropriate motion compensation can be performed and a high-resolution high-resolution image can be generated. Moreover, since the search range can be reduced by providing the search range determination unit, an increase in the amount of calculation can be suppressed.

It is a block diagram which shows the resolution converter of Embodiment 1 of this invention. 6 is a diagram illustrating an image used for describing motion estimation in Embodiment 1. FIG. It is a figure which shows the example of an ideal block matching when the block size in Embodiment 1 is set to the required size or less. It is a figure which shows the example of an incorrect block matching when the block size in Embodiment 1 is set to the request size or less. FIG. 10 is a diagram for explaining the effect of setting the block size to the requested size in the first embodiment. 6 is a diagram for explaining a search range determination method according to Embodiment 1. FIG. FIG. 10 is a diagram for explaining an effect of setting an effective range in the first embodiment. 6 is a diagram illustrating an example of resolution conversion in Embodiment 1. FIG. 6 is a diagram for explaining setting of a search range for sub-pixel motion estimation in Embodiment 1. FIG. FIG. 10 is a diagram for describing extraction of a sub-pixel motion estimation block in the first embodiment. It is a block diagram which shows the resolution conversion apparatus of Embodiment 2 of this invention. (A)-(d) is a figure explaining the effect of the flat area | region penalty introduction in Embodiment 2. FIG. It is a block diagram which shows the resolution conversion apparatus of Embodiment 3 of this invention. It is a block diagram which shows the video display apparatus of Embodiment 4 of this invention. (A) And (b) is a figure explaining the kind of field of the interlaced image in Embodiment 4. FIG. (A)-(c) is a figure explaining the motion vector candidate in Embodiment 4. FIG. It is a block diagram which shows the video display apparatus of Embodiment 5 of this invention.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing a configuration of a resolution conversion apparatus according to Embodiment 1 of the present invention.
The resolution conversion apparatus 100 according to the present embodiment includes a reference image that is an object of resolution conversion, a future image that is positioned one time later on the time axis with respect to the reference image, and one image on the time axis that is relative to the reference image. An apparatus that receives three images of previous past images as input and outputs a high-resolution image having a size approximately twice as long as that of the reference image based on pixel information acquired from the future image and the past image. A future image clipping unit 101, a past image clipping unit 102, a dissimilarity calculation unit 103, a motion estimation unit 104, a search range determination unit 105, a motion compensation interpolation processing unit 106, and a combination unit 107.
This device can be applied to video display devices such as televisions and monitors, and further to image processing on personal computers.

The future image cutout unit 101 cuts out a block composed of a plurality of pixels from the future image and outputs the block to the dissimilarity calculation unit 103. The block size is defined such that the predetermined search range is vertical (vertical direction in FIG. 2, ie, column direction) H (where H is an odd number) pixels, horizontal (horizontal direction, ie, row direction in FIG. 2) W (where W Is an odd number) pixel, the vertical size BH is (2 × H−1) pixels and the horizontal size BW is (2 × W−1) pixels. The reason for this will be described in detail in the description of the motion estimation unit 104. The cutout position of the block is determined by the coordinates of the pixel of interest that is the pixel to be interpolated on the reference image and the motion vector candidate in the search range determined by the search range determination unit 105 described later. Specifically, when the coordinate of the target pixel is (x _b , y _b ), the x component (horizontal component) of the motion vector candidate is mv _x , and the y component (vertical component) is mv _y , the future image is cut out. The center coordinates (x _n , y _n ) of the block cut out at the part are expressed as in equation (1).

The past image cutout unit 102 cuts out a block composed of a plurality of pixels from the past image and outputs it to the dissimilarity calculation unit 103. The block size is the same as the block cut out by the future image cutout unit 101. The block cutout position is point-symmetric with respect to the block cutout position in the future image cutout unit 101 with respect to the coordinates of the target pixel. Specifically, when the coordinate of the pixel of interest is (x _b , y _b ), the x component of the motion vector candidate is mv _x , and the y component is mv _y , the center coordinates (x _{p, y p)} is expressed as equation (2).

The dissimilarity calculation unit 103 calculates the dissimilarity between the two blocks cut out by the future image cut-out unit 101 and the past image cut-out unit 102, and outputs them to the motion estimation unit 104. The dissimilarity is a value that decreases when the blocks are similar and increases when the blocks are not similar. In the present embodiment, the degree of dissimilarity is calculated as the sum of absolute differences SAD of luminance shown by the equation (3). However, B _n (i, j) is a luminance value at coordinates (i, j) on the block cut out by the future image cutout unit 101, and B _p (i, j) is a block cut out by the past image cutout unit 102. Let it be the luminance value at the upper coordinates (i, j). The coordinates (i, j) on the block are values in which the upper left coordinates of each block are (0, 0). At this time, the lower right coordinates are (BH-1, BW-1) from the block size. | Α | is the absolute value of α.

  The motion estimation unit 104 uses the dissimilarity for each motion vector candidate output from the dissimilarity calculation unit 103 as a motion vector evaluation value, and uses a motion vector candidate that gives the minimum motion vector evaluation value as a motion estimation result as a search range determination unit. 105 and the motion compensation interpolation unit 106. One motion estimation result is selected from motion vector candidates within the search range for each pixel to be interpolated on the reference image.

  Here, the sizes of blocks to be cut out by the future image cutout unit 101 and the past image cutout unit 102 are vertical (2 × H−1) pixels and horizontal (2 × W−1) pixels (hereinafter, this block size is referred to as a required size). The reason for setting to “Yes” will be explained. As described above, the linear motion estimation may fail in block matching in the vicinity of the flat region. This is because the dissimilarity is reduced by setting both of the two blocks within the flat region. However, setting the block size to the required size solves this problem.

This is because the entire search range can be accommodated in the block by setting the block size to the required size. This is shown by way of example in the horizontal direction.
When the x coordinate of the target pixel is x _b and the search range is ± n pixels in the horizontal direction (where n is a positive integer), the left end coordinate of the search range is x _b−n , and the right end coordinate is x _{b + n} . The width of the search range is 2n + 1 pixels. At this time, the required size of the block is 4n + 1 pixels. The left end of the block is located at −2n pixels from the center, and the right end is located at + 2n pixels from the center. When the x component of the motion vector candidate is n, the center of the block cut out in the future image is the right end of the search range, that is, _{xb + n} . At this time, the left end of the block is located at _xb−n , that is, the left end of the search range. Since the block cut out in the past image is in a point-symmetrical position with respect to the target pixel, the center of the block is x _b−n and the right end of the block is x _{b + n} . Thus, it can be seen that both ends of the search range are within the block even when motion vector candidates are set at the end of the search range.

The effect of keeping the entire search range within the block will be described below. Here, in order to simplify the description, an example is used in which an interpolation image having the same size as those is generated from a past image and a future image. The size of each image is 7 × 7 pixels, and each pixel is identified as shown in FIG. In this case, the center pixels of the past image, the future image, and the interpolated image are a _3,3 , b _3,3 , and c _3,3 , respectively.

First, an example in which the block size is set to be smaller than the required size is shown. FIG. 3 shows an example of ideal block association. A block of 3 × 3 pixels is cut out with the center pixel c ₃ , ₃ of the interpolation pixel as the pixel of interest and the motion vector candidate as the vector (1, 0). The search range in which the center of the block moves is vertical and horizontal ± 1 pixel centered on the pixel of interest. In the figure, a portion indicated by a bold line is a block, and a portion indicated by a dotted line is a search range. At this time, the center of the block to be cut out is a _{2,3 in} the past image and b _{4,3 in} the future image.

A high brightness white vertical line is placed in a flat area with low brightness and little change in brightness in both past and future images. In the past image, one pixel is located to the left of the center, and in the future image, one pixel to the right of the center. There is a white line at the position. For this reason, in an interpolated image positioned between the past image and the future image on the time axis, it is ideal that there is a white line at the center position of the image. Blocks to be cut out in FIG. 3, Both block and capture the white line in the middle in, c _3,3 interpolated from these blocks are interpolated as part of the white pixel of the white line. Similarly, when ideal motion compensation is performed, the interpolated image becomes an ideal image with the white line positioned at the center, as shown in the figure.

FIG. 4 shows an example of incorrect block association. The motion vector candidate is a vector (-1, 0), and the target pixel, block size, and search range are the same as those in FIG. The center of the block to be cut out is a _{4,3 in} the past image and b _{2,3 in} the future image. At this time, both blocks are cut flat area, c _3,3 are interpolated from these blocks is interpolated as a pixel of a portion of low brightness of the flat region. Similarly, when erroneous motion compensation is performed, the interpolated image is an image having no white line as shown in the figure. In this way, the association of the blocks cut out in FIG. 4 gives an undesirable interpolation result.

  However, there is a high possibility that such an association error will occur. In the block pair as shown in FIG. 4, only a flat region with a small luminance change is captured in the block, so that the luminance difference between the pixels of both blocks is small. For this reason, the dissimilarity calculated in the dissimilarity calculating unit 103 is reduced, and as a result, the possibility that a motion vector candidate that gives an incorrect block association is selected in the motion estimating unit 104 increases.

Next, an example in which the block size is set to the requested size is shown in FIG. The target pixel is c _3,3 , the motion vector candidate is the vector (−1, 0), the search range is vertical and horizontal ± 1 pixel centered on the target pixel, and the conditions other than the block size are the same as in FIG. Yes. From the setting of the search range, the size of the search range is 3 pixels both vertically and horizontally. Accordingly, the required block size is 5 × 5 pixels. In this case, as shown in FIG. 5, a white line is captured at the left end of the block from the past image, and a white line is captured at the right end of the block from the future image. For this reason, in the evaluation of dissimilarity between blocks, at least the left and right ends of the block take the difference between the low-brightness pixels and the high-brightness pixels, resulting in high dissimilarity, resulting in incorrect association. Can be prevented.

  As described above, block matching errors can be reduced by setting the block size to the required size and keeping the entire search range within the block.

  Returning to the description of FIG. The search range determination unit 105 determines a search range based on the motion estimation result output from the motion estimation unit 104, and outputs the determined search range to the future image cutout unit 101 and the past image cutout unit 102. Since the motion estimation unit 104 outputs a motion estimation result for each pixel to be interpolated on the reference image, the search range determination unit 105 also determines a search range for each pixel to be interpolated on the reference image.

An example of the search range determination method is shown in FIG. FIG. 6 shows motion estimation at the coordinates (3, 3), which is the center of the image. The search range is set to ± 1 pixel from the center position in both the vertical and horizontal directions, and the motion vector is the vector (1, 1). The dotted line in the figure indicates the current search range, and the broken line indicates the next search range. Since the search range is determined as a predetermined range centered on the position moved by the motion vector from the coordinates of the target pixel, a new search range centered on the coordinates (4, 4) may be set as shown in the figure. Recognize. As a result, various motions can be estimated while keeping the search range small. Here, an example of a future image is shown, but the same processing is performed on a past image, and a search range of the same size is set at a point-symmetric position with respect to the target pixel.
Since the search range is a range of ± n pixels with respect to the center position, the search range is an odd number such as 2n + 1. This is why the vertical and horizontal sizes H and W of the search range described above are set to odd numbers.

  In this search range determination method, various vectors can be used as motion vector candidates without changing the size of the search range. In FIG. 6, in the current search range, vectors (-1, -1), (0, -1), (1, -1), (-1, 0), (0, 0), (1, 0 ), (-1, 1), (0, 1), (1, 1) are motion vector candidates, but in the next search range, vectors (0, 0), (1, 0), (2, 0), (0, 1), (1, 1), (2, 1), (0, 2), (1, 2), (2, 2) are motion vector candidates. As described above, the search range determination unit 105 can estimate various motions while keeping the size of the search range small by determining the next search range according to the current motion vector.

  Here, the effect of combining the size setting of the block to be cut out by the future image cutout unit 101 and the past image cutout unit 102 and the search range determination by the search range determination unit 105 will be described. In this embodiment, in order to prevent block matching errors, the block size is set to vertical (2 × H−1) pixels and horizontal (2 × W) with respect to the vertical and horizontal sizes H and W of the search range. -1) Since a pixel is set, the amount of calculation is enormous when searching a wide range. However, since the search range determination unit 105 can reduce the size of the search range with respect to the motion that can be estimated, it is possible to reduce block association errors while suppressing the amount of calculation.

  However, when the search range is determined indefinitely, the search range may be set at a position away from the target pixel while the search range determination is repeated. In this case, the estimated motion is only an extremely large motion. Such a situation is not desirable because generally no extremely large movement occurs between adjacent images on the time axis. Therefore, an effective range is set to the search range determined by the search range determination unit 105. The effective range is a predetermined range centered on the coordinates of the target pixel, and can be set, for example, twice as wide as the search range.

FIG. 7 shows an example when the search range is set outside the effective range. In FIG. 7, each pixel is identified as in FIG. In the figure, a dotted line indicates a newly set search range, and a triple line indicates an effective range. In the figure, pixels b _{3 and 3} are pixels of interest, and the effective range is ± 2 pixels vertically and horizontally with b ₃ and ₃ as the center. The search range is set to vertical and horizontal ± 1 pixel with the pixel b _5,1 as the center. At this time, the pixels b _4,0 , b _5,0 , b _6,0 , b _6,1 , b _6,2 are located outside the effective range within the search range. For this reason, the range actually searched is b _4,1 , b _5,1 , b _4,2 , b _5,2 . Further, when the next search range is determined, the search range is set around one of the four pixels, so that the center of the search range is always within the effective range.

  By setting the effective range as described above, the center of the search range is always located inside the effective range, so that the search range does not deviate from the coordinates of the target pixel by a certain amount or more. Thereby, it is possible to perform appropriate motion estimation excluding extremely large motion.

  Returning to the description of FIG. The motion compensation interpolation processing unit 106 generates an interpolation image based on the future image, the past image, and the motion estimation result output from the motion estimation unit 104, and outputs it to the combination unit 107. Specifically, an interpolation using the average value of the future image associated with the motion vector obtained by the motion estimation unit 104 and the pixel on the past image (that is, the central pixel of the block associated with the motion estimation) as the pixel value. Generate a pixel.

  The pixel to be interpolated in the motion compensation interpolation processing unit 106 is a pixel newly generated by resolution conversion. A pixel to be interpolated will be described with reference to the drawings. Since the resolution conversion apparatus according to the present embodiment outputs a high-resolution image approximately twice as long and vertically as the reference image, the relationship between the reference image and the high-resolution image is as shown in FIG. In the figure, circles and squares indicate pixels. Pixels indicated by circles are original pixels of the reference image, and pixels indicated by squares are pixels interpolated by motion compensation. The motion compensation interpolation processing unit 106 generates pixels indicated by squares.

  The pixels indicated by the squares in FIG. 8 are subpixels located in the middle of the original pixels on the reference image. In order to increase the resolution of the reference image, it is necessary to interpolate these subpixels. Therefore, motion estimation also needs to be performed for subpixels. So far, pixel motion estimation has been described for the sake of simplicity, so sub-pixel motion estimation will be described below.

In the case of subpixel motion estimation, the motion vector is a vector having a subpixel to be interpolated as a start point and a pixel within the search range as an end point.
FIG. 9 shows an example of setting a search range in the case of sub-pixel motion estimation. In the figure, white circles are pixels, black circles are subpixels, and triangles are subpixels at positions to be interpolated. It is assumed that an image composed of these pixels is a past image or a future image. The pixel count is 1 pixel between white circles and 0.5 pixel between white circles and black circles (that is, between pixels and sub-pixels). In the figure, the search range is set as ± 1 pixel around the position to be interpolated, and is indicated by a broken line. At this time, there are six pixels within the search range, and the motion vector candidates are vector (-0.5, -1), (0.5, -1), ( −0.5,0), (0.5,0), (−0.5,1), and (0.5,1).

  Next, FIG. 10 shows an example of block cutout in subpixel motion estimation. In the figure, white circles are pixels, black circles are subpixels, and triangles are pixels set at the center of the block. It is assumed that an image composed of these pixels is a past image or a future image. In the figure, the block size is 3 × 3 pixels, and the blocks are indicated by bold lines. In motion estimation of subpixels, motion vector candidates are set so that pixels within the search range end, so the center of the block is always a pixel. Therefore, similarly to pixel motion estimation, if the block size is a × b pixels, a × b pixels are included in the block. In FIG. 10, an example in which the block size is 3 × 3 pixels is shown to simplify the drawing. However, in actuality, the block size is set to the above-described required size in sub-pixel motion estimation.

  As described above, since the block to be cut out in the sub-pixel motion estimation is a block having a predetermined size centered on the pixel, the future image cutout unit 101, the past image cutout unit 102, the dissimilarity calculation unit 103, the motion The estimation unit 104 can perform motion estimation by the operation described in the pixel motion estimation. Also in the search range determination unit 105, the center of the newly set search range is the pixel, so that the search range can be determined by the same operation as the pixel motion estimation.

  Returning to the description of FIG. The combination unit 107 combines the reference image and the interpolation image obtained from the motion compensation interpolation unit 106, and outputs a high resolution image. In the high-resolution image of FIG. 8, pixels indicated by circles are obtained from the reference image, and pixels indicated by squares are obtained from the interpolated image.

  In the future image cutout unit 101 and the past image cutout unit 102, when the search range is vertical H pixels and horizontal W pixels, the block size is vertical (2 × H−1) pixels and horizontal (2 × W). -1) Although the pixel is used, the block size may be further increased. Even if the block size is smaller than this, if the vertical size is H pixels and the horizontal size is W pixels or more, a slight block matching error reduction effect can be obtained. Therefore, the vertical size is changed from H pixels to (2H-1) pixels. The horizontal size may be set to a value between W pixels and (2W-1) pixels.

  In the above example, the dissimilarity calculation unit 103 calculates the dissimilarity as the sum of absolute differences in luminance. However, the dissimilarity calculation method is not limited to this. For example, in the case of an RGB signal, a sum of absolute differences may be calculated for each of R, G, and B components, and these sums may be used as dissimilarities. Further, the dissimilarity may be calculated using gradient information.

  In the above example, the search range is determined based on the current motion vector at the interpolation position in the search range determination unit 105. However, the search range may be determined by other methods. For example, the search range may be determined from a vector obtained by averaging motion vectors around the interpolation position. Further, for example, motion vectors for the past several times may be recorded, and the search range may be determined from an average vector obtained by averaging them. Furthermore, the motion of the entire screen may be estimated from all the motion vectors, and the search range may be determined from the motion vector of the entire screen.

  According to the present embodiment, it is possible to realize generation of a high-quality high-resolution image by appropriate motion compensation with reduced block correspondence errors while suppressing an increase in calculation amount.

Embodiment 2. FIG.
In the present embodiment, the flatness of the image in the two associated blocks is calculated, and the penalty corresponding to the value is taken into account when estimating the motion. Thereby, the block correspondence error is reduced as compared with the resolution conversion apparatus of the first embodiment, and a high-resolution image with higher image quality is obtained.

FIG. 11 is a block diagram showing the configuration of the resolution conversion apparatus according to the second embodiment of the present invention.
The resolution conversion apparatus 200 according to the present embodiment includes a reference image that is an object of resolution conversion, a future image that is positioned one time later on the time axis with respect to the reference image, and one image on the time axis that is relative to the reference image. An apparatus that receives three images of previous past images as input and outputs a high-resolution image having a size approximately twice as long as that of the reference image based on pixel information acquired from the future image and the past image. The future image clipping unit 101, the past image clipping unit 102, the dissimilarity calculation unit 103, the flatness calculation unit 201a, the flatness calculation unit 201b, the motion estimation unit 202, the search range determination unit 105, and the motion compensation interpolation processing unit 106 and a combination unit 107.
This device can be applied to video display devices such as televisions and monitors, and further to image processing on personal computers.

  The difference between the present embodiment and the first embodiment is that a flatness calculation unit 201 a and a flatness calculation unit 201 b are added, and that a motion estimation unit 202 is placed instead of the motion estimation unit 104. It is. Since the operations of the future image cutout unit 101, the past image cutout unit 102, the dissimilarity calculation unit 103, the search range determination unit 105, the motion compensation interpolation processing unit 106, and the combination unit 107 are the same as those in the first embodiment, Description is omitted.

The flatness calculation unit 201 a and the flatness calculation unit 201 b calculate the flatness of the image in the cut out block, and output the result to the motion estimation unit 202. The block cut out by the future image cutout 101 is input to the flatness calculation unit 201a, and the block cut out by the past image cutout unit 102 is input to the flatness calculation unit 201b.
The flatness F is calculated by, for example, Expression (4). However, MAX is the maximum value (positive value) that the luminance value can take, and σ is the standard deviation of the luminance value of the pixels in the block.

  The flatness F calculated by Expression (4) is 1 when all the pixels in the block have the same luminance value. On the other hand, when the variation in luminance of the pixels in the block is large, a small value is taken.

  The motion estimation unit 202 estimates the motion vector in consideration of the flatness, and based on the dissimilarity obtained by the dissimilarity calculation unit 103 and the flatness obtained by the flatness calculation units 201a and 201b. A motion vector evaluation value is calculated, and a motion vector candidate that gives the minimum motion vector evaluation value is output to the search range determination unit 105 and the motion compensation interpolation unit 106 as a motion estimation result.

In this embodiment, the motion vector evaluation value is calculated by adding a flat area penalty corresponding to the flatness of the block to the dissimilarity of the associated block. The flatness of the two blocks is F ₁ and F ₂ , the threshold value for setting a flat area penalty is th1 (where 0 <th1 <1, for example, th1 = 0.8), and the weight of the penalty is a constant Wp (where Wp is a positive number), if F ₁ > th1 or F ₂ > th1, the flat area penalty FP = Wp, otherwise FP = 0.

  The effect of introducing the flat area penalty will be described with reference to FIGS. In the figure, an example of a block pair in the vicinity of a flat region is shown. Blocks 1 and 2 are 5 × 5 pixel blocks cut out from the future image cutout unit 101 and the past image cutout unit 102. In FIG. 12A, FIG. 12B, FIG. 12C, and FIG. 12D, pairs of different blocks are compared. Of these, FIG. 12 (c) is a block pair that causes a block matching error near the flat region, which is a problem of linear motion interpolation, and FIG. 12 (d) is an ideal block pair given by a correct motion vector. Suppose there is. In explaining this figure, it is assumed that a flat area penalty is imposed when there is no white line in the block.

  The block pairs compared in FIG. 12A and FIG. 12B have a high degree of dissimilarity because the presence or absence of the white line in the block and the position of the white line are different. On the other hand, in FIG. 12C, both blocks have no white line, and in FIG. 12D, the white line is in the center of both blocks, so the dissimilarity is low. For this reason, when motion vector candidates for extracting the block pairs of FIG. 12 (c) and FIG. 12 (d) simultaneously exist in the motion vector candidates to be evaluated, which motion vector candidate is correct only by the dissimilarity degree. I can't judge. If it is determined that the motion vector candidate that gives the block pair in FIG. 12C is correct, block correspondence errors frequently occur in the vicinity of the flat region.

  This problem can be solved by introducing a flat area penalty. When a flat area penalty is introduced, a flat area penalty is imposed on the motion vector candidate that gives the block pair of FIG. 12 (a) and FIG. 12 (c). As a result, a flat area penalty is imposed only on FIG. 12C in FIG. 12C and FIG. 12D with the same degree of dissimilarity. The motion vector candidate giving () is output as the motion estimation result.

  In this embodiment, not only the case where both block 1 and block 2 are flat as shown in FIG. 12C, but also one of block 1 and block 2 as shown in FIG. 12A is flat. If so, a flat area penalty is imposed. This is due to the following reason.

  When the position to be interpolated is inside the flat area, for example, the correspondence between the blocks set in the flat area as shown in FIG. 12C is ideal. On the other hand, at the end of the flat region, there may be a motion vector candidate that gives an association as shown in FIG. When the flat region penalty is imposed on FIGS. 12A and 12C, the motion vector candidate that gives the block pair of FIG. 12C with low dissimilarity is appropriately selected, but the flat region penalty is When imposed only on 12 (c), the motion vector candidate that gives the block pair in FIG. 12 (c) by reversing the magnitude of the motion vector evaluation values in FIG. 12 (a) and FIG. 12 (c) is used as the motion vector. May be chosen. Therefore, in this embodiment, if either one of the blocks is flat, a flat area penalty is imposed.

  In the above example, the flatness calculation unit 201a and the flatness calculation unit 201b calculate the flatness from the standard deviation of the luminance values of the pixels in the block. However, the flatness calculation method is not limited to this, and the flatness calculation unit 201a and the flatness calculation unit 201b Even if another feature amount indicating the variation of the pixel value is used. For example, the flatness may be calculated using variance instead of the standard deviation. Further, the flatness may be calculated from the edge component in the block.

In the above example, the motion estimation unit 202 uses the flat area penalty as a constant, and adds only when the flatness is higher than the threshold. However, for example, the flatness according to the flatness is calculated as in equation (5). You may change the penalty. However, _Kp is a positive constant.

  According to the present embodiment, since motion estimation is performed in consideration of the flatness of the two blocks to be associated with each other, the association error between the blocks in the vicinity of the flat region is further reduced, and a high-quality high-resolution image can be generated.

  Note that in the motion estimation unit 202 of the second embodiment, the dissimilarity obtained by the dissimilarity calculation unit 103 includes not only the flatness penalty FP but also a motion amount penalty MP according to the motion amount of the motion vector candidate. And the addition result may be used as a motion vector evaluation value.

  Similarly, in the first embodiment, the motion estimation unit 104 adds a motion amount penalty MP corresponding to the motion amount of the motion vector candidate to the dissimilarity obtained by the dissimilarity calculation unit 103, and uses the motion as a result of the addition. It may be used as a vector evaluation value.

For example, the motion amount penalty MP is given by Expression (6). However, a motion vector candidate is a vector (mv _x , mv _y ), and W _m is a coefficient of weight of a motion amount penalty (W _m is a positive number).

  By introducing a motion amount penalty, a block pair close to the position to be interpolated can be easily associated. In other words, an assumption is made that the estimated motion is a small motion. The movements in the image are not limited to small movements, but areas with large movements are blurred at the stage of shooting, and even if the movement is estimated appropriately, the effect of improving the sense of resolution when increasing the resolution may be small. Many. By introducing a motion amount penalty, it is possible to preferentially interpolate a portion with a small motion where a large effect can be obtained, so that the effect of improving the sense of resolution is enhanced.

Embodiment 3 FIG.
In the present embodiment, interpolation processing using pixels in an image is performed in addition to interpolation processing by motion compensation. By determining an appropriate interpolation value from the interpolation pixels generated by the two interpolation processes, a high-resolution image with higher image quality can be generated.

FIG. 13 is a block diagram showing a configuration of a resolution conversion apparatus according to Embodiment 3 of the present invention.
The resolution conversion apparatus 300 according to the present embodiment includes a reference image that is a target of resolution conversion, a future image that is positioned one time later on the time axis with respect to the reference image, and one time before the reference image on the time axis. A high-resolution image having a size approximately twice as long as that of the reference image based on the pixel information acquired from the future image and the past image. The output device includes a future image clipping unit 101, a past image clipping unit 102, a dissimilarity calculation unit 103, a flatness calculation unit 201a, a flatness calculation unit 201b, a motion estimation unit 202, a search range determination unit 105, and motion compensation. An interpolation processing unit 106, an intra-image interpolation processing unit 301, an interpolation value synthesis unit 302, and a combination unit 107 are provided.
This device can be applied to video display devices such as televisions and monitors, and further to image processing on personal computers.

The difference between the present embodiment and the second embodiment is that a parameter signal is input and an in-image processing unit 301 and an interpolation value synthesis unit 302 are provided.
Future image cutout unit 101, past image cutout unit 102, dissimilarity calculation unit 103, flatness calculation unit 201a, flatness calculation unit 201b, motion estimation unit 202, search range determination unit 105, motion compensation interpolation processing unit 106, and Since the operation of the combination unit 107 is the same as that of the second embodiment, the description thereof is omitted.

  The intra-image interpolation processing unit 301 generates an interpolated image using the pixels in the reference image. The interpolated image generated here is the same size as the high-resolution image output by the apparatus of the present embodiment. Bilinear interpolation or bicubic interpolation can be used to generate the interpolation image.

In the interpolation value synthesis unit 302, an interpolation image obtained from the motion compensation interpolation processing unit 106 (hereinafter referred to as a motion compensation interpolation image) and an interpolation image obtained from the intra image interpolation processing unit 301 (hereinafter referred to as an intra image interpolation image). And a new interpolation pixel value (composite interpolation value) is determined based on the parameter signal. The parameter signal is a parameter for determining a synthesis condition of the intra-image interpolation image and the motion compensated interpolation image, thereby determining an interpolation value. Here, for example, thresholds th2 and th3 are set. However, if the maximum value that the luminance value can take is MAX, 0 <th2 ≦ th3 <MAX. In the coordinates (x, y), the pixel value _E M of motion compensated interpolation image (x, y), when the pixel value in the image interpolation image _E F (x, y) and the interpolation value is newly obtained _{E S} (X, y) is expressed by equation (9).

  When the interpolation value is determined by equation (9), if the absolute value of the difference between the luminance values of the pixels of the motion compensated interpolated image and the intrapolated interpolated image is equal to or smaller than the threshold th2, the luminance value of the pixel of the motion compensated interpolated image is selected. If it is larger than the threshold th3, the luminance value of the pixel of the intra-image interpolation image is selected. If the absolute value of the difference between the luminance values is between th2 and th3, weighted averaging is performed on the luminance values of the two interpolated pixels, and the average value is used as the combined interpolation value. If th2 = th3 is set, weighted averaging is not performed, and one of the luminance values of the two interpolation pixels is always selected. If th2 and th3 are large values, the interpolation value is determined with emphasis on the pixels of the motion compensated interpolation image. If th2 and th3 are small values, the interpolation value is determined with emphasis on the pixels of the intra-image interpolation image.

Thus, in the above configuration example, the interpolation value synthesis unit 302 compares each pixel of the interpolation image generated by the motion compensation interpolation processing unit 106 and the interpolation image generated by the intra-image interpolation processing unit 301, and According to the comparison result, more specifically, the composition ratio is determined for each pixel according to the comparison result between the absolute value of the difference between the two (AD = | E _M −E _F |) and the parameters th2 and th3. Compositing is performed to generate a composite interpolated image.
It can be said that the synthesis conditions determined by the parameters th2 and th3 determine the synthesis method or the synthesis ratio in the respective ranges where the parameters th2 and th3 are the boundary values.

  The inter-image interpolation processing unit 301, the interpolation value synthesis unit 302, and the combination unit 107 integrate the reference image and the interpolation image generated by the motion compensation interpolation processing unit 106 or the synthesized interpolation image generated based on the reference image. Thus, an integrated processing unit 110 for obtaining an output image is configured. On the other hand, in Embodiments 1 and 2, it can be seen that the integration processing unit 110 is configured only by the combination unit 107.

  In the present embodiment, the interpolation value synthesis unit 302 more specifically calculates the difference between the absolute value of the luminance value and the threshold according to the absolute value of the difference between the motion compensated interpolation image and the intra-image interpolation image. Although the composite interpolation value is determined based on the comparison result, the interpolation value may be determined by other methods. For example, according to the motion vector evaluation value calculated by the motion estimation unit 202 in generating the pixel of the motion compensated interpolation image, for example, the motion vector evaluation value is compared with a threshold value. Select the luminance value of the pixel of the compensated interpolated image (or combine so that the motion compensated interpolated image shows a larger ratio), and if it is larger, select the luminance value of the pixel of the interpolated image in the image (or interpolated image in the image) May be combined so that indicates a larger ratio).

  Further, in the first embodiment, the second embodiment, and the present embodiment, the resolution conversion for doubling the reference image vertically and horizontally is performed, but the resolution conversion of another magnification may be performed. For example, the high resolution image obtained in the above example can be converted into an image with a different magnification by bilinear interpolation or the like. Further, resolution conversion for doubling only one of the vertical and horizontal directions may be performed.

  According to the present embodiment, an appropriate interpolation value is generated according to an input parameter signal based on two interpolation images obtained by interpolation processing using motion compensation and interpolation processing using pixel information in the image. Therefore, a higher resolution image with higher image quality can be generated.

Embodiment 4 FIG.
FIG. 14 is a block diagram showing the configuration of the scanning line interpolation apparatus according to the fourth embodiment of the present invention.
Scanning line interpolation apparatus 500 according to the present embodiment regards scanning line interpolation as double-resolution resolution conversion, and converts an interlaced image into a progressive image. This scanning line interpolation apparatus has the same configuration as the resolution conversion apparatus of the first embodiment except that the reference image, the future image, and the past image are interlaced images. The scanning line interpolation apparatus 500 of the present embodiment is It can also be regarded as a kind of resolution conversion device.

  However, in the interlaced image, the position where the pixel exists differs depending on the type of the field. Therefore, it is necessary to change the position where the sub-pixel is interpolated according to the type of the field. FIGS. 15A to 15C show the positions of pixels and sub-pixels according to the field type. In the figure, white circles indicate pixels, and black circles indicate subpixels. FIG. 15A shows a field having a pixel in the top row (hereinafter, the top field), and FIG. 15B shows a field having the top row being a sub-pixel (hereinafter, the bottom field). In the interlace method, since the top field and the bottom field appear alternately, the scanning line interpolating apparatus switches the interpolation position according to the type of the field and performs sub-pixel interpolation.

  Further, as described above, since the top field and the bottom field appear alternately in the interlace method, the type of field differs between the reference image and the future image / past image. Therefore, even if the search range set by the search range determining unit 105 is the same, motion vector candidates obtained from the search range are different from the progressive method. This will be described using FIGS. 16A to 16C as an example. In the figure, white circles are pixels, black circles are subpixels, and triangles are subpixels at positions to be interpolated. A broken line indicates a search range, and a range of 3 pixels × 3 pixels from a sub-pixel at a position to be interpolated is set as a search range. FIG. 16A shows a reference image, FIG. 16B shows a future image by the progressive method, and FIG. 16C shows a future image by the interlace method.

  In the progressive method, the positional relationship between pixels and sub-pixels in the reference image and the future image is the same. Since the motion vector candidate is a vector having the position of the sub-pixel to be interpolated as the start point and the pixel within the search range in the future image as the end point, in FIG. 16, (−1, −0.5), (0, −0) .5), (1, -0.5), (-1, 0.5), (0, 0.5), (1, 0.5) are motion vector candidates in the case of the progressive method. On the other hand, in the interlace method, since the field types of the reference image and the future image are different, the positional relationship between the pixel and the sub-pixel is different. In FIG. 16, (-1, -1), (0, -1), (1, -1), (-1, 0), (0, 0), (1, 0), (-1, 1). , (0, 1), (1, 1) are motion vector candidates in the case of the interlace method.

According to the present embodiment, a high-quality resolution conversion method based on appropriate motion compensation with reduced block correspondence errors can be used for scanning line interpolation, so that a high-quality scanning line interpolation image can be obtained.
The example in which the resolution conversion described in the first embodiment is used for scanning line interpolation has been described above. Similarly, the resolution conversion described in the second and third embodiments can be used for scanning line interpolation.

Embodiment 5 FIG.
FIG. 17 is a block diagram showing a configuration of a video display apparatus according to Embodiment 5 of the present invention.
The video display device 400 according to the present embodiment is a device that receives a user-specified signal and a low-resolution video, and displays a high-resolution video with an image quality according to the user's preference, and includes a scene detection unit 401, a parameter setting unit 402, It consists of a resolution conversion device 403 and a display unit 404.

  The scene detection unit 401 detects scene switching of the input low-resolution video. As a method for detecting scene switching, for example, the absolute value of the luminance value of each pixel is taken between adjacent images on the time axis, and there are a certain number of pixels in which the absolute value of the difference is greater than a predetermined threshold. In some cases, there is a method for determining a scene change. The detection result of the scene change is output as a detection signal and input to the parameter setting unit 402.

  The parameter setting unit 402 sets resolution conversion parameters based on a detection signal (scene change detection result) obtained from the scene detection unit 401 and a user-specified signal that is an input, and outputs the parameters to the resolution conversion apparatus 403 as parameter signals. . The user designation signal is a signal designated by a user from a remote controller or the like. By preparing a plurality of parameter sets according to the user-designated signal, it is possible to set parameters according to the user's preference. Also, a parameter set for scene switching can be prepared.

  The resolution conversion apparatus 403 is an apparatus that outputs a high-resolution image with an image quality corresponding to a parameter signal with respect to an input video. Specifically, it is realized by a device having the same function as the resolution conversion device of the third embodiment.

The parameter signals output from the parameter setting unit 402 and input to the resolution conversion device 403 are, for example, threshold values th2 and th3 for determining an interpolation value used in the interpolation value synthesis unit 302 in the device of the third embodiment. The parameter set is set by designation from the user, and before and after the scene switching, the parameter set is switched to the scene switching parameter set. For example, a parameter set in which th2 and th3 take large values so as to prioritize motion compensation interpolation according to the user's specification is output, and th2 and th3 are small values so that intra-image interpolation is prior to and after scene switching. It is also possible to output a parameter set that takes
As described above, the interpolation value synthesis unit 302 of the resolution conversion apparatus 403 determines the synthesis conditions for the motion compensated interpolation image and the intra-image interpolation image based on the parameters set by the parameter setting unit 402.

  The display unit 404 receives the high-resolution video output from the resolution conversion device 403 and displays it on the screen.

  According to the present embodiment, since the resolution conversion of the image quality according to the user setting is performed, a high-resolution high-resolution image according to the user preference can be displayed.

  100, 200, 300 Resolution conversion device, 101 Future image clipping unit, 102 Past image clipping unit, 103 Dissimilarity calculation unit, 104 Motion estimation unit, 105 Search range determination unit, 106 Motion compensation interpolation processing unit, 107 Combination unit, 110 Integration processing unit, 201a, 201b Flatness calculation unit, 202 Motion estimation unit, 301 Intra-image interpolation processing unit, 302 Interpolation value synthesis unit, 400 Video display device, 401 Scene detection unit, 402 Parameter setting unit, 403 Resolution conversion device 404 display unit, 500 scanning line interpolation device.

Claims (22)

In a plurality of continuous images, a reference image to be increased in resolution, a future image positioned one time later on the time axis with respect to the reference image, and one time previous on the time axis with respect to the reference image Use the past image located as the input image,
A search range determination unit for determining a search range of a motion vector;
Based on the motion vector candidate within the search range determined by the search range determination unit and the coordinates of the pixel to be interpolated on the reference image, a block composed of a plurality of pixels from the future image is larger than the width of the search range A future image cutout unit that cuts out with a predetermined size set to
A past image cutout unit that cuts out a block that is point-symmetric with a block cut out by the future image cutout unit with respect to the pixel to be interpolated on the reference image;
A dissimilarity calculation unit that calculates a dissimilarity between two blocks cut out by the future image cutout unit and the past image cutout unit;
A motion estimator that outputs a motion vector that minimizes a motion vector evaluation value obtained based on the dissimilarity calculated by the dissimilarity calculator;
A motion compensation interpolation processing unit that obtains pixel information from a future image and a past image based on a motion vector output from the motion estimation unit, and generates an interpolation image by generating an interpolation pixel from the pixel information;
An integrated processing unit that obtains an output image by integrating a reference image and an interpolation image generated by the motion compensation interpolation processing unit or a synthetic interpolation image generated based on the interpolation image;
The search range determination unit determines a search range in the high resolution processing when the next image is a reference image based on a motion vector output from the motion estimation unit when each image is a reference image. A resolution converter characterized by the above.   The resolution conversion apparatus according to claim 1, wherein the search range determination unit sets a predetermined range centered on a motion vector output from the motion estimation unit as a search range. The search range determination unit sets a predetermined range centered on a pixel to be interpolated on a reference image as an effective range, and excludes a range outside the effective range from the search range. The resolution converter described in 1.   The future image cutout unit determines that the future when the width of the search range in which the center of the block moves is vertical h pixels (where h is a positive integer) and horizontal w pixels (where w is a positive integer). 2. A block composed of a plurality of pixels is cut out from an image with a predetermined size set to vertical (2 × h−1) pixels or more and horizontal (2 × w−1) pixels or more. 4. The resolution conversion apparatus according to any one of 3 above. A first flatness calculation unit that calculates flatness of an image of a block cut out by the future image cutout unit and outputs a flatness calculation result;
A second flatness calculation unit that calculates the flatness of the image of the block cut out by the past image cutout unit and outputs the flatness calculation result;
The motion estimation unit adds a flat area penalty determined based on the flatness calculation results output from the first flatness calculation unit and the second flatness calculation unit to the dissimilarity. The resolution conversion apparatus according to claim 1, wherein the motion vector evaluation value is obtained. The said motion estimation part calculates | requires the said motion vector evaluation value by adding the motion amount penalty according to the magnitude | size of a motion vector to the said dissimilarity degree, The any one of Claim 1 to 5 characterized by the above-mentioned. The resolution converter described in 1. The integrated processing unit
An intra-image interpolation processing unit that generates an interpolated image based on information in the image with respect to the reference image;
An interpolation value combining unit that combines the interpolation image generated by the motion compensation interpolation processing unit and the interpolation image generated by the intra-image interpolation processing unit to generate a combined interpolation image;
The interpolation value synthesis unit compares the value of each pixel of the interpolation image generated by the motion compensation interpolation processing unit and the interpolation image generated by the intra-image interpolation processing unit, and according to the comparison result, The resolution conversion apparatus according to claim 1, wherein a composition ratio in the composition is determined. The interpolation value synthesizing unit is configured to perform the synthesis according to an absolute value of a difference between pixel values of the interpolation image generated by the motion compensation interpolation processing unit and the interpolation image generated by the intra-image interpolation processing unit. The resolution conversion apparatus according to claim 7, wherein a composition ratio is determined. In a plurality of continuous images, a reference image to be subjected to scanning line interpolation, a future image positioned one time later on the time axis with respect to the reference image, and one time previous on the time axis with respect to the reference image Use the past image located as the input image,
A search range determination unit for determining a search range of a motion vector;
Based on the motion vector candidate within the search range determined by the search range determination unit and the coordinates of the pixel to be interpolated on the reference image, a block composed of a plurality of pixels from the future image is larger than the width of the search range A future image cutout unit that cuts out with a predetermined size set to
A past image cutout unit that cuts out a block that is point-symmetric with a block cut out by the future image cutout unit with respect to the pixel to be interpolated on the reference image;
A dissimilarity calculation unit that calculates a dissimilarity between two blocks cut out by the future image cutout unit and the past image cutout unit;
A motion estimator that outputs a motion vector that minimizes a motion vector evaluation value obtained based on the dissimilarity calculated by the dissimilarity calculator;
A motion compensation interpolation processing unit that obtains pixel information from a future image and a past image based on a motion vector output from the motion estimation unit, and generates an interpolation image by generating an interpolation pixel from the pixel information;
An integrated processing unit that obtains an output image by integrating a reference image and an interpolation image generated by the motion compensation interpolation processing unit or a synthetic interpolation image generated based on the interpolation image;
The search range determination unit determines a search range in a scanning line interpolation process when a next image is a reference image based on a motion vector output from the motion estimation unit when each image is a reference image. A scanning line interpolation apparatus characterized by the above.   An image display device comprising the resolution conversion device according to any one of claims 1 to 8, or the scanning line interpolation device according to claim 9. The resolution conversion device according to claim 7 or 8,
A scene detection unit for detecting a scene change of the input low-resolution video;
A parameter setting unit for setting a parameter for resolution conversion based on a user-specified signal and a detection result of scene switching obtained by the scene detection unit;
The image display device, wherein the interpolation value synthesis unit of the resolution conversion device determines a synthesis condition for the motion compensated interpolation image and the intra-image interpolation image based on the parameter set by the parameter setting unit. In a plurality of continuous images, a reference image to be increased in resolution, a future image positioned one time later on the time axis with respect to the reference image, and one time previous on the time axis with respect to the reference image Use the past image located as the input image,
A search range determining step for determining a search range of a motion vector;
Based on the motion vector candidates in the search range determined in the search range determination step and the coordinates of the pixel to be interpolated on the reference image, a block composed of a plurality of pixels from the future image is larger than the width of the search range A future image cutout step to cut out at a predetermined size set in
A past image cutout step of cutting out a block in a point-symmetrical position with respect to the block cut out by the future image cutout step with respect to the pixel to be interpolated on the reference image;
A dissimilarity calculating step of calculating a dissimilarity between the two blocks cut out by the future image cutting step and the past image cutting step;
A motion estimation step for outputting a motion vector that minimizes a motion vector evaluation value obtained based on the dissimilarity calculated in the dissimilarity calculating step;
A motion compensation interpolation processing step of acquiring pixel information from a future image and a past image based on a motion vector output from the motion estimation step, and generating an interpolation image by generating an interpolation pixel from the pixel information;
An integration processing step of integrating a reference image and the interpolation image generated in the motion compensation interpolation processing step or a synthetic interpolation image generated based on the interpolation image to obtain an output image;
The search range determining step determines a search range in the high-resolution processing when a next image is set as a reference image based on a motion vector output from the motion estimation step when each image is set as a reference image. A resolution conversion method characterized by the above.   13. The resolution conversion method according to claim 12, wherein the search range determination step uses a predetermined range centered on the motion vector output from the motion estimation step as a search range. The search range determination step includes setting a predetermined range centered on a pixel to be interpolated on a reference image as an effective range, and excluding a range outside the effective range from the search range. Resolution conversion method described in 1.   The future image cut-out step is performed when the search range in which the center of the block moves is vertical h pixels (where h is a positive integer) and horizontal w pixels (where w is a positive integer). 13. A block composed of a plurality of pixels is cut out from an image at a predetermined size set to vertical (2 × h−1) pixels or more and horizontal (2 × w−1) pixels or more. 14. The resolution conversion method according to any one of 14 above. Calculating a flatness of the image of the block cut out by the future image cutout step, and outputting a flatness calculation result;
A flatness calculation step of calculating a flatness of the image of the block cut out by the past image cutout step and outputting a flatness calculation result; and
The motion estimation step adds a flat area penalty determined based on the flatness calculation results output from the first flatness calculation step and the second flatness calculation step to the dissimilarity. The resolution conversion method according to claim 12, wherein the motion vector evaluation value is obtained. The motion estimation step is to obtain the motion vector evaluation value by adding a motion amount penalty corresponding to the magnitude of a motion vector to the dissimilarity. Resolution conversion method described in 1. The integration process steps are
Intra-image interpolation processing step for generating an interpolated image based on information in the image with respect to the reference image;
An interpolation value synthesizing step for synthesizing the interpolation image generated in the motion compensation interpolation processing step and the interpolation image generated in the intra-image interpolation processing step to generate a synthetic interpolation image;
The interpolation value synthesis step compares the value of each pixel of the interpolation image generated in the motion compensation interpolation processing step and the interpolation image generated in the intra-image interpolation processing step, and according to the comparison result, The resolution conversion method according to claim 12, wherein a composition ratio in the composition is determined. The interpolation value synthesizing step is performed in the synthesis according to an absolute value of a difference between pixel values of the interpolation image generated in the motion compensation interpolation processing step and the interpolation image generated in the intra-image interpolation processing step. The resolution conversion method according to claim 18, wherein a composition ratio is determined. In a plurality of continuous images, a reference image to be subjected to scanning line interpolation, a future image positioned one time later on the time axis with respect to the reference image, and one time previous on the time axis with respect to the reference image Use the past image located as the input image,
A search range determining step for determining a search range of a motion vector;
Based on the motion vector candidates in the search range determined in the search range determination step and the coordinates of the pixel to be interpolated on the reference image, a block composed of a plurality of pixels from the future image is larger than the width of the search range A future image cutout step to cut out at a predetermined size set in
A past image cutout step of cutting out a block in a point-symmetrical position with respect to the block cut out by the future image cutout step with respect to the pixel to be interpolated on the reference image;
A dissimilarity calculating step of calculating a dissimilarity between the two blocks cut out by the future image cutting step and the past image cutting step;
A motion estimation step for outputting a motion vector that minimizes a motion vector evaluation value obtained based on the dissimilarity calculated in the dissimilarity calculating step;
A motion compensation interpolation processing step of acquiring pixel information from a future image and a past image based on a motion vector output from the motion estimation step, and generating an interpolation image by generating an interpolation pixel from the pixel information;
An integration processing step of integrating a reference image and the interpolation image generated in the motion compensation interpolation processing step or a synthetic interpolation image generated based on the interpolation image to obtain an output image;
The search range determining step determines a search range in a scanning line interpolation process when a next image is set as a reference image based on a motion vector output from the motion estimation step when each image is set as a reference image. A scanning line interpolation method characterized by the above.   An image display method comprising the resolution conversion method according to any one of claims 12 to 19 or the scanning line interpolation method according to claim 20. The resolution conversion method according to claim 18 or 19,
A scene detection step for detecting a scene change of the input low-resolution video;
A parameter setting step for setting a parameter for resolution conversion based on a user-specified signal and a detection result of a scene change obtained in the scene detection step;
The interpolating value synthesizing step of the resolution conversion method determines a synthesizing condition of the motion compensated interpolated image and the intra-image interpolated image based on the parameter set by the parameter setting step.

Images