JP5251544B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing device, an image processing method, and an image processing program. A frame for generating an interpolated frame image by estimating a motion vector from a plurality of frame images and interpolating using the estimated motion vector. It relates to rate conversion technology.

  In the image display apparatus, when a moving image having a frame rate lower than the displayable frame rate is input, an interpolation frame image generated by interpolation is inserted between two consecutive frame images, thereby reducing the A frame rate conversion technique for smoothly displaying a moving image at a frame rate is known (Patent Document 1). In this method, a motion vector between two previous and subsequent frame images or more is estimated by performing block matching, and a new interpolation frame image is generated and inserted using the estimated motion vector. ing.

  In general, it is difficult to correctly estimate a motion vector when a repeated pattern region exists in an input frame image. For example, as shown in FIG. 15, the target block corresponding to the target pixel in the repetitive pattern area in the previous frame image includes a plurality of corresponding block candidates such as blocks indicated by different broken lines in the subsequent frame image. It is impossible to determine which of these candidates is the correct corresponding block. This is because it is very difficult in image processing to determine which part of a repetitive pattern area in one input frame image corresponds to a repetitive pattern area in another input frame image. Therefore, in the repetitive pattern region, pixels with a correct motion vector estimated and pixels with an incorrect motion vector estimated are mixed, and the quality of the generated interpolated frame image deteriorates.

In order to cope with this problem, in Patent Document 2, it is determined whether or not the motion vector of the pixel is correct from the rate of coincidence between the motion vector of the pixel to be determined and the motion vectors of the surrounding pixels. In the interpolation, the interpolation using the estimated motion vector is switched if it is correct, and the zero vector interpolation not using the motion vector is switched if it is not correct.
Further, in Patent Document 3, a repeated pattern detection unit is provided, and during interpolation, interpolation using an estimated motion vector for a non-repeated pattern area and zero vector interpolation for a repeated pattern area are switched.

JP 2005006275 A JP 2008-135980 A JP 2007-235403 A

  In general, since the frequency of occurrence of small motion is higher than the frequency of occurrence of large motion, when a motion vector is estimated in an area including a plurality of similar patterns, the combination of a plurality of patterns that can be matched is the most. If a short motion vector is selected, there is a higher possibility that interpolation using the correct motion vector can be executed.

However, when the method of Patent Document 2 is used, if there are many pixels in which a motion vector that is longer than a pixel in which a short motion vector is estimated in the vicinity of the pixel to be determined, the short motion vector is determined as an erroneous motion vector. On the contrary, since interpolation using a long motion vector is executed for a pixel for which a long motion vector has been estimated without being determined, the image quality of the interpolation frame may deteriorate.
Further, when the method of Patent Document 3 is used, even when a correct motion vector can be detected, generation of a high-quality interpolated frame image is generated because 0 vector interpolation is used uniformly in the repeated pattern region. It becomes difficult.

  As described above, when the input frame image has a repeated pattern region, there is a problem that the image quality of the generated interpolated frame image is deteriorated.

  An object of the present invention is to solve the above-described problems, and is an image processing capable of generating and outputting a high-quality interpolated frame image even for an input frame image having a repeated pattern region. An apparatus, an image processing method, and an image processing program are provided.

  An image processing apparatus according to the present invention is an image processing apparatus that generates an interpolated frame image to be interpolated between a plurality of frame images, and based on a change in a pixel value in each frame of the plurality of frame images, A search range calculation unit that calculates a search range of a motion vector of an interpolation pixel included in an interpolation frame image, a motion vector estimation unit that estimates the motion vector within the search range calculated by the search range calculation unit, and the motion Interpolation image generation means for generating the interpolation frame image based on the motion vector calculated by the vector estimation means is provided.

  According to the present invention, it is possible to provide an image processing apparatus, an image processing method, and an image processing program capable of generating and outputting a high-quality interpolated frame image.

1 is a block diagram showing an overview of an image processing apparatus according to a first embodiment of the present invention. It is a block diagram which shows the structure of the image processing apparatus concerning 1st embodiment of this invention. It is explanatory drawing about a suitable search range. It is explanatory drawing about an inappropriate search range. It is explanatory drawing showing a matching pattern. 3 is a block diagram showing a configuration of search range calculation means 31. FIG. It is explanatory drawing about calculation of a brightness | luminance change parameter | index. It is explanatory drawing about the speed-up using an adjacent pixel. It is explanatory drawing about luminance change information. It is a flowchart which shows operation | movement of the image processing apparatus concerning 1st embodiment. It is a block diagram which shows the structure of the image processing apparatus concerning 2nd embodiment of this invention. It is a flowchart which shows operation | movement of the image processing apparatus concerning 2nd embodiment. It is a block diagram which shows the structure of the image processing apparatus concerning 3rd embodiment of this invention. It is a flowchart which shows operation | movement of the image processing apparatus concerning 3rd embodiment. It is explanatory drawing about the problem of block matching.

Next, embodiments for carrying out the present invention will be described in detail with reference to the drawings.
First embodiment.
First, the outline of the image processing apparatus according to the first embodiment of the present invention will be described. FIG. 1 is a block diagram showing an outline of the image processing apparatus according to the first embodiment of the present invention.
The image processing apparatus includes a motion vector estimation unit 61, a search range calculation unit 62, and an interpolated image generation unit 63. Here, the case of two frame images before and after is illustrated as the plurality of frame images input to the image processing apparatus in FIG. 1, but the number of input frame images is not limited to the number illustrated here.

The motion vector estimation means 61 estimates a motion vector of an interpolation pixel included in an interpolation frame image to be interpolated between the plurality of frame images based on the input plurality of frame images.
The search range calculation means 62 determines a search range for estimating a motion vector based on the input pixel values of a plurality of frame images.
The interpolated image generating unit 63 generates an interpolated frame image based on the motion vector generated by the motion vector estimating unit 61.

Next, an outline of processing of the image processing apparatus according to the first embodiment of the present invention will be described.
First, when the preceding and following frame images are input, the search range calculating unit 62 determines a motion vector search range based on the input pixel values of the preceding and following frame images. Then, the search range calculation unit 62 outputs the determined search range to the motion vector estimation unit 61.
The motion vector estimation unit 61 acquires the search range output from the search range calculation unit 62. The motion vector estimation means 61 estimates the motion vector of the interpolation pixel included in the interpolation frame image within the acquired search range based on the plurality of input frame images. Then, the motion vector estimation unit 61 outputs the motion vector to the interpolation image generation unit 63.
The interpolated image generation means 63 acquires the motion vector output from the motion vector estimation means 61. Then, the interpolated image generation means 63 generates an interpolated frame image based on the input plurality of frame images and the acquired motion vector.

Next, details of the image processing apparatus according to the first embodiment of the present invention will be described. In this description, the form in the case where two front and rear frame images are input has been described. However, the present method can be easily extended to three or more input frame images.
In the present description, the pixel value may be any value as long as it is based on the pixel, and includes a value including a luminance value of a pixel, a color component such as RGB, a luminance value, and a color value. It may be a lightness value obtained by conversion or a value of an element of a color component such as RGB. Note that the calculation method of the search range in the present embodiment will be described by exemplifying a case where a luminance value is used as a pixel value.

  FIG. 2 is a block diagram showing the configuration of the image processing apparatus according to the first embodiment of the present invention. Referring to FIG. 2, the configuration of the image processing apparatus according to the first embodiment of the present invention includes a search range calculation unit 36, a motion vector estimation unit 31, a motion compensation interpolation unit 32, a 0 vector interpolation unit 33, and a search range. The evaluation unit 34 and the interpolation image synthesis unit 35 are included. The motion compensation interpolation unit 32, the 0 vector interpolation unit 33, the search range evaluation unit 34, and the interpolation image synthesis unit 35 function as an interpolation image generation unit.

The motion vector estimation means 31 receives the previous and next frame images as input and estimates the motion vector of each interpolation pixel included in the motion compensation interpolation image generated by the motion compensation interpolation means 32 using the block matching method. The block matching method is a method of detecting matching image blocks by comparing an image block of a predetermined size in a certain frame image with an image block of the same size in another frame image. In motion vector estimation using the block matching method, the motion vector (dx (x, y), dy (x, y)) of the interpolated pixel at the pixel position (x, y) is expressed by Equation (1) or Equation (2). The evaluation function P0 (x, y, dx, dy) represented by However, F1 (x, y) and F2 (x, y) are the pixel values of the previous frame image and the rear frame image at the pixel position (x, y), respectively, and Bl is the block size.

The motion compensation interpolation unit 32 generates a motion compensation interpolation image based on the estimated motion vector and the preceding and following frame images, and outputs the motion compensation interpolation image to the interpolation image synthesis unit 35. The pixel value of each interpolation pixel of the motion compensated interpolation image is determined as in Expression (3). However, MC (x, y) represents the pixel value of the motion compensated interpolation image at the pixel position (x, y).

The zero vector interpolation unit 33 synthesizes the previous and next frame images without using the motion vector, generates a zero vector interpolation image, and outputs the zero vector interpolation image to the interpolation image synthesis unit 35. The zero vector interpolation image is generated by motion compensation interpolation when all the motion vectors are set to 0 as represented by Expression (4), or is generated by copying one of the preceding and following frame images. However, V0 (x, y) represents the pixel value of the 0 vector interpolation image at the pixel position (x, y).

  The search range calculation means 36 receives the previous and next frame images as input, and sends the search range for estimating the motion vector for each interpolation pixel to the motion vector estimation means 31 and the search range evaluation means 34 to change the luminance of the previous and next frame images within the search range. Information is output to the motion vector estimation means 31. The search range for each interpolation pixel is a three-dimensional space formed by changes in luminance values centered on the target interpolation pixel position in the preceding and following frame images on the X axis, the Y axis, and the luminance value on the Z axis. In the above, a range that provides a similar convex curved surface is set. For example, this range may be a maximum range that provides a similar convex curved surface. The greatest advantage of setting a range that provides a similar convex curved surface is that the candidate area of the pixel position of the other frame image corresponding to the target pixel of the target frame image can be uniquely determined within the range. It is.

  For example, in the two-dimensional space formed by taking the pixel position on the X-axis and the luminance value on the Y-axis as shown in the frame images before and after illustrated in FIGS. 3 (a) and 3 (b), respectively, search is performed. If the change in luminance value is a similar convex curve within the range, the increase area of the other frame image with respect to the increase area of one frame image and the decrease area of the other frame image with respect to the decrease area of one frame image are uniquely determined. be able to. However, in the case where the change in the luminance value can be divided into two or more convex curves in different directions within the search range, as in the frame images before and after illustrated in FIGS. 4 (a) and 4 (b), respectively. In other words, there are a plurality of areas in which the other frame image is increased or decreased with respect to an increase or area in one frame image. That is, there are a plurality of candidate areas for the pixel position of the other frame image corresponding to the target pixel of the target frame image. When motion vector estimation matching processing is executed in such a search range, a different corresponding region is selected for each pixel, and the image quality of the interpolated frame image may deteriorate.

When this is expanded to a three-dimensional space formed by taking the X-axis, Y-axis, and luminance value on the Z-axis, a range in which a similar convex curved surface is formed between the preceding and following frame images with the target interpolation pixel position at the center. The candidate area of the pixel position of the other frame image corresponding to the target pixel of the target frame image can be uniquely determined.
Here, when the change in luminance value can be divided into two or more convex curved surfaces in different directions within the search range, as a result of motion vector estimation, a different corresponding region is selected for each pixel, and the interpolation frame image Image quality deteriorates. However, the search range calculation means 36 sets the search range for each interpolation pixel as the maximum range that gives a similar convex curved surface, so that the change in the luminance value in two or more different directions within the search range is achieved. Image quality deterioration is suppressed so that it cannot be divided into convex curved surfaces.
In addition, when the search range set by the search range calculation unit 36 is used, in a region where there are a plurality of similar patterns, a range including one pattern that is closest to the preceding and following frames is automatically calculated. For this reason, experience shows that the shortest motion vector is detected between similar patterns that are least likely to degrade the image quality of the motion compensated interpolation image.

The search range calculation means 36 calculates the search range by either the first calculation method or the second calculation method described below.
The first search range calculation method prepares a plurality of patterns (each value in the pattern is 0.0 to 1.0) that can be accommodated on a scaleable convex surface as shown in FIG. By setting a range of 1 pixel x 1 pixel at the target interpolation pixel position and expanding the range one pixel at a time up, down, left, and right, and matching each pattern, the range of each frame image (hereinafter “designated range”) The maximum range that satisfies the conditions in each determination is obtained as the search range by repeating the determination of whether the luminance change in () is within the convex curved surface and the determination of whether the convex surface is similar between the frame images.
Thus, the convex curved surface in the calculation of the search range of the present embodiment is a curved surface that falls within the convex curved surface, and in addition to convex curved surfaces such as patterns 10 to 19 shown in FIG. A curved surface that is a part of a convex curved surface is also included, such as a curved surface that monotonously increases or decreases monotonously. A plane such as pattern 1 is also included.

As a method of discriminating whether or not the luminance change in a certain range is within the convex curved surface for each of the front and rear frame images, a pattern that minimizes the evaluation function Q (x, y, ρ, i) given by Equation (5) is used. If it is selected and the value of the evaluation function in the selected pattern is below a preset threshold value, it is determined that it is within the convex curved surface. The evaluation function Q (x, y, ρ, i) is the absolute difference between the luminance value of the pixel within the specified range normalized with Max 1.0 and Min 0.0 and the value of the pattern Zρ, i corresponding to the pixel position. This is a function for calculating the similarity between the designated range of the frame image of interest and the pattern Zρ, i by calculating the sum of values. Where x and y are the x and y coordinates of the target pixel position, ρ is the width of the range to be evaluated and the length of one side of the specified range, i is the index number uniquely indicating each of the patterns to be evaluated, Y ( x, y) is the luminance value of the frame image of interest at the coordinates (x, y), Zρ, i is the specified pattern image with index number i scaled according to the evaluation range, normalized to 1.0 for Max and 0.0 for Min The luminance values of the pixels within the range, Max and Min, mean the maximum value and the minimum value of the luminance values within the specified range of the target frame image, respectively.

To determine whether a convex surface is similar between the previous and next frame images, the combination of the index number of the pattern that minimizes the evaluation function in each of the input previous and subsequent frame images is similar to the combination given in advance manually. Judgment is made based on information on whether or not the curved surface is a curved surface. This information may be stored in an arbitrary storage means (not shown) such as a memory or an HDD (hard disk drive), for example, and may be acquired by the search range calculation means 36 reading it.
In addition, the search range calculation unit 36 outputs the index number of the pattern in which the evaluation function in the search range is minimized to the motion vector estimation unit 31 as the luminance change information of the previous and next frame images in the search range.

  In addition, as a second calculation method of the search range, a change in the luminance value between adjacent pixels in the search range is indexed, and by integrating these, a convex curved surface having similar changes in the luminance value in the search range It may be determined whether or not. FIG. 6 is a block diagram showing a configuration of the search range calculation means 36 in the case where the search range is calculated from a change in luminance value between adjacent pixels. The search range calculation unit 36 in FIG. 6 includes a luminance change index calculation unit 361 and a luminance change information calculation unit 362. The search range calculation unit 36 estimates the motion vector for each interpolation pixel using the preceding and following frame images as input. The luminance change information of the previous and next frame images within the search range is output to the estimation unit 31 and the search range evaluation unit 34 to the motion vector estimation unit 31.

  The luminance change index calculation unit 361 receives the previous and next frame images, and outputs to the luminance change information calculation unit 362 a luminance change index that represents the luminance change between the pixels of the input frame image in the horizontal direction and the vertical direction. To do. The luminance change index is a value given as 1 if the luminance change of adjacent pixels increases, -1 if it decreases, or 0 if it is flat. The luminance change index DH (x, y) in the horizontal direction between coordinates (x, y) and coordinates (x + 1, y) in the image, and coordinates (x, y) and coordinates (x, y + 1) An example of the calculation method of the luminance change index DV (x, y) in the vertical direction between the two is shown in Expression (6) and Expression (7). However, Y (x, y) is a luminance value of coordinates (x, y) in the image, and thres is a preset threshold value.

Further, the luminance change index may be obtained using not only a difference from adjacent pixels but also a difference from some pixels apart. For example, when the luminance change index of each line in the horizontal direction is calculated in order from the left, the luminance change index at the position indicated by “?” In which the luminance change index of the left several pixels is flat and continuous as shown in FIG. May be determined using the difference value between the pixel on the left and the pixel immediately before the luminance change index becomes flat. For example, the luminance change index is determined based on the larger difference value.

  The luminance change information calculation unit 362 is based on the luminance change index output from the luminance change index calculation unit 361, and the luminance change within the search range for estimating the motion vector for each interpolation pixel and the previous and next frame images. Is calculated and output to the motion vector estimation means 31. In the luminance change information calculation means 362, the search range for each interpolation pixel is such that the change in luminance value centered on the target interpolation pixel position in the preceding and following frame images takes the pixel position on the X axis, the Y axis, and the luminance value on the Z axis. A maximum range is set so as to form a similar convex curved surface in the three-dimensional space formed by the above.

  The second calculation method of the search range is a method for obtaining a maximum range (φ (x, y) pixel × φ (x, y) pixel) that forms a similar convex curved surface in the preceding and following frame images at the target interpolation pixel position. First, a range of 1 pixel × 1 pixel of the target interpolation pixel position in the preceding and following frame images is set as an initial value, and the range is expanded for each of the preceding and following frame images (hereinafter referred to as “designated”). By repeating the determination of whether the luminance change in the search range (ρ (x, y) × ρ (x, y)) ”is within the convex surface and the determination of whether the convex surface is similar between frame images The maximum range that satisfies the conditions in each determination is obtained as the search range.

An example of a method for determining whether the luminance change in a specified range in the frame image is within the convex curved surface will be described below.
First, for each horizontal and vertical line within a specified range, the luminance change in the line is: 1. Flat, 2. Monotonic increase Monotonous decrease 4. convex upward Convex downward, 6. Classify into the other six indicators. Hereinafter, this luminance change index in a line is referred to as a line luminance change index. When the target interpolation pixel position (x, y) and the specified search range (ρ (x, y) × ρ (x, y)), p (-(ρ (x, y) -1) / 2 from the interpolation pixel position ≦ p ≦ (ρ (x, y) −1) / 2) The line luminance change index LH (x, y, ρ (x, y), p) in the horizontal direction is output from the luminance change index calculation means 361. Brightness change index DH (x- (ρ (x, y) -1) / 2, y + p) to DH (x + (ρ (x, y) -1) ) / 2-1, y + p) can be easily obtained by considering the combination.

  For example, the line brightness change index when the brightness change index in the range of the target line is all 0 is flat, and the line brightness change index when 1 or 0 is monotonically increased. Similarly, when the target interpolation pixel position (x, y) and the specified search range (ρ (x, y) × ρ (x, y)), p (− (ρ (x, y) −1) from the interpolation pixel position. ) / 2 ≤ p ≤ (ρ (x, y) -1) / 2) The vertical line brightness change index LV (x, y, ρ (x, y), p) is Consider the combination of luminance change index DV (x + p,-(ρ (x, y) -1) / 2) to DV (x + p, (ρ (x, y) -1) / 2-1) Therefore, it can be easily obtained. In addition, when the specified search range is expanded from ρ (x, y) × ρ (x, y) by one pixel up, down, left, and right (ρ (x, y) +2) × (ρ (x, y) +2) A new range of line luminance change index can be obtained at high speed using the original range of line luminance change index. For example, the horizontal line luminance change index LH (x, y, ρ (x, y) + 2, p) of the new line is the line luminance change index LH (x, y, ρ (x , y), p) and the luminance change index DH (x- (ρ (x, y) -3) / 2, y + p) and DH (x + (ρ (x, y) +1) / 2-1, It is also possible to obtain it by three combinations of y + p).

Next, the line luminance change indexes in the horizontal direction and the vertical direction are integrated in the respective directions, and it is determined whether or not the luminance change in the designated search range is within the convex curved surface. An example of a condition as to whether or not the luminance change in the designated search range is within the convex curved surface is shown below.
Condition 1: If there is a line having a line brightness change index classified as other, it is determined that the line does not fit on the convex curved surface.
Condition 2: When there are both a line having a line luminance change index classified as convex upward and a line having a line luminance change index classified as convex downward, it is determined that the line does not fit on the convex curved surface .
Condition 3: There are a plurality of lines having a line brightness change index classified as convex upward or a line brightness change index classified as convex downward, and a line classified as another index exists between these lines. If it exists, it is determined that it does not fit on the convex curved surface.
Condition 4: A line having a line luminance change index classified as convex upward or a line luminance change index classified as convex downward exists, and a line having either a monotonically increasing or monotonic decreasing index exists in this line If it exists across the surface, it is determined that it does not fit on the convex curved surface.
Condition 5: If any of the conditions 1 to 4 is not satisfied, it is determined that the object falls on the convex curved surface.
Finally, the integrated results of the horizontal and vertical line luminance change indexes are compared. As a comparison method, when it is judged that either one does not fit on the convex curved surface, and when it includes lines that are convex in different directions, upward and downward in the horizontal and vertical directions, the convex curved surface In other cases, it is determined that it is within the convex curved surface.

  As a method of discriminating whether or not the brightness change in the specified range between the frame images is a similar convex curved surface, the front frame image and the rear frame image are convex in different directions of upward and downward convexity. It is determined that the convex surface is not similar to the case where

  The search range for motion vector estimation in the target interpolation pixel can also be obtained at high speed by using the search range for motion vector estimation in adjacent interpolation pixels. For example, the search range in the target interpolation pixel at the pixel position (x, y) is changed to the search range (φ (x−1, y) pixel × φ (x−1) in the adjacent interpolation pixel at the pixel position (x−1, y). , y) pixel)). As shown in FIG. 8, the pixel position (x−1, y) is the search range (φ (x−1, y) pixel × φ (x−1, y) pixel) in the adjacent interpolation pixel. The brightness change in the range (black frame in the figure) has a similar convex curved surface in the front and back frame images, and the convex curved surface similar in the front and rear frame images disappears by expanding the region from the top and bottom and left and right or either of them. It is shown that. Therefore, the search range candidates for the target interpolation pixel at the pixel position (x, y) are (φ (x-1, y) -2) pixels × (φ (x-1, y) -2) pixels, φ ( x-1, y) pixel × φ (x-1, y) pixel, (φ (x-1, y) +2) pixel × (φ (x-1, y) +2) pixel Become. For this reason, it is not necessary to set a range of 1 pixel × 1 pixel as an initial value and repeat the determination while expanding the range by one pixel at a time, and the search range at a new pixel position can be calculated by only these three determinations. It becomes possible.

  Further, in the motion vector estimation calculation executed by the motion vector estimation means 31 in the present invention, it is possible to increase the accuracy by using the luminance gradient information of each pixel of the preceding and following frame images within the search range. Therefore, the luminance change information calculating unit 362 moves the horizontal and vertical line luminance change indexes LH1 and LV1 of the previous frame image and the horizontal and vertical line luminance change indexes LH2 and LV2 of the subsequent frame image as luminance change information. It outputs to the vector estimation means 31. In this case, for example, in the target interpolation pixel at the pixel position (x, y), the luminance gradient of the pixel position (x + u, y + v) of the previous frame image within the search range φ (x, y) is LH1 ( When x, y, φ (x, y), v) is monotonically increasing and LV1 (x, y, φ (x, y), u) is monotonically decreasing, it can be said that it increases in the vertical direction and decreases in the horizontal direction. It becomes possible to determine the brightness gradient.

  However, if LH1 (x, y, φ (x, y), v) or LV1 (x, y, φ (x, y), u) is convex upward or convex downward, horizontal or vertical It is difficult to describe whether the direction is increasing or decreasing. In addition, as shown in FIG. 9, when the luminance change index of a pixel adjacent in the vicinity of the maximum value or the minimum value is 0 (flat) in a line having an upward or downward convex index, both increase and decrease An area that can be captured is generated. Therefore, the luminance change information calculation unit 362 has, for a line having an upwardly convex or downwardly convex index, an end point of an area where the luminance change index of adjacent pixels near the maximum value or the minimum value is 0 (flat). By adding the position and outputting it as luminance change information, it is possible to discriminate whether each pixel in the line is increased or decreased or both.

The motion vector estimation unit 31 inputs the front and rear input frame images, the search range for each interpolation pixel and the luminance change information, estimates the motion vector of each interpolation pixel, and outputs the motion vector to the motion compensation interpolation unit 32. When the motion vector of each interpolated pixel is calculated by matching with a pattern whose search range is prepared in advance, the block matching method represented by Expression (1) or Expression (2) is used regardless of the luminance change information. Presumed. In addition, when the search range is calculated based on an index of change in luminance value between adjacent pixels, the motion vector (dx (x, y), dy ( x, y)) is determined from the input luminance change information as to whether or not the pair to be matched has the same luminance gradient, and the evaluation function represented by the equation (8) or the equation (9) using the block matching method It may be calculated by setting d (x, dy) to minimize P (x, y, dx, dy). However, the range of dx and dy is-(φ (x, y) -1) / 2 ≦ dx, dy ≦ (φ () based on the search range input from the search range calculation means 36 for each interpolation pixel. x, y) -1) / 2. F1 (x, y) and F2 (x, y) are the pixel values of the previous frame image and the rear frame image at the pixel position (x, y), respectively, and Bl is the block size.

  Whether the luminance gradient is the same or not is determined by reading out the luminance gradient in the horizontal and vertical directions at the pixel position for matching the preceding and following frame images from the luminance change information, and the luminance gradient in each direction between the pixels to be matched. Judgment is made by checking whether they match. However, when the luminance gradient in a certain direction of the target pixel is flat, or when the increase or decrease is near the maximum / minimum values, such as pixels “L” to “R” on the line shown in FIG. Is considered to apply to both increases and decreases.

The search range evaluation unit 34 receives the search range in each interpolation pixel output from the search range calculation unit 36 and outputs a search range evaluation result indicating the certainty of the motion vector at each interpolation pixel position to the interpolation image synthesis unit 35. To do.
The narrow search range output from the search range calculation means 36 means that there are a plurality of similar patterns such as repetitive patterns at relatively close positions around the target interpolation pixel position of the input frame image. On the other hand, the fact that the search range output from the search range calculation means 36 is sufficiently wide means that a plurality of similar patterns do not exist around the target interpolation pixel position of the input frame image. The search range output from the search range calculation means 36 is set so as to estimate the shortest motion vector that frequently occurs even in a region where a plurality of similar patterns exist. The possibility that the vector is correct is less likely that the motion vector estimated in an area where such a pattern does not exist is correct. Therefore, it can be said that the certainty of the motion vector estimated by the motion vector estimation unit 31 when the search range output from the search range calculation unit 36 is narrow is lower than that when the search range is wide.

Therefore, the search range evaluation means 34 is a search range evaluation that represents the certainty of motion vector estimation based on the search range (φ (x, y) × φ (x, y)) at the pixel position (x, y). The result α (x, y) (0 ≦ α (x, y) ≦ 1) is set so that it approaches 1 as φ (x, y) increases and approaches 0 as φ (x, y) decreases, Output to the interpolated image synthesis means 35. An example of a method for calculating α (x, y) is shown in Expression (10). However, Tr1 and Tr2 are preset threshold values.

The interpolation image synthesis means 35 receives the motion compensated interpolation image, the 0 vector interpolation image, and the search range evaluation result, and synthesizes the motion compensated interpolation image and the 0 vector interpolation image based on the search range evaluation result. Generate an image. An example of generating an interpolated frame image is shown in Equation (11). However, Fo (x, y) represents the pixel value of the interpolated frame image at the pixel position (x, y).

Next, the overall operation of the present embodiment will be described in detail with reference to the flowcharts of FIGS.
The search range calculation unit 36 calculates the search range for estimating the motion vector and the luminance change information representing the luminance change of the previous and next frame images within the search range based on the input pixels of the previous and next frame images ( S001).
The motion vector estimation means 31 estimates a motion vector for each interpolation pixel using the block matching method based on the input search range and the luminance change information in the search range (S002).
Based on the estimated motion vector, the motion compensation interpolation unit 32 synthesizes the input previous frame image and the subsequent frame image to generate a motion compensation interpolation image (S003).
The zero vector interpolation means 33 generates a zero vector interpolation image from the input previous frame image and subsequent frame image (S004).
The search range evaluation means 36 calculates a search range evaluation result from the search range of motion vector estimation for each interpolation pixel (S005).
The interpolated image synthesizing unit 35 synthesizes the motion compensated interpolated image and the 0 vector interpolated image based on the search range evaluation result to generate an interpolated frame image (S006).

  In the present embodiment, by setting a range that has a similar convex curved surface with the interpolation pixel position as the center as a search range, two similar image patterns are included in the input previous and subsequent frame images. The search range is determined so as not to occur. Here, in a three-dimensional locus space in which the pixel position is taken on the X axis and the Y axis, and the pixel value is taken on the Z axis, the distribution of the pixel values of the preceding and following frame images centered on the position of the interpolation pixel is the search range. By making a search range a range in which the total number of local maximum points and local minimum points included in the range excluding the boundary is 1 or less, two similar image patterns can be included in the input previous and subsequent frame images. The search range may be determined so as not to exist.

Second embodiment.
FIG. 11 is a block diagram showing a configuration of an image processing apparatus according to the second embodiment of the present invention. Referring to FIG. 11, the image processing apparatus according to the second embodiment of the present invention includes a search range calculation means 36, a motion vector estimation means 31, a motion compensation interpolation means 32, a 0 vector interpolation means 33, and a search range evaluation means. 34, luminance change information evaluation means 47, and interpolation image synthesis means 45. The configuration of the image processing apparatus according to the second embodiment of the present invention is different from the configuration of the image processing apparatus according to the first embodiment in the luminance change information evaluation unit 47 and the interpolated image synthesis unit 45. Different. Details of the luminance change information evaluation unit 47 and the interpolated image synthesis unit 45 will be described below. The motion compensation interpolation unit 32, the 0 vector interpolation unit 33, the search range evaluation unit 34, the interpolation image synthesis unit 35, and the luminance change information evaluation unit 47 function as an interpolation image generation unit.

The luminance change information evaluation unit 47 receives the luminance change information output from the search range calculation unit 36, and evaluates the luminance change information indicating the certainty of the estimated motion vector estimated from the luminance change information for each interpolation pixel. The result is output to the interpolated image synthesis means 35.
In general, it can be said that it is easier to obtain an accurate motion vector when the luminance change within the same range is simple than when the luminance change of the preceding and following frame images at the interpolation pixel position is complicated. Therefore, the luminance change information evaluation unit 47 uses the luminance change information evaluation result β (x, y) (0 ≦ β (x, y) ≦ 1) at the pixel position (x, y) as input luminance change information. Decide on the basis. When the search range is calculated by matching with a pattern prepared in advance, the luminance change information evaluation result is determined by a value corresponding to a preset combination of luminance change information. This value may be stored, for example, in an arbitrary storage means (not shown) and acquired by the luminance change information evaluation means 47 reading it out. In addition, when the search range is calculated based on the index of change in luminance value between adjacent pixels, the search range is determined as follows using the line luminance change index in the luminance change information in the search range.

First, the luminance change information is classified into one of the following categories based on the line luminance change index.
Category 1. All the line brightness change indexes are flat.
Category 2. All line brightness change indicators are flat or monotonically increasing or flat or monotonically decreasing.
Category 3. All the line brightness change indexes are flat or monotonically increasing or monotonically decreasing.
Category 4. Line brightness change index that is convex upward or downward is included. Next, luminance change information evaluation result of category 1> luminance change information evaluation result of category 2> luminance change information evaluation result of category 3> luminance of category 4 Β (x, y) is determined so that the change information evaluation result is obtained. For example, the value of the corresponding category is read from a table stored in an arbitrary storage means (not shown) and substituted for β (x, y).

The interpolated image synthesis means 45 receives the motion compensated interpolated image, the zero vector interpolated image, the search range evaluation result, and the luminance change information evaluation result, and calculates each interpolation pixel calculated from the search range evaluation result and the luminance change information evaluation result. The motion compensated interpolation image and the 0 vector interpolation image are synthesized as shown in Expression (12) based on the synthesis weight of (1) to generate an interpolation frame image. However, w1 (x, y) represents the composite weight at the pixel position (x, y).

The composite weight of each interpolation pixel is calculated based on Expression (13), Expression (14), and Expression (15).

Next, the overall operation of the present embodiment will be described in detail with reference to the flowcharts of FIGS. 11 and 12.
The search range calculation unit 36 calculates the search range for estimating the motion vector and the luminance change information representing the luminance change of the previous and next frame images within the search range based on the input pixels of the previous and next frame images ( S101).
The motion vector estimation means 31 estimates a motion vector using the block matching method based on the input search range and luminance change information in the search range for each interpolation pixel (S102).
The motion compensation interpolation unit 32 combines the input previous frame image and the subsequent frame image based on the estimated motion vector to generate a motion compensation interpolation image (S103).
The zero vector interpolation means 33 generates a zero vector interpolation image from the input previous frame image and subsequent frame image (S104).
The search range evaluation unit 36 calculates a search range evaluation result from the search range of motion vector estimation for each interpolation pixel (S105).
The luminance change information evaluation unit 47 calculates the luminance change information evaluation result from the luminance change information within the search range of motion vector estimation for each interpolation pixel (S106).
The interpolated image synthesizing unit 45 synthesizes the motion compensated interpolated image and the 0 vector interpolated image based on the synthesized weight calculated from the search range evaluation result and the luminance change information evaluation result, and generates an interpolated frame image (S107). ).

Third embodiment.
FIG. 13 is a block diagram showing a configuration of an image processing apparatus according to the third embodiment of the present invention. Referring to FIG. 13, the image processing apparatus according to the third embodiment of the present invention includes a search range calculation means 36, a motion vector estimation means 51, a motion compensation interpolation means 32, a 0 vector interpolation means 33, and a search range evaluation means. 34, luminance change information evaluation means 47, block difference evaluation means 58, and interpolated image composition means 55. Compared with the configuration of the image processing apparatus according to the second embodiment, the configuration of the image processing apparatus according to the third embodiment of the present invention includes a motion vector estimation unit 51, a block difference evaluation unit 58, and an interpolation. It differs in the image composition means 55. Details of the motion vector estimation unit 51, the block difference evaluation unit 58, and the interpolated image synthesis unit 55 will be described below. The motion compensation interpolation unit 32, the 0 vector interpolation unit 33, the search range evaluation unit 34, the interpolation image synthesis unit 35, the luminance change information evaluation unit 47, and the block difference evaluation unit 58 function as an interpolation image generation unit.

  The motion vector estimation means 51 receives the preceding and following frame images, the search range for each interpolation pixel and the luminance change information, estimates the motion vector of each interpolation pixel, and blocks the motion vector for each interpolation pixel to the motion compensation interpolation means 32. The difference value is output to the block difference evaluation means 58. The motion vector estimation means 51 differs from the motion vector estimation means 31 in the first and second embodiments in that it outputs a block difference value for each interpolation pixel. The block difference value S (x, y) at the interpolated pixel at the pixel position (x, y) is the value of the evaluation function (P0 (x, y, y)) for the estimated motion vector (dx (x), dy (y)). y, dx (x), dy (y)) or P (x, y, dx (x), dy (y))).

The block difference evaluation means 58 receives the block difference value at each interpolation pixel output from the motion vector estimation means 51 as an input, and the block difference evaluation representing the certainty of the motion vector at each interpolation pixel position based on the block difference value. The result is output to the interpolated image synthesis means 35.
The block difference value represents the similarity between the pixel pairs of the preceding and following frame images associated with the estimated motion vector. The smaller the block difference value, the higher the similarity, and the larger the block difference value, the lower the similarity. To do. Therefore, it can be said that the certainty factor of the motion vector estimated by the motion vector estimation means 51 is lower when the block difference value is larger than when the block difference value is small. Therefore, the block difference evaluation means 58 uses the block difference evaluation result γ (x, y) (0 ≦ γ (x, y)) based on the block difference value S (x, y) at the pixel position (x, y). ≦ 1) is set so as to approach 1 as S (x, y) decreases, and approaches 0 as S (x, y) increases, and output. An example of a method for calculating γ (x, y) is shown in Expression (16). However, Tb1 and Tb2 are preset threshold values.

The interpolated image synthesizing means 55 receives the motion compensated interpolated image, the 0 vector interpolated image, the search range evaluation result, the luminance change information evaluation result, and the block difference evaluation result, and inputs the search range evaluation result, the luminance change information evaluation result, and the block difference. Based on the synthesis weight of each interpolation pixel calculated from the evaluation result, the motion compensated interpolation image and the 0 vector interpolation image are synthesized as in Expression (17) to generate an interpolation frame image. However, w2 (x, y) represents the composite weight at the pixel position (x, y).

The composite weight of each interpolation pixel is calculated based on Equation (18), Equation (19), and Equation (20).

Next, the overall operation of the present embodiment will be described in detail with reference to the flowcharts of FIGS.
The search range calculation unit 36 calculates the search range for estimating the motion vector and the luminance change information representing the luminance change of the previous and next frame images within the search range based on the input pixels of the previous and next frame images ( S201).
The motion vector estimation means 51 estimates the motion vector using the block matching method based on the input search range and the luminance change information in the search range for each interpolation pixel, and the motion vector and the block difference at that time The value is output (S202)
The motion compensation interpolation unit 12 synthesizes the input previous frame image and the subsequent frame image based on the estimated motion vector to generate a motion compensation interpolation image (S203).
The zero vector interpolation means 13 generates a zero vector interpolation image from the input previous frame image and subsequent frame image (S204).
The search range evaluation unit 36 calculates a search range evaluation result from the search range of motion vector estimation for each interpolation pixel (S205).
The luminance change information evaluation unit 47 calculates a luminance change information evaluation result from the luminance change information within the search range of motion vector estimation for each interpolation pixel (S206).
The block difference evaluation means 58 calculates a block difference evaluation result from the block difference value for each interpolation pixel (S207).
The interpolation image synthesis means 55 synthesizes the motion compensated interpolation image and the 0 vector interpolation image based on the synthesis weight calculated from the search range evaluation result, the luminance change information evaluation result, and the block difference evaluation result, Is generated (S208).

  As described above, the image processing apparatus according to the present embodiment limits the range of motion vector estimation to a range in which image quality degradation is unlikely to occur by examining the luminance change in the preceding and following frame images around the interpolation pixel position. In addition, in order to suppress interpolation using an incorrect motion vector by calculating the certainty of motion vector estimation from the range of the range, high-quality interpolated frame images even for input frame images having repetitive pattern regions Can be generated and output.

The image processing apparatus according to the present invention described above supplies a storage medium storing a program for realizing the functions of the above-described embodiments to the system or apparatus, and the computer or CPU (Central Processing Unit) included in the system or apparatus. MPU (Micro Processing Unit) can be configured by executing this program.
In addition, this program can be stored in various types of storage media and can be transmitted via a communication medium. Here, the storage medium includes, for example, a flexible disk, hard disk, magnetic disk, magneto-optical disk, CD-ROM (Compact Disc Read Only Memory), DVD (Digital Versatile Disc), BD (Blu-ray Disc), ROM ( A read only memory (RAM) cartridge, a battery backup RAM (Random Access Memory) memory cartridge, a flash memory cartridge, and a nonvolatile RAM cartridge are included. The communication medium includes a telephone line wired communication medium and a microwave line wireless communication medium, and includes the Internet.

Further, when the computer executes the program that realizes the functions of the above-described embodiment, not only the functions of the above-described embodiment are realized, but also the computer is operating on the basis of the instructions of this program. A case where the functions of the above-described embodiment are realized in cooperation with an OS (Operating System) or application software is also included in the embodiment of the invention.
Further, when the functions of the above-described embodiment are realized by performing all or part of the processing of the program by a function expansion board inserted into the computer or a function expansion unit connected to the computer, the present invention may be implemented. It is included in the form.

  The present invention can be applied to a frame rate conversion device that converts a frame rate by generating an interpolated frame image from the input previous and subsequent frame images.

31, 51, 61 Motion vector estimation means 32 Motion compensation interpolation means 33 0 Vector interpolation means 34 Search range evaluation means 35, 45, 55 Interpolated image synthesis means 36, 62 Search range calculation means 47 Luminance change information evaluation means 58 Block difference evaluation Means 63 Interpolated image generation means 361 Brightness change index calculation means 362 Brightness change information calculation means

Claims (36)

An image processing device for generating an interpolated frame image to be interpolated between a plurality of frame images,
Search range calculation means for calculating a search range of a motion vector of an interpolation pixel included in the interpolation frame image based on a change in a pixel value in each frame of the plurality of frame images;
Motion vector estimation means for estimating the motion vector within the search range calculated by the search range calculation means;
Interpolation image generation means for generating the interpolation frame image based on the motion vector estimated by the motion vector estimation means ,
The search range includes a distribution of pixel values of each of the plurality of frame images centered on the position of the interpolated pixel in a three-dimensional space in which the pixel position is on the X axis and the Y axis and the pixel value is on the Z axis. A range of convex curves,
Image processing device.   The image processing apparatus according to claim 1, wherein the search range is determined so as not to include two image patterns similar to each other in each frame image.   The search range includes a distribution of pixel values of each of the plurality of frame images centered on the position of the interpolated pixel in a three-dimensional space in which the pixel position is on the X axis and the Y axis and the pixel value is on the Z axis. The image processing apparatus according to claim 1, wherein the total number of local maximum points and local minimum points included in a range excluding the boundary of the search range is one or less. The search range, the distribution of the pixel values of each of the plurality of frame images, in any one of claims 1 to 3 is a range of a similar convex surface between each of the plurality of frame images The image processing apparatus described. The search range calculation unit determines that the distribution of the pixel values is a convex curved surface when the distribution of the pixel values is similar to any of the predetermined pattern images,
The image processing apparatus according to claim 4 , wherein when each of the pattern images similar to the plurality of frame images is a set of pattern images determined to be similar in advance, the image processing apparatus determines that the similar convex curved surface. The search range calculation means is based on information indicating the type of change in the pixel value of each line in the X axis direction and the Y axis direction when the pixel position of the frame image is on the X axis and the Y axis. The image processing apparatus according to claim 4 , wherein it is determined whether the similar convex curved surface is obtained. The interpolated image generating means includes a motion compensated interpolated image generated based on the motion vector, and a 0 vector interpolated image generated based on a motion vector set to 0 vector or any one of the plurality of frame images. Are combined by a ratio based on the width of the search range to generate the interpolated frame image,
The interpolation pixels, the image processing apparatus according to any one of claims 1 to 6, which is a pixel included in the motion compensated interpolation image. The image processing apparatus according to claim 7 , wherein the interpolation image generation unit increases a ratio of synthesizing the motion compensated interpolation image as the search range becomes wider. The interpolation image generating unit, and based on the change in pixel value of the frame image in said search range, the image processing apparatus according to claim 7 or 8 to determine the percentage. 10. The image according to claim 9 , wherein the interpolation image generation unit increases the ratio of synthesizing the motion-compensated interpolation image as the change of the pixel value of the frame image within the search range becomes simple. Processing equipment. The interpolation image generating unit, and based on the block difference value based on the difference between the pixel values of the plurality of frame images, the image processing apparatus according to claim 9 or 10 to determine the ratio. The image processing apparatus according to claim 11 , wherein the interpolation image generation unit increases the ratio of synthesizing the motion compensated interpolation image as the block difference value decreases. An image processing method for generating an interpolated frame image to be interpolated between a plurality of frame images,
A search range calculation step of calculating a search range of a motion vector of an interpolation pixel included in the interpolation frame image based on a change in a pixel value in each frame of the plurality of frame images;
A motion vector estimation step for estimating the motion vector within the search range;
An interpolation image generation step of generating the interpolation frame image based on the motion vector ;
The search range includes a distribution of pixel values of each of the plurality of frame images centered on the position of the interpolated pixel in a three-dimensional space in which the pixel position is on the X axis and the Y axis and the pixel value is on the Z axis. A range of convex curves,
Image processing method. The image processing method according to claim 13 , wherein the search range is determined so as not to include two image patterns similar to each other in each frame image. The search range includes a distribution of pixel values of each of the plurality of frame images centered on the position of the interpolated pixel in a three-dimensional space in which the pixel position is on the X axis and the Y axis and the pixel value is on the Z axis. The image processing method according to claim 13 or 14 , wherein the total number of local maximum points and local minimum points included in a range excluding the boundary of the search range is one or less. 16. The search range according to claim 13 , wherein the search range is a range in which a distribution of the pixel values of each of the plurality of frame images becomes a convex curved surface similar to each of the plurality of frame images. The image processing method as described. In the search range calculation step, when the distribution of the pixel values is similar to any one of the predetermined pattern images, it is determined that the distribution of the pixel values is a convex curved surface,
The image processing method according to claim 16 , wherein when each of the pattern images similar to the plurality of frame images is a set of pattern images determined to be similar in advance, it is determined as the similar convex curved surface. In the search range calculation step, based on the information indicating the type of change in the pixel value of each line in the X axis direction and the Y axis direction when the pixel position of the frame image is on the X axis and the Y axis. The image processing method according to claim 16 , wherein it is determined whether or not the similar convex curved surface is obtained. In the interpolation image generation step, a motion compensated interpolation image generated based on the motion vector and a 0 vector interpolation image generated based on a motion vector set to 0 vector or any one of the plurality of frame images Are combined by a ratio based on the width of the search range to generate the interpolated frame image,
The interpolation pixels, the image processing method according to any one of claims 13 to 18 is a pixel included in the motion compensated interpolation image. 20. The image processing method according to claim 19 , wherein, in the interpolation image generation step, the ratio of combining the motion compensated interpolation images is increased as the search range becomes wider. Wherein in the interpolation image generation step, and based on the change in pixel value of the frame image in said search range, the image processing method according to claim 19 or 20 to determine the ratio. The image according to claim 21 , wherein, in the interpolation image generation step, the ratio of synthesizing the motion compensated interpolation image is increased as the change in the pixel value of the frame image within the search range becomes simple. Processing method. The image processing method according to claim 21 or 22 , wherein in the interpolation image generation step, the ratio is further determined based on a block difference value based on a difference between pixel values of the plurality of frame images. 24. The image processing method according to claim 23 , wherein, in the interpolation image generation step, the proportion of the motion compensated interpolation image is increased as the block difference value decreases. An image processing program for generating an interpolated frame image to be interpolated between a plurality of frame images,
A search range calculation step of calculating a search range of a motion vector of an interpolation pixel included in the interpolation frame image based on a change in a pixel value in each frame of the plurality of frame images;
A motion vector estimation step for estimating the motion vector within the search range;
An interpolation image generation step of generating the interpolation frame image based on the motion vector ;
The search range includes a distribution of pixel values of each of the plurality of frame images centered on the position of the interpolated pixel in a three-dimensional space in which the pixel position is on the X axis and the Y axis and the pixel value is on the Z axis. A range of convex curves,
Image processing program. The image processing program according to claim 25 , wherein the search range is determined so as not to include two image patterns similar to each other in each frame image. The search range includes a distribution of pixel values of each of the plurality of frame images centered on the position of the interpolated pixel in a three-dimensional space in which the pixel position is on the X axis and the Y axis and the pixel value is on the Z axis. the image processing program according to claim 25 or 26 the total number of local maximum points and local minimum point in the range of the one below is included in the range excluding the boundary of the search range. 28. The search range according to any one of claims 25 to 27, wherein the search range is a range in which the distribution of the pixel values of each of the plurality of frame images becomes a similar convex curved surface between each of the plurality of frame images. The image processing program described. In the search range calculation step, when the distribution of the pixel values is similar to any one of the predetermined pattern images, it is determined that the distribution of the pixel values is a convex curved surface,
29. The image processing program according to claim 28 , wherein when each of the pattern images similar to the plurality of frame images is a set of pattern images determined to be similar in advance, the image processing program is determined to be the similar convex curved surface. In the search range calculation step, based on the information indicating the type of change in the pixel value of each line in the X axis direction and the Y axis direction when the pixel position of the frame image is on the X axis and the Y axis. The image processing program according to claim 28 , wherein it is determined whether or not the similar convex curved surface is obtained. In the interpolation image generation step, a motion compensated interpolation image generated based on the motion vector and a 0 vector interpolation image generated based on a motion vector set to 0 vector or any one of the plurality of frame images Are combined by a ratio based on the width of the search range to generate the interpolated frame image,
The interpolation pixels, the image processing program according to any one of claims 25 to 30 is a pixel included in the motion compensated interpolation image. 32. The image processing program according to claim 31 , wherein, in the interpolated image generation step, the ratio of synthesizing the motion compensated interpolated image is increased as the search range becomes wider. The image processing program according to claim 31 or 32 , wherein in the interpolation image generation step, the ratio is further determined based on a change in a pixel value of the frame image within the search range. 34. The image according to claim 33 , wherein, in the interpolation image generation step, the ratio of synthesizing the motion compensated interpolation image is increased as the change in the pixel value of the frame image within the search range becomes simple. Processing program. The image processing program according to claim 33 or 34 , wherein in the interpolation image generation step, the ratio is determined based on a block difference value based on a difference between pixel values of the plurality of frame images. 36. The image processing program according to claim 35 , wherein, in the interpolated image generation step, the ratio of synthesizing the motion compensated interpolated image is increased as the block difference value decreases.