KR20090023099A - Image processing apparatus, method thereof, and program - Google Patents

Abstract

An image processing apparatus, a method thereof and a program are provided to reduce distortion in an image generated by performing interpolation even in an image whose brightness significantly changes and in which distortion tends to occur in motion compensation processing using a motion vector when an interpolation image is generated. An image processing apparatus(1) generates an interpolation image using an input image which is one of moving images having a predetermined frame rate, and displays moving images having a different frame rate on a display unit(2) constituted by an LCD(Liquid Crystal Display). The input image is temporarily stored in a frame memory(11). The frame memory supplies an image preceding the input image by one frame to a motion vector estimation unit(12) and a motion compensation processing unit(13). The motion vector estimation unit includes a sum-of-absolute-difference calculation section(12a), a comparing section(12b), and a motion vector calculation section(12c). For each of pixels in the input image, a motion vector for the pixel is calculated by the motion vector estimation unit using the input image and the immediately preceding image. More particularly, the sum-of-absolute-difference calculation section reads pixels in a block of interest corresponding to a pixel of interest in the immediately preceding image. For each of the reference blocks corresponding to one of the reference pixels, the sum-of-absolute-difference calculation section calculates a sum of absolute differences between pixel values at pixel positions in the block of interest and pixel values at corresponding pixel positions in the reference block. The comparing section compares sums of absolute differences, each of which is obtained for one of the reference pixels, determines a reference pixel whose sum of absolute differences is minimum, and supplies information regarding the reference pixel whose sum of absolute differences is minimum to the motion vector calculation section. The motion vector calculation section calculates a motion vector using the pixel of interest in the input image and the reference pixel whose sum of absolute differences is minimum, and supplies the motion vector to the motion compensation processing unit. The motion compensation processing unit includes a vector analyzing section(13a), a pixel generation section(13b), and a motion compensation image memory(13c). The motion compensation processing unit executes motion compensation processing using the input image, the immediately preceding image, and motion vectors, generates an MC image (motion-compensated image), and supplies the MC image to an error processing unit(14).

Description

Image processing apparatus and method, and program

The present invention relates to Japanese Patent No. JP 2007-219421, filed with the Japan Patent Office on August 27, 2007, Japanese Patent No. JP 2008-186099, filed on July 17, 2008, and Japan, July 17, 2008. It includes topics related to Japanese Patent No. JP 2008-186098 filed with the Japan Patent Office, the contents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus, a method, and a program, and in particular, a distortion of an image generated interpolation can be suppressed even in a motion compensation process using a motion vector in an image in which distortion is likely to occur and a large change in luminance occurs. The present invention relates to an image processing apparatus, a method, and a program.

Generally, a motion vector is detected by a block matching method, a motion compensation process is performed using the motion vector, and a process of temporarily creating a new interpolation image (interpolation frame) from an image (frame) before and after the interpolation image. Is generally present.

However, when the subject moves outside of the scene change, the search range, or when the luminance change is intense during fade-in / fade-out, it is difficult to detect the exact motion vector or the reliability of the motion vector becomes low. . As a result, when the motion compensation process is performed, distortion occurs in the interpolated generated image.

Therefore, a technique has been proposed that provides a function of detecting a scene change, executes a motion compensation process in response to the detected scene change, and suppresses distortion (see Japanese Unexamined Patent Application Publication No. 2007-060192).

In addition, a technique for investigating the spatial correlation of interpolated images, executing a motion compensation process, and suppressing distortion has been proposed (see Japanese Unexamined Patent Application Publication No. 2006-260527).

However, in the technique described in Japanese Unexamined Patent Application Publication No. 2007-060192, since a mechanism for detecting a scene change is required separately, even if the apparatus cost increases and the reliability of the motion vector itself is low, frame interpolation processing is performed as it is. In this case, distortion of the generated interpolation image could not be suppressed.

In addition, in the technique described in Japanese Unexamined Patent Application Publication No. 2006-260527, although the spatial correlation of the interpolation image is examined as an error process, the reliability of the motion vector itself is not necessarily investigated. There is a problem in that it is impossible to suppress distortion of the generated interpolated image.

The present invention has been devised in consideration of such a situation. In particular, in the case of generating an interpolated image, it is possible to suppress distortion of an image generated interpolation even for an image in which distortion is likely to occur in a motion compensation process using a motion vector and the luminance change is large. To make it possible.

The image processing apparatus of the present invention is a pixel value of a pixel of a block of interest that includes a plurality of pixels corresponding to a pixel of interest in a first image, and a second image having a display timing different from that of the first image. Difference absolute value calculation means for calculating an absolute difference value between pixel values of a plurality of pixels of substantially the same arrangement as the block of interest in a reference block corresponding to the reference pixel; First comparing means for comparing the difference absolute values calculated by the difference absolute value calculating means to obtain a minimum difference absolute value; Motion vector estimating means for estimating a motion vector of the pixel of interest based on the reference pixel which is the minimum of the absolute difference values obtained by the first comparing means and the pixel of interest; Motion compensation image generating means for generating a motion compensation image by using a pixel value of each pixel of the first image as a pixel value of corresponding pixels in the motion compensation image based on the motion vector of each pixel of the first image and; First cumulative adding means for cumulatively adding the difference absolute value calculated when a motion vector corresponding to a pixel of the motion compensation image is obtained for each pixel of the first image; Interpolation image generating means for synthesizing each pixel in the first image and each pixel in the corresponding motion compensation image based on the cumulative addition result obtained by the cumulative addition means, to generate an interpolation image; And each of the pixels of the first image corresponds to each of the pixels in the movement compensation image.

The image processing apparatus of the present invention may further include quantization means for quantizing the cumulative addition result using a ratio of the cumulative addition result to a predetermined maximum value set for the cumulative addition result, wherein the interpolation image The generating means synthesizes each pixel of the moving compensation image corresponding to each pixel in the first image based on the cumulative result quantized by the quantization means, and generates an interpolated image.

The interpolation image generating means generates an interpolation image by using the pixels in the first image as it is for pixels whose accumulation result quantized by the quantization means is smaller than a first predetermined threshold amount, For the pixels whose cumulative result quantized by the quantization means is larger than the second predetermined threshold amount, each pixel in the movement compensation image is used as it is to generate an interpolated image.

The reference pixel is in the second image and includes a near range corresponding to the pixel of interest in the first image.

An image processing apparatus of the present invention includes: subtraction means for subtracting a predetermined value from the cumulative addition result in the second image; The cumulative addition in the second image is compared with a value obtained by subtracting a predetermined value from the cumulative addition result in the second image by the subtraction means and the cumulative addition result in the first image. If the value from which the predetermined value is subtracted from the result is larger than the cumulative addition result in the first image, the value obtained by subtracting the predetermined value from the cumulative addition result in the second image is determined in the first image. Is replaced with the cumulative addition result of, and the value obtained by subtracting a predetermined value from the cumulative addition result in the second image is not larger than the cumulative addition result in the first image. The second comparison means for outputting the cumulative addition result of as is may be further included.

An image processing apparatus of the present invention corresponds to a pixel of interest in the first image, and pixel values of pixels in the block of interest including a plurality of pixels, and are disposed in the second image at the same position as the pixel of interest, and A stationary-block sum of absolute differences in blocks of the same structure corresponding to pixels of the same structure as the first image, between pixel values of pixels having the same structure as the pixels in the block of interest. Fixed block sum calculation means for calculating the absolute difference value; Calculating means for calculating a variable for each pixel of interest according to the fixed block sum of the difference absolute values calculated by the fixed block sum calculation means of the difference absolute values and the minimum value of the difference absolute values obtained by the first comparing means; And second cumulative adding means for cumulatively adding the variable calculated by the calculating means to the pixel of interest, wherein the interpolation image generating means is in accordance with the cumulative addition result obtained by the second cumulative adding means. The pixels in the first image and the motion compensation image are synthesized to generate the compensation image, wherein each of the pixels in the first image corresponds to each of the pixels of the motion compensation image.

An image processing apparatus of the present invention corresponds to a pixel of interest in the first image and includes pixel values of pixels in the block of interest including a plurality of pixels, and is disposed in the second image at the same position as the pixel of interest. A fixed block of differential absolute value that calculates a fixed block sum of difference absolute values between pixel values of pixels having the same structure as pixels in the block of interest in a block of the same structure corresponding to pixels of the same structure as the first image. Sum calculation means; Of the fixed block sums of the absolute difference values calculated by the fixed block sum calculating means of the difference absolute values, the absolute absolute sums are accumulated in the region including the pixel of interest corresponding to the fixed block sums of the difference absolute values larger than a predetermined threshold amount. And a second cumulative adding means for adding, wherein the interpolation image generating means includes the pixels in the first image and the motion compensation image according to a cumulative addition result obtained by the second cumulative adding means. Are synthesized to generate the compensation image, wherein each of the pixels in the first image corresponds to each of the pixels of the movement compensation image.

The image processing apparatus of the present invention may further include an average calculating means for calculating an average of the fixed block sums of difference absolute values according to the cumulative addition result obtained by the second cumulative addition means.

The image processing apparatus of the present invention may further include change amount calculation means for calculating a change amount with time with respect to the cumulative addition result obtained by the first cumulative addition means, wherein the interpolation image generation means includes: The compensation image is generated by combining the pixels in the first image and the movement compensation image according to the variation amount obtained by the variation calculation means, and each of the pixels in the first image is assigned to each of the pixels in the movement compensation image. Corresponds.

An image processing apparatus of the present invention comprises: dividing means for dividing the first image into n regions; and multiplication means for multiplying a maximum value n times among the absolute differences of the n areas, wherein the cumulative addition of the absolute absolute sums is performed for one of the n areas divided by the dividing means. Each of the difference absolute values obtained by the cumulative addition means of 1 corresponds to each of the n regions, and the interpolation image generating means is a pixel in the first image according to the difference absolute values obtained by the multiplication means. And the motion compensation image to generate the compensation image, wherein each of the pixels in the first image corresponds to each of the pixels of the motion compensation image.

The first cumulative adding means cumulatively adds the difference absolute values to an area except one area located at the end of the first image.

 An image processing method of the present invention includes a pixel value of pixels of a block of interest including a plurality of pixels corresponding to a pixel of interest in a first image, and a second image having a different display timing from the first image. A difference absolute value calculation step of calculating a difference absolute value difference between pixel values of a plurality of pixels of substantially the same arrangement as the block of interest in a reference block corresponding to the reference pixel; A comparison step of comparing the difference absolute value calculated by the difference absolute value calculation step to determine a minimum difference absolute value; A motion vector estimating step of estimating a motion vector of the pixel of interest based on the reference pixel which is the minimum of the difference absolute values obtained by the comparison step and the pixel of interest; A motion compensation image generation step of generating a motion compensation image by using a pixel value of each pixel of the first image as a pixel value of corresponding pixels in the motion compensation image based on the motion vector of each pixel of the first image and; A first cumulative adding step of cumulatively adding, to each pixel of the first image, a difference absolute value calculated when a motion vector corresponding to a pixel of the motion compensation image is obtained; An interpolation image generation step of generating an interpolation image by synthesizing each pixel in the first image and each pixel in the corresponding motion compensation image based on the cumulative addition result obtained in the cumulative addition step; , Each of the pixels of the first image corresponds to each of the pixels in the movement compensation image.

A computer readable program for causing a computer to execute the following processing, wherein the processing includes: pixel values of pixels of a block of interest including a plurality of pixels corresponding to pixels of interest in a first image, and the first value; A difference absolute value calculation step of calculating a difference absolute value difference between the pixel of interest and a pixel value of a plurality of pixels in substantially the same arrangement in a reference block corresponding to a reference pixel of a second image having a different display timing from and; A comparison step of comparing the difference absolute value calculated by the difference absolute value calculation step to determine a minimum difference absolute value; A motion vector estimating step of estimating a motion vector of the pixel of interest based on the reference pixel which is the minimum of the difference absolute values obtained by the comparison step and the pixel of interest; A motion compensation image generation step of generating a motion compensation image by using a pixel value of each pixel of the first image as a pixel value of corresponding pixels in the motion compensation image based on the motion vector of each pixel of the first image and; A first cumulative adding step of cumulatively adding, to each pixel of the first image, a difference absolute value calculated when a motion vector corresponding to a pixel of the motion compensation image is obtained; An interpolation image generation step of generating an interpolation image by synthesizing each pixel in the first image and each pixel in the corresponding motion compensation image based on the cumulative addition result obtained in the cumulative addition step; , Each of the pixels of the first image corresponds to each of the pixels in the movement compensation image.

The program recording medium is recorded therein and may have a computer readable program.

The image processing apparatus, the method and the program include a pixel value of the pixels of the block of interest including a plurality of pixels corresponding to the pixel of interest in the first image, and a second pixel having a different display timing from the first image. In the reference block corresponding to the reference pixel of the image, the difference absolute value difference between the pixel values of the plurality of pixels in substantially the same arrangement as the block of interest is calculated. A minimum absolute absolute difference is obtained by comparing the absolute difference values calculated by the difference absolute value calculation unit. A motion vector of the pixel of interest is estimated based on the reference pixel which is the minimum of the absolute difference values obtained by the comparison unit and the pixel of interest. A motion compensation image is generated using the pixel value of each pixel of the first image as the pixel value of corresponding pixels in the motion compensation image based on the motion vector of each pixel of the first image. For each pixel of the first image, the absolute difference value calculated when the motion vector corresponding to the pixel of the motion compensation image is obtained is cumulatively added. Based on the cumulative addition result obtained by the cumulative adder, each pixel in the first image and each pixel of the corresponding motion compensation image are synthesized to generate an interpolated image. Each of the pixels of the first image corresponds to each of the pixels in the movement compensation image.

The image processing apparatus of the present invention may be an independent apparatus or may be a block for performing image processing.

According to one embodiment of the present invention, it is possible to suppress the distortion of the image to be interpolated even for the motion compensation process using the motion vector in the image where distortion is likely to occur and the luminance change is large.

1 is an image processing apparatus showing a configuration of an embodiment to which the present invention is applied.

The image processing apparatus 1 of FIG. 1 generates an interpolation image from an input image constituting a moving image of a predetermined frame rate, and displays a moving image having a different frame rate by using an LCD (Liquid Crystal Display) or the like. (2) is indicated.

The frame memory 11 temporarily stores an input image, and the motion vector estimating unit 12 and the motion compensation processing unit 13 store an image immediately before the input image (hereinafter, referred to as a previous image) for the input image. Supplies).

The motion vector estimating unit 12 includes a difference absolute value calculating unit 12a, a comparing unit 12b, and a moving vector calculating unit 12c, and moves each pixel of the input image based on the input image and the previous image. Find the vector. More specifically, the difference absolute value calculating section 12a is configured to determine the pixels of the block of interest corresponding to the pixel of interest (the pixel to be processed) of the input image and the reference block corresponding to each pixel (reference pixel) of the previous image. The pixels are read, and the difference absolute values of the pixel values at respective pixel positions are obtained. The comparison unit 12b compares the sum of the difference absolute values obtained for each of the reference pixels, obtains a reference pixel at which the sum of the difference absolute values is minimum, and supplies the information about the reference pixel to the motion vector calculation unit 12c. do. The motion vector calculation unit 12c calculates a motion vector using the reference pixel whose minimum absolute difference with the pixel of interest of the input image is the minimum, and supplies it to the motion compensation processing unit 13.

The motion compensation processor 13 includes a vector analyzer 13a, a pixel generator 13b, and an MC (Motion Compensation) image memory 13c. The motion compensation processor 13 moves on the basis of an input image, a previous image, and a motion vector. The compensation process is performed to generate an MC image (moving compensation image), and the MC image is supplied to the error processing unit 14. More specifically, the vector analyzer 13a analyzes the motion vector for each pixel of the input image, which is supplied from the motion vector estimator 12, and is required for generation of each pixel of the MC image. The pixel of the input image and the pixel of the previous image are searched. The pixel generation unit 13b generates each pixel in the motion compensation image by using the pixels of the input image retrieved by the vector analysis unit 13a and the pixels of the previous image, and generates the pixels in the MC image memory 13c. Remember Here, in this embodiment, the vector analyzer 13a searches only the pixels of the input image, and the pixel generator 13b generates each pixel in the motion compensation image using only the pixels of the input image. . However, as a matter of course, the pixels of the previous image may also be searched to use both of the pixels of the input image and the pixels of the previous image.

The error processor 14 includes an accumulator 14a, an MEL calculator 14b, and a synthesizer 14c, and the absolute difference value corresponding to the motion vector of each pixel supplied from the motion vector estimator 12 is provided. Based on the sum, the MC image and the input image are synthesized, output to the display unit 2, and displayed. More specifically, the accumulator 14a accumulates and adds each of the minimum difference absolute values calculated when obtaining the motion vector of each pixel of the input image, and supplies them to the MEL calculator 14b. The MEL calculation unit 14b quantizes the cumulative addition result of the difference absolute values, obtains the average error level MEL in the MC image, and supplies it to the synthesis unit 14c. The combining unit 14c synthesizes the input image and the MC image based on the average error level MEL, and outputs the displayed image to the display unit 2 for display.

Next, image processing will be described with reference to the flowchart of FIG. 2.

In step S11, the frame memory 11 is used to determine whether an input image has been supplied, and this process is repeated until it is determined that it has been supplied. In step S11, for example, when an input image exists, in step S12, the frame memory 11 temporarily stores the input image, and immediately before the input image stored at the immediately preceding timing. The image is supplied to the motion vector estimator 12 and the motion compensation processor 13 as an image. By this processing, the input image and the immediately preceding image are supplied to the motion vector estimating unit 12 and the motion compensation processing unit 13, respectively, and only the input image is supplied to the error processing unit 14. Here, in the case where there is no previous image, in step S12, after storing the input image, the state in which the input image and the previous image can be supplied to the motion vector estimating unit 12 and the movement compensation processing unit 13. Since the process after step S13 cannot be realized until it becomes, the process returns to step S11.

In step S13, the motion vector estimating unit 12 executes the motion vector extraction process, extracts the motion vector for each pixel of the input image, supplies it to the motion compensation processing unit 13, and simultaneously extracts the motion vector. The difference absolute value calculated at the time of supply is supplied to the error processing unit 14.

Here, the motion vector extraction process will be described with reference to the flowchart in FIG.

In step S31, the difference absolute value calculation unit 12a sets the unprocessed pixel in the input image to the pixel of interest p (i, j) as the processing target. Here, (i, j) is a coordinate indicating the position of the pixel of the input image.

In step S32, the difference absolute value calculation unit 12a sets the unprocessed reference pixel q (x, y) in the immediately preceding image within the reference range for the pixel of interest. That is, for example, as shown by the oblique portion of FIG. 4, the reference pixel q (x, y) of the previous image F2 corresponding to the pixel of interest p (i, j) in the input image F1 is the pixel of interest p. It is set corresponding to (i, j) and is set within the reference range Z indicated by the one-dot long dotted line in FIG. The reference range Z is a range of the reference pixel q (x, y) that can be established as an end point from the range in which the pixel of interest p (i, j) is movable, that is, the pixel of interest p (i, j) that is the starting point of the motion vector. The range is set. Here, the image F1 in the upper part of FIG. 4 represents an input image, and the image F2 represents the immediately preceding image. Image F11 shows an MC image to be generated. In addition, each cell represents a pixel, and (x, y) is a coordinate representing the position of the pixel of the previous image.

In step S33, the difference absolute value calculating unit 12a extracts the pixel values of the respective pixels constituting the block of interest B corresponding to the pixel of interest p (i, j) and the reference block B 'corresponding to the reference pixel. That is, in the case of FIG. 4, the block of interest B corresponding to the pixel of interest p (i, j) is a block composed of a plurality of pixels in a range indicated by a solid line in the input image F1, and the block of reference pixels q (x, y) The corresponding reference block B 'is a block composed of a plurality of pixels in a range indicated by a solid line in the previous image F2. In Fig. 4, the block of interest B and the reference block B 'are blocks of 15 pixels each.

In step S34, the difference absolute value calculation unit 12a calculates the difference absolute value difference of the pixel values between the pixels at positions corresponding to the block of interest B and the reference block B ', respectively. That is, in the case of FIG. 4, the difference absolute value calculation unit 12a calculates the following equation (1) to obtain the difference absolute value d (S (i, j)) for the pixel p (i, j) of interest.

d (S (i, j, x, y) = Σ | p (i-a, j-b) -q (x-a, y-b) |

(a = -2, -1, 0, 1, 2 b = -1, 0, 1) ............... (1)

In formula (1), Σ | A | indicates that the sum of the absolute values A is obtained when the variable a is changed to -2, -1, 0, 1, 2 and the variable b is changed to -1, 0, 1, respectively. have. That is, the sum total of the absolute difference value of each pixel value is calculated | required about all 15 pixels.

In step S35, the difference absolute value calculating unit 12a determines whether there is an unprocessed pixel within the reference range Z, and if there is an unprocessed pixel, the process returns to step S32. That is, the reference pixel q (x, y) is set for all the pixels in the reference range Z set for the pixel of interest p (i, j), and the absolute difference value dS (i, j, x, y) is set for all the pixels. ), The processing of steps S32 to S35 is repeated.

In step S35, when there is no unprocessed pixel in the reference range Z, that is, when the absolute absolute difference is obtained for all the pixels in the reference range Z, the process proceeds to step S36.

In step S36, the comparison unit 12b compares all the difference absolute values dS (i, j, x, y) obtained for the pixel of interest p (i, j), and has a minimum difference having a minimum value. The absolute value dSm is determined, and the minimum reference pixel qm (x, y) corresponding to the difference absolute value dSm is obtained. That is, when the comparison unit 12b takes the reference pixel q (x, y) on the horizontal axis and takes the absolute absolute match dS on the vertical axis, as shown in FIG. 5, the difference absolute value dS is minimum. Let reference pixel q (x, y) be the minimum reference pixel qm (x, y), and let the absolute absolute difference at that time be the minimum differential absolute value dSm. Here, in the following description, this minimum difference absolute value dSm is referred to as a difference absolute value with respect to the pixel p (i, j) of the input image.

In step S37, the motion vector calculation unit 12c calculates a motion vector v (i, j) based on the information of the pixel of interest p (i, j) and the minimum reference pixel qm (x, y). . That is, the motion vector is obtained by the following formula (2).

v (i, j) = (qmx-pi, qmy-pj) ....... (2)

Here, for equation (2), qmx is the horizontal coordinate of qm (x, y), qmy is the vertical coordinate of qm (x, y), and pi is the horizontal of p (i, j). Is a coordinate in the vertical direction, and pj is a coordinate in the vertical direction of p (i, j). Therefore, equation (2) indicates that the motion vector v (i, j) is based on the pixel p (i, j) on the input image. It shows that it is a vector which makes pixel q (x, y) on an immediately before image into an end point.

In step S38, the comparison unit 12b supplies the minimum difference absolute value dSm to the error processing unit 14 as the difference absolute value dSm of the pixel of interest.

In step S39, the motion vector calculation unit 12c supplies the calculated motion vector v (i, j) of the pixel of interest p (i, j) to the motion compensation processing unit 13.

In step S40, the difference absolute value calculation unit 12a determines whether or not an unprocessed pixel exists in the input image, and when there is an unprocessed pixel, the process returns to step S31. That is, the processes of steps S31 to S40 are repeated until the motion vector and the absolute minimum difference value are obtained for all the pixels in the input image. In step S40, when no unprocessed pixel exists, that is, a motion vector is obtained for all the pixels of the input image, the process ends.

By the above processing, for each pixel of the input image, a reference in which the absolute absolute difference between the pixels of the corresponding positions of the block of interest corresponding to the pixel of interest set and the reference block set to the reference pixel on the previous image is minimized. The pixel is set as the end point of the motion vector, the motion vector of the pixel of interest is obtained, and the motion vector is obtained for the phone of the input image. In addition, when a motion vector is obtained for each pixel in the input image, the minimum difference absolute value obtained for each pixel in the input image is the difference absolute value dSm of each pixel in the input image as an error processing unit ( 14).

Here, it returns to description of the flowchart of FIG.

When the movement vector extraction process ends in step S13, in step S14, the movement compensation processing unit 13 executes the movement compensation process, generates an MC image, and supplies it to the error processing unit 14.

Here, with reference to the flowchart of FIG. 6, the movement compensation process will be described.

In step S51, the motion compensation processing unit 13 sets the unprocessed pixel of the MC image to be interpolated to be the pixel of interest P (s, t).

In step S52, the vector analyzing unit 13a analyzes the motion vector of the input image supplied from the motion vector estimating unit 12, and has an input having a motion vector passing through the pixel of interest P (s, t). The pixel p (i, j) on the image is searched.

That is, for example, as shown in FIG. 7, the vector analyzing unit 13a has a pixel p having a motion vector v (i, j) passing through the pixel of interest P (s, t) of the MC image F11. Search for (i, j). Here, in FIG. 7, as described with reference to FIG. 4, the pixel P (s of interest P (s) of the MC image F11 is among the motion vectors v (i, j) obtained for one pixel among the pixels p (i, j) of the input image F1. , t).

In step S53, the pixel generating unit 13b reads the pixel value of the obtained pixel p (i, j) of the input image F1 as the pixel value of the pixel P (s, t) of interest, and the MC image memory 13c Remember). That is, in this example, the pixel generating unit 13b selects the pixel of the pixel p (i, j) of the input image F1 which is the starting point of the motion vector v (i, j), and the pixel of interest P (s) in the MC image F11. By simply setting as a pixel value of, t), a pixel is generated. The pixel of interest P (s, t) corresponds to the pixel of interest p (i, j). However, as long as the pixel determined based on the motion vector v (i, j) is used, the pixel of interest P (s, t) is, for example, the pixel p (i which becomes the viewpoint of the motion vector v (i, j). , j) and the end point, and the average of the pixel values of the pixel q (x, y) to be the previous image, that is, (p + q) / 2. p and q represent pixel values of the pixel p (i, j) and the pixel q (x, y), respectively, and the pixel of interest P (s, t) is immediately before the MC image and the temporal distance between the input images of the MC image. The weighted average obtained based on the ratio by the temporal distance from the image, that is, the weighted average of the pixel values of the pixels p (i, j) and the pixel q (x, y). That is, w x p + (1-w) x q. p and q represent pixel values of the pixel p (i, j) and the pixel q (x, y), respectively, and w (0 ≦ w ≦ 1) represents a weighting coefficient.

In step S54, the motion compensation processing unit 13 determines whether or not an unprocessed pixel exists in the MC image to be generated by interpolation. For example, when there is an unprocessed pixel, the process proceeds to step ( S51). That is, for each pixel of the MC image to be generated by interpolation, the processing of steps S51 to S54 is repeated until the pixel value of the pixel in the MC image is replaced with the pixel value in the input image. The pixel value in the input image is a pixel of the input image serving as a starting point of the motion vector passing through each of the MC images. In step S54, when there is no unprocessed pixel, that is, for each of all the pixels of the MC image, the pixel value in the input image is the pixel of the input image which is the starting point of the motion vector passing through each of the MC images. If the pixel value of the pixel in the MC image is replaced with the pixel value in the input image, in step S55, the motion compensation processing unit 13 reads out the data of the MC image stored in the MC image memory 13c and replaces the error processing unit ( 14).

By the above process, the MC image which the motion compensation was performed from the input image by the motion compensation process is produced | generated, and is supplied to the error processing part 14. As shown in FIG.

Here, it returns to description of the flowchart of FIG.

In step S14, when the motion compensation process is executed, and the MC image is generated, in step S15, the error processing unit 14 in each pixel of the input image supplied from the motion vector estimating unit 12. On the basis of the difference absolute value of each pixel obtained when extracting the motion vector of, the error processing is executed, the input image and the MC image are synthesized, and an output image is generated and displayed on the display unit 2.

Here, error processing will be described with reference to the flowchart of FIG. 8.

In step S71, the accumulator 14a accumulates and adds the absolute difference value dSm supplied from the motion vector estimator 12. Each of the difference absolute values is obtained at pixel values at corresponding pixel positions in the target block and the reference block at the start and end points of the motion vector. The error processing unit 14 performs normalization by dividing the final total by the maximum value, for example, and then supplies the normalized value to the MEL calculation unit 14b as a dSm cumulative addition result.

In step S72, MEL calculation part 14b compares dSm accumulation addition result with the threshold amount th1, and determines whether dSm accumulation addition result is smaller than threshold amount th1. For example, when the dSm cumulative addition result is smaller than the threshold amount th1, in step S73, the MEL calculation unit 14b sets the average error level MEL of the MC image to the minimum value and supplies it to the combining unit 14c. .

On the other hand, in step S72, for example, when the dSm cumulative addition result is not smaller than the threshold amount th1, in step S74, the MEL calculation unit 14b matches the dSm cumulative addition result with the threshold amount th2 (> th1). In comparison, it is determined whether the dSm cumulative addition result is larger than the threshold amount th2.

In step S74, for example, when the dSm cumulative addition result is larger than the threshold amount th2, in step S75, the MEL calculation unit 14b sets the average error level MEL of the MC image to the maximum value, and the combining unit It supplies to 14c.

In addition, in step S74, for example, when the dSm cumulative addition result is not larger than the threshold amount th2, in step S76, the MEL calculation unit 14b performs a predetermined coefficient based on the dSm cumulative addition result. The multiplied value is calculated and set as the average error level MEL, and then supplied to the combining unit 14c.

That is, for example, as shown in FIG. 9, when the dSm cumulative addition result is smaller than the threshold amount th1, the average error level MEL is set to 0 of the minimum value, and when the dSm cumulative addition result is larger than the threshold amount th2, the average error. The level MEL is set to the maximum MELmax. If the dSm cumulative addition result is not smaller than the threshold amount th1 and the dSm cumulative addition result is not greater than the threshold amount th2, the average error level MEL is set to a value between 0 and MELmax that changes linearly in response to the dSm cumulative addition result. .

In this way, the average error level MEL is set to a value proportional to the dSm cumulative addition result which is the cumulative addition value of the absolute difference sum dSm indicating the magnitude of the change in the pixel value.

For example, in the case of successive images in which a scene change occurs and the subject in the image moves so fast that the subject using the motion vector cannot be tracked, as shown in FIG. 10, the dSm cumulative addition result is a large value. Will have For example, in the case of continuous images in which there is no scene change and a subject can be tracked in a frame using a motion vector, as shown in FIG. 11, the dSm cumulative addition result has a small value. That is, in the case of continuous images in which there is no scene change and a subject can be tracked in a frame using a motion vector, as shown in FIG. do. However, in the case of continuous images in which a scene change occurs and the subject in the image moves so fast that the subject using the motion vector cannot be tracked, as shown in FIG. 10, the dSm cumulative addition result is a value approximated to 1200. Will have 10 and 11, the horizontal axis represents the frame number, and the vertical axis represents the dSm cumulative addition result. 10 and 1 show examples obtained in which dSm cumulative addition results in a constant value used for normalization.

That is, when the minimum absolute difference value of the pixel values between the pixels for which the motion vector is set is large and the difference between the input image and the previous image is large, the average error level MEL becomes large. Therefore, it can be considered that the reliability of the motion vector is low. If the mean error level MEL is small, the minimum value of the absolute difference value of the pixel value between the pixels to which the motion vector is set is small, and the difference between the input image and the previous image becomes small, so that the tracking by the motion vector is made to some extent, so that the motion vector Can be considered to have high reliability.

In step S77, the combining unit 14c synthesizes the input image and all corresponding pixels in the MC image based on the average error level MEL to generate a synthesized image. That is, the combining unit 14c synthesizes each pixel by the operation expressed by the following formula (3).

Pe = (P (i, j) × (MELmax-MEL) + p (i, j) × MEL) / MELmax .......... (3)

In the above formula (3), Pe denotes a pixel generated by synthesis, P (i, j) denotes a pixel value of the pixel P (i, j) of the MC image, and p (i, j) denotes an input image. The pixel value of the pixel p (i, j) is denoted by MEL as the average error level and MELmax as the maximum value of the average error level MEL.

In step S78, the combining unit 14c outputs the synthesized image to the display unit 2 as an output image.

Here, it returns to description of the flowchart of FIG.

In step S16, the display unit 2 displays the output image supplied from the error processing unit 14, the process returns to step S11, and the subsequent processing is repeated.

Here, the above processing has been described for processing using an input image and a previous image, but other images can be used as long as the images have different timings. For example, the images may be an input image and an image immediately after the image, or images two frames before and an input image. Moreover, although the example which uses two images of an input image and the immediately before image is demonstrated, you may make it use more images. For example, processing using an input image, immediately preceding image, and immediately after image can be executed, and also processing using four or more images can be executed.

By the above process, since there is no scene change, an image with high reliability of the motion vector and a high ratio including the MC image generated by the motion compensation process is output to the image in which the movement of the subject is within the frame. Since a change occurs or the movement of the subject moves to an area outside the frame, if the movement of the subject using the movement vector is not tracked and the reliability of the movement vector is low, an image having a high ratio including the input image is output. Therefore, for images with high reliability of the motion vector, images based on the motion compensation process are generated and displayed by interpolation at a high rate. On the contrary, for images with low reliability of the motion vector, the input image is interpolated at a high ratio. Is generated by

As a result, in the case of generating the interpolation image, it is possible to suppress the distortion of the image generated by interpolation even for the interpolation generation process adopting the motion compensation process using the motion vector obtained from the image which is easily distorted and has a large luminance change. Become.

In the above, when the motion vector is extracted, the reference pixel is set within a range that can be established as the end point of the motion vector for the pixel of interest that is the starting point of the motion vector, and each pixel of the corresponding block of interest and the reference block is established. An example of block matching between two frames in which a minimum reference pixel is set as an end point pixel of a motion vector as a difference absolute value difference between two frames has been described. However, by reducing the setting range of the reference pixel with respect to the pixel of interest, block matching is repeatedly performed for a plurality of frames, and the accuracy of the motion vector of the pixel of interest can be gradually improved. Also, the distortion of the image interpolated by the same process can be suppressed.

12 is a diagram illustrating a configuration example of an embodiment according to another image processing apparatus designed to obtain a motion vector by frame traversal block matching. Here, in the image processing apparatus 1 of FIG. 12, the components having the same functions as those provided in the image processing apparatus 1 of FIG. 1 are actually given the same names and symbols, and the description thereof will be omitted. do.

That is, the image processing apparatus 1 of FIG. 12 is different from the image processing apparatus 1 of FIG. 1, instead of the motion vector estimating unit 12 and the error processing unit 14, the cyclic motion vector estimating unit 111. ) And the error processing unit 112.

The cyclic motion vector estimating unit 111 includes a difference absolute value calculating unit 111a, a comparing unit 111b, and a motion vector calculating unit 111c, and basically executes the same process as the motion vector estimating unit 12. However, as described above, when the motion vector is extracted, the setting range of the reference pixel for the pixel of interest is made small so as to cover the range corresponding to the pixel position of the pixel of interest. For this reason, since the cyclic type motion vector estimator 111 is unlikely to be able to detect a reference pixel which is the end point of the correct motion vector in the image of two frames, compared with the motion vector estimator 12. It is unlikely that you will be able to extract the correct motion vector. For this reason, the cyclic type motion vector estimating unit 111 repeats the same process repeatedly between a plurality of frames, and gradually obtains a correct motion vector.

The error processor 112 includes an accumulator 112a, an MEL calculator 112b, a comparator 112c, an adder 112d, and a combiner 112e. Among these, about the accumulator 112a, the MEL calculator 112b, and the combiner 112e, the accumulator 14a, the MEL calculator 14b, and the combiner 14c of the error processor 14 are respectively. Do the same. In the following explanation, however, the MEL calculator 112b expresses the expression of the average error level MEL obtained by the MEL calculator 14b as the average error level MELc of the input image. The comparing unit 112c compares the value MELp-m subtracted by the predetermined value m from the average error level MELp in the previous image using the adder 112d, and the average error level MELc in the input image. . The comparing unit 112c outputs the larger value of the value MELp-m and the average error level MELc to the combining unit 112e as the average error level MELc of the input image, and at the same time, the average error level MELc of the input image. Is stored as the average error level MELp of the previous image. In the subsequent processing, the average error level MELp of the immediately preceding image stored is output to the adder 112d. The combining unit 112e synthesizes the input image and the MC image based on the average error level MELc of the input image, and outputs the displayed image to the display unit 2 for display.

Next, image processing by the image processing apparatus 1 of FIG. 12 will be described. Here, the basic image processing is the same as the processing described with reference to the flowchart of FIG. 2, and therefore the description of the flowchart of FIG. 2 is omitted. The image processing of the image processing apparatus 1 of FIG. 12 differs from the image processing of the image processing apparatus 1 of FIG. 1 in the motion vector extraction processing and the error processing of the processing in the flowchart of FIG. 2. Therefore, the motion vector extraction processing and the error processing by the image processing apparatus 1 of FIG. 12 will be described below.

First, with reference to the flowchart of FIG. 13, the motion vector extraction process by the image processing apparatus 1 of FIG. 12 is demonstrated. Here, the processing of steps S101 and S106 to S110 in the flowchart of FIG. 13 is the same as the processing of steps S31 and S36 to S40 described with reference to the flowchart of FIG. 3, and therefore the description thereof will be omitted. do.

That is, in step S102, the difference absolute value calculating section 111a sets the unprocessed reference pixel q (x, y) in the immediately preceding image within the circular reference range for the pixel of interest. That is, the circular reference range is defined, for example, as shown by the oblique portion in FIG. 14, and refers to the reference pixel q (x, x) in the immediately preceding image F2 corresponding to the pixel of interest p (i, j) of the input image F1. a reference range Z 'including y). The reference range Z 'which is the circular reference range is a reference pixel q that can be established as an end point from a range in which the pixel of interest p (i, j) is movable, that is, the pixel of interest p (i, j) that is the starting point of the motion vector. It is a part of the range of (x, y), and compared with reference to FIG. 4, the reference range Z 'covers the vicinity of the position corresponding to the pixel position of the pixel of interest p (i, j). That is, in FIG. 14, a region including a position corresponding to the position of the pixel of interest p (i, j), and a range up to two pixels in the up, down, left, and right directions corresponding to the position of the pixel of interest p (i, j), It is set as a traversal reference range.

Here, in FIG. 14, the image F1 in the upper portion of FIG. 14 represents the input image, and the image F2 represents the immediately preceding image. Moreover, image F11 has shown the generated MC image. Each cell represents a pixel, and (x, y) is a coordinate representing the position of the pixel of the previous image.

In step S103, the difference absolute value calculating section 111a extracts the pixel values of each pixel constituting the block of interest B corresponding to the pixel of interest p (i, j) and the reference block B 'corresponding to the reference pixel. That is, in the case of FIG. 14, the block of interest B corresponding to the pixel of interest p (i, j) is a block composed of a plurality of pixels in a range indicated by a solid line in the input image F1 and corresponds to the reference pixel q (x, y). The reference block B 'to be described is a block composed of a plurality of pixels in a range indicated by a solid line in the previous image F2. In Fig. 14, the block of interest B and the reference block B 'are each composed of 15 pixels.

In step S104, the difference absolute value calculation unit 111a calculates the difference absolute value difference of the pixel values between the pixels at positions corresponding to the block of interest B and the reference block B ', respectively, by the above-described equation (1).

In step S105, the difference absolute value calculating section 111a determines whether or not an unprocessed pixel exists in the reference range Z ', and if there is an unprocessed pixel, the process returns to step S102. That is, the reference pixel q (x, y) is set for all the pixels in the reference range Z 'set for the pixel of interest p (i, j), and the difference absolute value d (S (i, j, The process of steps S102 to S105 is repeated until x, y is obtained.

Through the above processing, the absolute difference between the pixels of the pixel of interest is obtained for all the reference pixels within the reference range, and the reference pixel at which the obtained absolute difference between the pixels is minimum is determined as the end point of the motion vector in the pixel of interest. do. The motion vector of which the pixel of interest becomes the starting point and the reference pixel is the end point is determined as the motion vector of the pixel of interest.

In this process, as described with reference to FIG. 14, since the reference range Z 'is smaller than the reference range Z described with reference to FIG. 4 and includes a near range corresponding to the pixel position of the pixel of interest, two frames are provided. The accuracy of the motion vector found in the liver becomes low. However, since the same processing is repeated for a plurality of frames, the position of the reference pixel which is the end point of the motion vector becomes close to the end point position of the correct motion vector.

Next, with reference to the flowchart of FIG. 15, the error process by the image processing apparatus 1 of FIG. 12 is demonstrated. Here, since the processing of steps S121 to S126 in the flowchart of FIG. 15 is the same as the processing of steps S71 to S76 described with reference to the flowchart of FIG. 8, the description thereof is omitted. However, the average error level MELc of the input image that can be obtained in steps S1 23, S125, and S126 of the flowchart of FIG. 15 is the average error level of the input image that can be obtained in steps S73, S75, and S76 of FIG. 8. It is the same as MEL.

In other words, when the average error level MELc of the input image is obtained in steps S123, S125, and S126, in step S127, the comparison unit 112c stores the average error level MELp that is stored at the time of processing the previous image. Compares the value MELp-m subtracted by the predetermined value m with the average error level MELc of the input image, and the value MELp-m subtracted by the predetermined value m from the average error level MELp of the immediately preceding image is the average of the input image. It is determined whether the error level is greater than MELc.

In step S127, for example, when it is determined that the value MELp-m subtracted by the predetermined value m from the average error level MELp of the immediately preceding image is larger than the average error level MELc of the input image, in step S128, The comparing unit 112c replaces the average error level MELc of the input image with the value MELp-m subtracted by the predetermined value m from the average error level MELp of the immediately preceding image, to the combining unit 112e and the adder 112d. Output

On the other hand, in step S127, for example, when it is determined that the value MELp-m subtracted by the predetermined value m from the average error level MELp of the immediately preceding image is not larger than the average error level MELc of the input image, step S129. ), The comparing unit 112c outputs the average error level MELc of the input image to the combining unit 112e and the adder 112d as it is.

In step S130, the combining unit 112e synthesizes all the pixels corresponding to each of the input image and the MC image, respectively, based on the average error level MELc by the operation shown in the above formula (3), and the synthesized image. Create In step S130, however, the average error level MELc is used in place of the average error level MEL in the above calculation (3).

In step S131, the combining unit 112e outputs the synthesized image to the display unit 2 as an output image.

In step S132, the adder 112d treats the average error level MELc of the supplied input image as the average error level MELp of the immediately preceding image in the subsequent processing and subtracts by a predetermined value m (MELp−). m) is calculated and supplied to the comparator 112c for storage.

By the above process, even when the average error level of an input image becomes large rapidly and predetermined time is required until it stabilizes, the value (MELp-m) slightly smaller than the average error level MELc of an input image and the average error level of the previous image. By comparing with and selecting a large group, the average error level MELc of the input image is gradually lowered. Until the state of the average error level MELc of the input image is stabilized, the synthesis ratio of the input image is increased, so that the synthesis ratio of MC images having a high probability of distortion can be reduced.

For example, as shown in FIG. 16, even if the first to fourth frames are continuous images, and even if a scene change occurs for the fifth frame, for the sixth to thirteenth frames, the input image is gradually changed. The average error level MELc is reduced. And in the 14th to 16th frames, continuous images are input, for example, in the 17th frame, in the 18th to 21st frames, even if the reliability of the motion vector is lowered and the average error level is increased, Again, the average error level of the input image is reduced step by step, and after the twenty-first frame, it is shown that the average error level of the input image is stable.

In particular, when a motion vector is obtained by frame traversal block matching, the accuracy of the motion vector is gradually improved. Therefore, some time is required to display several frames until the correct motion vector is obtained. For this reason, even if a scene change occurs, a predetermined time is required before the motion vector is stabilized, and distortion is seen in the MC image therebetween.

Therefore, as described above, it is possible to slow down the attenuation speed of the average error level of the input image, gradually reduce the composition ratio of the distorted MC image, display the input image, and display the image naturally.

Here, by setting the predetermined value m, the time until the average error level MELc of the input image is stabilized can be set. If the predetermined value m is made large, the attenuation of the average error level MELc is fast, but the possibility of displaying a distorted MC image is high. On the other hand, when the predetermined value m is reduced, the attenuation of the average error level MELc is slow, but the possibility of displaying a distorted MC image is reduced. Thus, since the predetermined value m can be considered to be a relaxation time for determining the attenuation rate of the average error level MELc, the predetermined value m may be set as necessary to control the composition of the MC image and the input image.

In addition, instead of the error processing part 14 in the image processing apparatus 1 of FIG. 1, the error processing part 112 is arrange | positioned, and even if a motion vector is calculated | required using the absolute difference value between the pixels between two frames, an average error The attenuation of the level may be delayed and the synthesis ratio of the MC image having high distortion potential may be reduced.

In any case, as a result, in the case of generating the interpolation image, the interpolation generation processing adopting the motion compensation process using the motion vector in the image where distortion is likely to occur and the luminance change is large is also distorted. Can be suppressed. The distortion of the MC image can be suppressed even if a method in which the average error level of the input image is greatly changed, such as a motion vector extraction method in which frame-cyclic block matching is performed.

In the above, in frame traversal block matching, the average error level is attenuated even if a distortion in the MC picture requires a predetermined time to obtain an accurate motion vector, especially when the scene change occurs, especially for the discontinuous pictures that are generated. Examples have been described in which the speed is slowed down and distortion in the MC image is suppressed. Here, the predetermined time for obtaining the correct motion vector can be shortened.

17 shows an exemplary view according to another embodiment of an image processing apparatus that can reduce the time for obtaining an accurate motion vector. The image processing apparatus is used to obtain a motion vector by frame traversal block matching. Here, in the image processing apparatus shown in FIG. 17, elements having substantially the same function as those supplied to the image processing apparatus shown in FIG. Indicated. Descriptions thereof will be omitted as necessary.

That is, the difference between the image processing apparatus of FIG. 17 and the image processing apparatus of FIG. 1 is that the cyclic type motion vector estimating unit 131 and the error processing unit 132 have a motion vector estimating unit 12 and an error difference unit 14. Is instead placed.

The cyclic type motion vector estimating unit 131 includes a difference absolute value calculating unit 131a, a comparison unit 131b, a motion vector calculating unit 131c, and a motion vector resetting unit 131d, among which the difference absolute value totals are included. The calculation unit 131a, the comparison unit 131b, and the motion vector calculation unit 131c include, in the image processing apparatus described with reference to FIG. 12, the absolute difference value calculation unit 111a of the iterative motion vector estimation unit 111; The processes performed in practice are substantially the same as those executed by the comparator 111b and the motion vector calculator 111c. In addition, in accordance with the reset signal supplied from the error processing unit 132, a repetitively estimated motion vector (movement vector that becomes a stop vector having a movement amount of 0) is reset. That is, the cyclic motion vector estimator 131 executes substantially the same processes as that performed by the cyclic motion vector estimator 111 and resets the motion vector according to the reset signal.

The error processor 132 includes an accumulator 132a, an MEL calculator 132b, a synthesizer 132c, a threshold value determiner 132d, a frame memory 132e, an adder 132f, and a threshold value determiner 132g. ) And a reset determination unit 132h. Among these, the accumulator 132a, the MEL calculator 132b, and the combiner 132c are executed by the accumulator 14a, the MEL calculator 14b, and the combiner 14c of the error processor 14. And each of them actually executes the same processes. That is, the error processing unit 132 synthesizes, for each of the pixels, the MC image and the input image according to the difference absolute value corresponding to each of the motion vectors supplied from the circular type motion vector estimating unit 131. The synthesized image is output to the display unit 2 for display.

Next, the error processing unit 132 generates a reset signal indicating the resetting of the motion vector according to the average error level and the absolute difference value supplied from the traversal motion vector estimating unit 131, and converts the reset signal into a circular type. Supply to the motion vector estimator 131. In particular, the threshold value determination unit 132d determines whether the average error level calculated by the MEL calculation unit 132b is larger than a predetermined threshold. The frame memory 132e stores the accumulated value of the absolute difference values for the image one frame ahead, and supplies the accumulated value of the absolute difference values to the adder 132f. The adder 132f is configured to increase the cumulative value of the difference absolute value according to the accumulated value of the absolute difference value supplied from the accumulator 132a and the accumulated value of the absolute difference value of the image supplied from the frame memory 132e and advanced by one frame. Calculate. The threshold value determining unit 132g determines whether an increased amount of the accumulated absolute value difference difference supplied from the adding unit 132f is larger than a predetermined threshold value. The reset determination unit 132h generates a reset signal according to the determination result supplied from the threshold determination unit 132d and the determination result supplied from the threshold determination unit 132g, and sends the signal to the circular motion vector estimator 111. Supply.

Next, image processing executed by the image processing apparatus 1 shown in FIG. 17 will be described. Here, the image processing is basically the same as that described with reference to the flowchart shown in FIG. Therefore, the description described with reference to FIG. 2 is omitted. In addition, in terms of the motion vector estimation processing and the error processing shown by the flowchart shown in FIG. 2, the image processing executed by the image processing apparatus 1 shown in FIG. 17 is performed by the image processing apparatus 1 shown in FIG. 1. It is different from the image processing executed by. Therefore, motion vector estimation processing and error processing executed by the image processing apparatus 1 shown in FIG. 17 are described next.

First, the error processing executed by the image processing apparatus 1 shown in FIG. 17 will be described with reference to the flowchart of FIG. 18. Here, the processes executed in steps S171-S176 in the flowchart of FIG. 18 are substantially the same as the processes executed in steps S71-S76 in the flowchart of FIG. 8. Therefore, description thereof is omitted.

That is, in step S173, S175, and S176, when the average error level MEL of the input image is obtained, in step S177, the average error level MEL is larger than the predetermined threshold a.

In step S177, for example, if it is determined that the average error level MEL is larger than the predetermined threshold a, in step S178, the threshold value determining unit 132g accumulates the difference absolute value sum supplied from the adding unit 132f. Determine if the increased amount of (dsm cumulative) is greater than threshold b.

In step S178, for example, when it is determined that the increased amount of the difference absolute value sum (dsm cumulative value) is greater than the threshold value b, in step S179, the reset determination unit 132h generates a reset signal and generates the signal. Is supplied to the cyclic type motion vector estimating unit 131 (outputted).

On the other hand, if it is determined in step S177 that the average error level MEL is smaller than the predetermined threshold a, or in step S178, that the increased amount of the cumulative absolute difference value is smaller than the threshold b, the process proceeds to step S180. Return to).

In step S180, the combining unit 132c synthesizes all the pixels in the input image and all the images in the MC image to generate a short circuit image.

In step S181, the combining unit 132c outputs the synthesized image to the display unit 2 as an output image.

Next, with reference to the flowchart of FIG. 19, the motion vector estimation process performed by the image processing apparatus 1 shown in FIG. 17 will be described next. Here, the processes executed in steps 203-S212 in the flowchart of FIG. 19 are substantially the same as the processes executed in steps S101-110 of the flowchart of FIG. 13. Therefore, description thereof is omitted.

That is, in step S201, the motion vector reset unit 131d determines whether the reset signal is supplied.

If it is determined in step S201 that the reset signal has been supplied, in step S202, the motion vector reset unit 131d repeatedly processes the processed motion vector v (i, j) to be a stop vector having a movement amount of zero. Reset.

On the other hand, if it is determined in step S201 that the reset signal is not supplied, step S202 is omitted and the process returns to step S203.

In the above process, even when the motion vector whose motion amount is gradually different from the exact motion vector is processed by resetting one motion vector in the discontinuous image generated when a scene change occurs, the motion vector gradually differs from the exact motion vector. This cyclical processing of the motion vector is prevented, and the time required to obtain an accurate motion vector can be shortened. Therefore, the distortion in which the MC image can be suppressed can be suppressed within a shorter time.

For example, as shown in Fig. 20, for the images in which the scene change occurs in the eleventh frame, in the case of the existing processing indicated by the dotted line, any predetermined time until the correct motion vector is obtained after the scene change is obtained. As necessary, it is necessary for the attenuation of the dSm cumulative addition result. On the other hand, in the case of the processing according to the embodiment indicated by the broken line, it is necessary to display only a few frames until the correct motion vector is obtained after the scene change. Therefore, the dSm cumulative addition result decreases rapidly. Here, in FIG. 20, the frame number was taken on the horizontal axis, and the dSm cumulative addition result was shown on the vertical axis. In addition, FIG. 20 shows an example in which the dSm cumulative addition result is obtained when the value used to perform normalization is constant.

Next, with reference to the block diagram shown in FIG. 21, another embodiment of the image processing apparatus which is designed to reduce a predetermined time required to obtain an accurate motion vector, and performs frame cyclic block matching to obtain a motion vector A representative example according to the example is described. Here, in the image processing apparatus shown in FIG. 21, elements that have substantially the same functions as those provided in the image processing apparatus 1 shown in FIG. 17 are denoted with substantially the same reference numerals. Description thereof is omitted as necessary.

That is, the image processing apparatus 1 shown in FIG. 21 differs from the image processing apparatus 1 shown in FIG. 17 in that the error processing unit 151 is disposed instead of the error processing unit 132.

The error processor 151 includes an accumulator 151a, an MEL calculator 151b, a comparator 151c, an adder 151d and a combiner 151e, a threshold value determiner 151f, and a frame memory 151g. And an adder 151h, a threshold value determiner 151i, and a reset determiner 151j. Among these, the accumulator 151a, the MEL calculator 151b, the comparator 151c, the adder 151d, and the combiner 151e are the accumulator 112a of the image processing apparatus 1 of FIG. The processing similar to that executed by the MEL calculating section 112b, the comparing section 112c, the adding section 112d, and the combining section 112e is executed. That is, the error processing unit 151 synthesizes, for each of the pixels, the MC image and the input image according to the difference absolute value corresponding to each of the motion vectors supplied from the circular type motion vector estimating unit 131. The synthesized image is output to the display unit 2 for display.

In addition, the threshold determination unit 151f, the frame memory 151g, the addition unit 151h, and the threshold determination unit 151i and the reset determination unit 151j include the threshold determination unit 132d, the frame memory 132e, and the addition unit. Processing substantially the same as that executed by the unit 132f, the threshold value determining unit 132g, and the reset determining unit 132h. That is, the error processing unit 151 generates a reset signal indicating the resetting of the motion vector according to the average error level and the absolute difference value supplied from the circular motion vector estimating unit 131, and converts the reset signal into the circular motion. Supply to the vector estimating unit 131.

Next, image processing executed by the image processing apparatus 1 shown in FIG. 21 will be described. Here, the image processing is basically the same as that described with reference to the flowchart shown in FIG. Therefore, the description described with reference to FIG. 2 is omitted. In addition, in terms of the error processing shown in the flowchart of FIG. 2, the image processing executed by the image processing apparatus 1 shown in FIG. 21 is different from the image processing executed by the image processing apparatus shown in FIG. Therefore, image processing executed by the image processing apparatus shown in FIG. 21 is described with reference to the flowchart of FIG. Here, the processing of steps S222-S226 and S230 to S235 in the flowchart of FIG. 22 is the same as the processing of steps S121-S126 and S127-S132 described with reference to the flowchart of FIG. 15. Is omitted. In addition, steps S227-S229 in the flowchart of FIG. 22 are substantially the same as the processing of steps S177-S179 in the flowchart of FIG. 18. Therefore, the description is omitted.

Therefore, even if a motion vector having a large difference between the motion vector and the correct motion vector is repeatedly processed in a discontinuous image generated when a scene change occurs, it is necessary to reset the motion vector to obtain an accurate motion vector. It can save time.

When a motion vector having a low reliability is generated in a part of an area in the image, if the part of the area is about half as large as the entire image, the cumulative addition result of the entire image may not be a large value. do. Therefore, even in this case, a structure capable of detecting the occurrence of the motion vector having low reliability is described.

FIG. 23 shows an exemplary view according to another embodiment of an image processing apparatus capable of detecting the generation of a motion vector with low reliability in a portion of an area of the screen. Here, in the image processing apparatus shown in FIG. 23, elements having substantially the same functions as those provided in the image processing apparatus 1 shown in FIG. 1 are denoted with substantially the same reference numerals and have substantially the same reference numerals. Is indicated. Description thereof is omitted as necessary.

That is, the image processing apparatus 1 shown in FIG. 23 differs from the image processing apparatus 1 shown in FIG. 1 in that the error processing unit 211 is disposed in place of the error processing unit 14.

The error processor 211 includes an area divider 211a, an accumulator 211b, a multiplier 211c, a MEL calculator 211d, and a combiner 211e. Among them, the MEL calculating section 211d and the combining section 211e execute substantially the same processing as those executed by the MEL calculating section 14b and the combining section 14c of the error processing section 14. The area dividing unit 211a divides the input image evenly into n areas. For each of the n regions of the input image, the accumulation unit 211b accumulates and adds the absolute difference value to the multiplier 211c. The multiplier 211c multiplies the largest n times among the cumulative addition results supplied from the accumulator 211b, and each of the cumulative addition results is obtained by adding the difference absolute value for the corresponding region by a cumulative method. . The n times multiplied result is supplied to the MEL calculation unit 211d. That is, the error processing unit 211 synthesizes the MC image and the input image on the basis of the difference absolute value respectively corresponding to one of the nro regions where the input image is divided, and outputs the synthesized image to the display unit 2. To display.

Next, image processing executed by the image processing apparatus 1 shown in FIG. 23 will be described. Here, the image processing is basically the same as that described with reference to the flowchart shown in FIG. Therefore, the description described with reference to FIG. 2 is omitted. In addition, in terms of the error processing shown in the flowchart of FIG. 2, the image processing executed by the image processing apparatus 1 shown in FIG. 23 is different from the image processing executed by the image processing apparatus shown in FIG. Therefore, the motion vector estimation process and the error process executed by the image processing apparatus shown in FIG. 23 are described with reference to the flowchart of FIG. Here, the processing of steps S274-S280 in the flowchart of FIG. 24 is the same as the processing of steps S72-S78 described with reference to the flowchart of FIG. 8, and therefore description thereof will be omitted.

That is, in step S271, the area dividing unit 211a divides the input image evenly into n areas. For example, the area divider 211a groups the absolute absolute differences corresponding to the respective pixels included in one of the n areas where the input image is divided, and supplies them to the accumulator 211b. do. The difference absolute value corresponds to one of the pixels of the input image, and is supplied from the motion vector estimating unit 12.

In step S272, the accumulator 211b accumulates and adds the difference absolute value to each of the n areas where the input image is divided, and supplies it to the multiplier 211c.

In step S273, the multiplier 211c multiplies the largest result among the cumulative addition results supplied from the accumulation unit 211b by n times, and each of the cumulative addition results adds the absolute difference value for the corresponding area. It is obtained by cumulative addition. The multiplier 211c supplies an n-fold multiplication of the cumulative addition result as the dsm cumulative addition result to the MEL calculation unit 211d, and the process proceeds to step S274.

In the above processing, even when a motion vector having a low reliability occurs in a part of an area in the image, generation of a motion vector having a low reliability can be detected. Therefore, even in a screen in which a motion vector having low reliability is intensively generated in one part of an area, distortion in the MC image can be suppressed.

In the above, an example has been described in which a motion vector having low reliability is detected in a part of an area. However, in particular, in the region near the edge of the image, since the corresponding pixels are not often present in the previous frame and the next frame, it is not possible to obtain an accurate motion vector, so that the differential absolute matching is wrongly obtained. Therefore, the structure in which the MC image is generated while removing the effect occurring in the area near the edge of the image is described.

25 shows an exemplary view according to one embodiment of an image processing apparatus designed to remove an effect occurring in an area near an edge of an image and to generate an MC image. Here, in the image processing apparatus shown in FIG. 25, elements that have substantially the same functions as those provided in the image processing apparatus 1 shown in FIG. Is indicated. Description thereof is omitted as necessary.

That is, the image processing apparatus 1 shown in FIG. 25 is different from the image processing apparatus 1 shown in FIG. 1 in that the error processing unit 311 is disposed in place of the error processing unit 14.

The error processing section 311 includes a region exclusion section 311a, an accumulation section 311b, a MEL calculation section 311c, and a combining section 311d. Among them, the MEL calculating section 311c and the combining section 311d execute substantially the same processing as those executed by the MEL calculating section 14b and the combining section 14c of the error processing section 14. The area exclusion unit 311a sets an area other than the area near the edge of the input image as a target area for input processing. The accumulator 311b accumulates and adds the absolute difference value corresponding to the pixels included in the target area to supply the MEL calculator 311c. In other words, the error processing unit 311 synthesizes the MC image and the input image according to the difference absolute value for an area that is not an area near the edge of the input image. Then, the synthesized image is output to the display unit 2 and displayed.

Next, image processing executed by the image processing apparatus 1 shown in FIG. 25 will be described. Here, the image processing is basically the same as that described with reference to the flowchart shown in FIG. Therefore, the description described with reference to FIG. 2 is omitted. In addition, in terms of the error processing shown in the flowchart of FIG. 2, the image processing executed by the image processing apparatus 1 shown in FIG. 25 is performed by the image processing apparatus 1 shown in FIG. 1. Is different from Therefore, error processing executed by the image processing apparatus shown in FIG. 25 is described with reference to the flowchart of FIG. Here, the process of step S373-S379 in the flowchart of FIG. 26 is the same as the process of step S72-S78 described with reference to the flowchart of FIG. 8, and the description is abbreviate | omitted.

That is, in step S371, the area exclusion unit 311a excludes the area near the edge of the input image. Then, another area is set as the target area for error processing. For example, as shown in FIG. 27, the region exclusion section 311a excludes the region near the edge, and the region excludes the number of lines excluded from the top, the number of lines excluded from the bottom, and the leftmost portion. It is defined by predetermined variables such as the number of pixels that are pixelated and the number of pixels that are excluded from the rightmost.

In step S372, for example, the accumulation unit 311b accumulates and adds the absolute difference value corresponding to the pixels included in the target area shown in FIG. 27. The accumulation unit 311b supplies the cumulative addition result as the dsm cumulative addition result to the MEL calculation unit 311c, and the processing proceeds to step S373.

In the above process, even when it is impossible to obtain an accurate sum dsm of the exact motion vector and the absolute difference value, the effect occurring in the area near the edge of the image is eliminated, so that distortion in the MC image can be suppressed.

Here, since the exact sum dsm of the difference absolute values is obtained based on the luminance difference between the frames, it becomes large in the image with high contrast and small in the image with low contrast. Therefore, in error processing, error detection becomes too sensitive for an image with high contrast, so that an error tends to occur for an image with low contrast at the time of error detection. Therefore, an embodiment in which an excessively high sensitivity is prevented at the time of error detection for a high contrast image, and the occurrence of the error at the time of error detection is reduced for an image with low contrast will be described.

Fig. 28 shows a representative structure according to another embodiment of the image processing apparatus which prevents excessively high sensitivity upon error detection for a high contrast image and reduces error occurrence upon error detection for a low contrast image. One drawing. Here, in the image processing apparatus 1 shown in FIG. 28, elements having substantially the same functions as those provided in the image processing apparatus 1 shown in FIG. Are denoted by the same reference numerals. Description thereof is omitted as necessary.

That is, the image processing apparatus 1 shown in FIG. 28 differs from the image processing apparatus 1 shown in FIG. 17 in that the error processing unit 411 is disposed instead of the error processing unit 14.

The error processing unit 411 includes an accumulator 411a, a MEL calculator 411b, a frame delay unit (FD) 411c, 411d, a change amount calculator 411e, a change amount calculator 411f, and a synthesizer 411g. Equipped) Among these, the accumulating unit 411a, the MEL calculating unit 411b, and the combining unit 411g are respectively formed by the accumulating unit 14a, the MEL calculating unit 14b, and the combining unit 14c of the error processing unit 14. It does the same processing that is actually done. The FD unit 411c delays the dsm cumulative addition result supplied from the accumulation unit 411a by one frame, and supplies the delayed dsm cumulative addition result to the change amount calculation unit 411e and the FD unit 411d. The FD unit 411d further delays the delayed dsm cumulative addition result supplied from the FD unit 411c by one frame, and supplies the delayed dsm cumulative addition result to the change amount calculation unit 411e. The change amount calculating section 411e is supplied from the accumulating section 411a to obtain the dsm cumulative addition result obtained at the present time, and the delayed dsm cumulative adding (obtained from the time one frame before the present time) supplied from the FD section 411c. Based on the result and the additional delayed dsm cumulative addition result supplied from the FD unit 411d (obtained at the time two frames before the current time), the amount of change in the dsm cumulative addition result with respect to time is calculated, and the change amount calculation unit ( 411f) is supplied with the variation over time. The change amount calculation unit 411f adjusts the average error level MEL supplied from the MEL calculation unit 411b based on the change amount of the dsm cumulative addition result supplied from the change amount calculation unit 411e. Then, the adjusted average error level MEL is supplied to the combining unit 411g. That is, the error processing unit 411 synthesizes the MC image and the input image based on the amount of change in the dsm cumulative addition result with respect to time. Then, the synthesized image is output to the display unit 2 and displayed.

Next, image processing executed by the image processing apparatus 1 shown in FIG. 28 will be described. Here, the image processing is basically the same as that described with reference to the flowchart shown in FIG. Therefore, the description described with reference to FIG. 2 is omitted. In addition, in terms of the error processing shown in the flowchart of FIG. 2, the image processing executed by the image processing apparatus 1 shown in FIG. 28 is performed by the image processing apparatus 1 shown in FIG. 1. Is different from Therefore, error processing executed by the image processing apparatus shown in FIG. 28 is described with reference to the flowchart of FIG. Here, the processing of steps S471-S476, S480, and S481 in the flowchart of FIG. 29 is the same as the processing of steps S71-S76, S77, and S78 described with reference to the flowchart of FIG. Is omitted.

Here, in steps S473, S475, and S476, the MEL calculator 411b supplies the variation amount calculator 411f with the already set average error level MEL.

In step S477, the change amount calculating unit 411f is supplied from the accumulating unit 411a and is supplied from the FD unit 411c and the dsm cumulative addition result obtained at the current time (at a time before one frame of the current time). Calculate the amount of change in the dsm cumulative addition result over time based on the delayed dsm cumulative addition result obtained and the additional delayed dsm cumulative addition result (obtained from the time two frames before the current time) supplied from the FD unit 411d. Thus, the amount of change with respect to time is supplied to the amount of change calculation unit 411f.

In step S478, the change amount calculation unit 411f determines whether the change amount with respect to time is smaller than a predetermined threshold.

If it is determined in step S478 that the amount of change with respect to time is smaller than the predetermined threshold value, then in step S479, the change amount calculation unit 411f sets the average error level MEL supplied from the MEL calculation unit 411b to a minimum value. Then, it is supplied to the synthesis section 411g. Here, if the average error level MEL supplied from the MEL calculator 411b (at step S473) has already been set to the minimum value, the change amount calculator 411f supplies the average error level MEL to the synthesis unit 411g. .

On the other hand, if it is determined in step S478 that the amount of change with respect to time is not smaller than the predetermined threshold, the procedure skips step S479 and the process proceeds to step S480.

In the above process, the average error level MEL can be set according to the amount of change in the dsm cumulative addition result over time.

In general, the higher the reliability of the motion vector, the smaller the amount of change in the dsm cumulative addition result over time. Therefore, if the amount of change in the dsm cumulative addition result over time is smaller than the predetermined threshold, an accurate motion vector can be obtained, and it can be determined that the average error level MEL can be set to the minimum value. That is, an error can be detected irrespective of the change in the absolute difference sum dsm caused by the degree of high or low contrast.

Therefore, for images with high contrast, excessively high sensitivity is prevented at the time of error detection, and occurrence of errors at error detection is reduced for images with low contrast. Distortion in the MC image can be suppressed.

In the above, the image processing apparatus for detecting an error based on the amount of change in the cumulative addition result with respect to time has been described, but the image processing apparatus can be designed to detect the error based on the movement ratio of the input image.

30 shows an exemplary view according to one embodiment of an image processing apparatus designed to detect an error based on a movement ratio of an input image. Here, in the image processing apparatus shown in FIG. 30, elements having substantially the same functions as those provided in the image processing apparatus 1 shown in FIG. 1 are denoted with substantially the same reference numerals and have substantially the same reference numerals. Is indicated. Description thereof is omitted as necessary.

That is, since the error processing unit 512 and the motion vector estimating unit 511 are disposed in place of the error processing unit 14 and the motion vector estimating unit 12, the image processing apparatus 1 shown in FIG. It is different from the image processing apparatus 1 shown in FIG.

The motion vector estimating unit 511 includes a difference absolute value calculating unit 511a, a comparing unit 511b, a motion vector calculating unit 511c, and a fixed block sum calculating unit 511d of the absolute difference value. Among them, the difference absolute value calculation unit 511a, the comparison unit 511b, and the motion vector calculation unit 511c include the difference absolute value calculation unit 12a, the comparison unit 12b, and the motion vector of the motion vector estimation unit 12. The processing which is substantially the same as that executed by the calculation unit 12c is performed. The fixed block sum calculating unit 511d of the difference absolute value selects the pixels of the block located in the immediately preceding image at the same position as the block of interest and the pixels of the block of interest corresponding to the target image (processing target pixel) in the input image. Reads and calculates the absolute difference value between the pixels at the corresponding pixel positions (hereinafter referred to as the fixed block sum of the absolute difference values). The pixels in the block have a structure substantially identical to that of the pixels in the block of interest. .

The error processing unit 512 includes a movement ratio calculating unit 512a, an accumulating unit 512b, an MEL calculating unit 512c, and a combining unit 512d. Among them, the MEL calculating unit 512c and the combining unit 512d execute substantially the same processes as those executed by the MEL calculating unit 14b and the combining unit 14c of the error processing unit 14. For each of the pixels of the input image, the movement ratio calculator 512a calculates the difference absolute value supplied from the motion vector estimator 511 as the fixed block sum of the corresponding difference absolute values to indicate the degree of movement in the input image. As a variable, a calculation result, which is a movement rate, is supplied to the accumulation unit 512b. For each of the pixels of the input image, the accumulation unit 512b accumulatively adds the movement degree ratio result to the difference absolute value and supplies it to the MEL calculation unit 512c. That is, the error processing unit 512 synthesizes the MC image and the input image for each of the pixels of the input image according to the ratio of the fixed block sum of the absolute difference value and the absolute difference value. Then, the synthesized image is output to the display unit 2 and displayed.

Next, image processing executed by the image processing apparatus 1 shown in FIG. 30 will be described. Here, the image processing is basically the same as that described with reference to the flowchart shown in FIG. Therefore, the description described with reference to FIG. 2 is omitted. In addition, in terms of the motion vector estimation processing and error processing shown in the flowchart of FIG. 2, the image processing executed by the image processing device 1 shown in FIG. 30 is transferred to the image processing device 1 shown in FIG. 1. It is different from the image processing performed by. Therefore, the motion vector estimation process performed by the image processing apparatus shown in FIG. 30 is described with reference to the flowchart of FIG. Next, motion vector estimation processing and error processing executed by the image processing apparatus 1 of FIG. 30 will be described.

First, with reference to the flowchart of FIG. 31, the motion vector estimation process performed by the image processing apparatus 1 shown in FIG. 30 will be described. Here, the processes executed in steps S531-S539 in the flowchart of FIG. 31 are substantially the same as the processes executed in steps S31-S39 in the flowchart of FIG. 3. Therefore, description thereof is omitted.

That is, in step S540, the fixed block sum calculating unit 511d corresponds to the pixels of the block located in the immediately preceding image at the substantially same position as the block of interest and the target image (processing target pixel) in the input image. The pixels of the block of interest are read, and the absolute difference value between the pixels at the corresponding pixel positions is calculated. The pixels in the block are substantially the same in structure as the pixels in the block of interest.

For example, as shown by the oblique line in FIG. 32, the reference pixel p (i, j) of the immediately preceding image F2 having a substantially identical structure corresponding to the pixel of interest p (i, j) in the input image F1 is disposed. It is. The pixel of interest p (i, j) having substantially the same structure is disposed at a position substantially the same as the position of the pixel of interest p (i, j) in the input image F1.

Here, in FIG. 32, the image F1 of the upper part of FIG. 32 represents an input image, and the image F2 represents the immediately previous image. Image F1 represents the MC image to be generated. Each of the cells represents a pixel, and (x, y) represents the position coordinates of the pixel of the previous image.

The fixed block sum calculation unit 511d of the difference absolute value has a substantially identical structure corresponding to the pixel values of the pixels constituting the block B of interest corresponding to the pixel of interest p (i, j) and corresponding to a pixel of the same structure. Pixel values of the pixels constituting the block B 'are extracted. That is, in the case shown in Fig. 32, the block of interest B corresponding to the pixel of interest p (i, j) is a block composed of a plurality of pixels indicated by a solid line in the input image F1. In addition, a block B 'having a substantially identical structure is a block composed of a plurality of pixels indicated by a solid line in the previous image F2. In FIG. 32, as shown in FIG. 4, the block of interest B ′ having the same structure as that of the block of interest B is a block composed of 15 pixels.

In Fig. 32, the movement vector v ₀ (i, j) whose starting point is the pixel p (i, j) of interest in the input image and the end point is the pixel p (i, j) of the same structure is substantially 0. Has That is, it becomes a stop vector.

The fixed block sum calculator 511d of the difference absolute value is a fixed block of absolute absolute value obtained between the pixel values of the pixels constituting the block B of interest and the pixel values of the pixels constituting the block B 'of the same structure. The sum sds (i, j) is calculated, and the fixed block sum sds (i, j) of the difference absolute value is supplied to the error processing unit 512.

In step S541, the difference absolute value calculation unit 511a determines whether there is an unprocessed image in the input image. If there is an unprocessed image therein, the process returns to step S531. That is, steps S531 to S541 are repeatedly performed until the fixed block sum of the motion vector, the difference absolute value and the difference absolute value is obtained for all the pixels of the input image. In step S541, if the difference absolute value calculation unit 511a determines that there is no unprocessed image in the input image, that is, if the motion vector is obtained for all the pixels in the input image, the process is completed.

In the above processing, the minimum difference absolute value dsm and the fixed block sum sds (i, j) of the difference absolute value are obtained for all the pixels in the input image, and are supplied to the error processing unit 512.

Next, referring to FIG. 33, error processing executed by the image processing apparatus 1 shown in FIG. 30 will be described. Here, the processes executed in steps S572-S579 in the flowchart in FIG. 33 are substantially the same as the processes executed in steps S71-S79 in the flowchart in FIG. Therefore, description thereof is omitted.

In step S571, the movement ratio calculator 512a fixes the fixed absolute sum sds (i, j) of the difference absolute value dsm and the difference absolute value supplied from the motion vector estimation unit 511 to all the pixels in the input image. The movement ratio is calculated as a variable representing the degree of movement in the input image using. For all the pixels in the input image, the movement ratio calculator 512a supplies the movement ratio for the pixel to the accumulation unit 512b with the difference absolute value dsm '.

For example, the movement ratio calculator 512a calculates the movement ratio by performing the calculation expressed by the following equation (4). Here, in Formula (4), the constant (alpha) is a value adjusted within the range of 0-1.

dsm '= dsm (i, j) / {α × sds (i, j)} .................. (4)

In addition, for example, the following equation (5) is used to determine the moving ratio as a variable representing the moving ratio of the input image. Here, in the formula (5), the constant β is a value adjusted within the range of 0 to 1.

dsm '= dsm (i, j) -β × sds (i, j) ........ (5)

After step S571, in step S572, the accumulation unit 512b cumulatively adds the difference absolute value dsm 'which is a movement ratio with respect to one of the pixels in the input image. And the steps are executed.

In the above processing, the average error level MEL can be set to a fixed block sum of the difference absolute values according to the ratio of the difference absolute values.

In general, when the reliability of the motion vector becomes high, the difference absolute value dsm becomes smaller than the fixed block sum sds of the difference absolute value. In other words, the higher the reliability of the motion vector, the smaller the degree of motion. For example, as shown in Fig. 34, in an image capable of obtaining an accurate motion vector, the difference absolute value indicated by the (filled) diamond symbol line is compared to the sds cumulative addition result represented by the (colored) square symbol line. dsm is very small. In addition, as shown in FIG. 35, in an image where it is impossible to obtain an accurate motion vector, the difference absolute dsm represented by the (filled) diamond symbol line is added to the sds cumulative addition result represented by the (colored) square symbol line. In this regard, it is not very small as shown in FIG. 34. In FIG. 34, the horizontal axis represents the frame number, and the vertical axis represents the dsm cumulative addition result and the sds cumulative addition result.

Therefore, if the moving ratio that is the difference absolute value dsm 'represented by equation (4) is small, an accurate moving vector is obtained, and the average error level MEL can be set to the minimum. In other words, an error can be detected regardless of the change in the absolute difference sum dsm caused by the contrast in the range between the high level and the low level.

Therefore, excessively high sensitivity is prevented at the time of error detection for an image with high contrast, and error occurrence at the time of error detection is reduced for an image with low contrast, so that distortion in the MC image can be suppressed.

In the above, the image processing apparatus which can detect an error according to the movement degree detected in the whole input image was demonstrated. However, if a predetermined or more ratio of the image is a fixed region, even if a motion vector having a low reliability is obtained in another region in the image, the absolute absolute difference dsm of the whole image does not become a large value. Therefore, even in this case, an embodiment in which a motion vector having low reliability is detected is described.

36 shows an exemplary view according to another embodiment of an image processing apparatus designed to detect the presence of a motion vector with low reliability. Here, in the image processing apparatus 1 shown in FIG. 36, elements having substantially the same functions as those provided in the image processing apparatus 1 shown in FIG. The same reference numerals are used. Description thereof is omitted as necessary.

That is, since the error processing unit 611 and the motion vector estimating unit 511 are disposed in place of the error processing unit 14 and the motion vector estimating unit 12, the image processing apparatus 1 shown in FIG. It is different from the image processing apparatus 1 shown in FIG. In addition, the motion vector estimator 511 shown in FIG. 36 is functionally substantially the same as the motion vector estimator 511 shown in FIG. Therefore, description thereof is omitted.

The error processing unit 611 includes a moving area determining unit 611a, a counter 611b, a moving area accumulating unit 611c, an average calculating unit 611d, a MEL calculating unit 611e, and a combining unit 611f. . Among them, the MEL calculator 611e and the combiner 611f perform the processes substantially the same as those executed by the MEL calculator 14b and the combiner 14c in FIG. The moving area determination unit 611a determines a region in which the movement (movement region) exists in the input image, according to the fixed block sum of the absolute difference value obtained for one of the predetermined regions. The fixed block sum sds of the absolute difference value is supplied from the motion vector estimating unit 511. The counter 611b counts the number of areas (unit areas) determined as the moving areas by the moving area determining unit 611a. Then, the number of regions is supplied to the average calculating unit 611d. The moving area accumulation unit 611c accumulates and adds the absolute difference value to the areas determined as the moving areas in the input image, and supplies the dsm cumulative addition result to the average calculating unit 611d. The average calculating unit 611d calculates a dsm average that is an absolute absolute difference for each unit area based on the number of areas supplied from the counter 611b and the dsm cumulative addition result supplied from the moving area accumulation unit 611c. Calculate. Then, the dsm average is supplied to the MEL calculator 611e. In other words, the error processing unit 611 synthesizes the MC image and the input image according to the difference absolute value difference of the moving regions in the input image, and outputs the synthesized image to the display unit 2 for display.

Here, the error processing unit 611 calculates an average of the difference absolute values of the moving areas in the input image, and detects an error regardless of the size of the area (fixed area) that is not the moving area. For example, if half of the input image is a fixed area, and the other area is a moving area where the movement of the subject in the image can be followed, a correct moving vector can be obtained, as shown in FIG. The dsm cumulative addition result represented by the square symbol line is larger than the dsm cumulative addition result of the entire image represented by the (filled) diamond symbol line. Here, in FIG. 37, the horizontal axis represents a frame number, and the vertical axis represents a dsm average.

Next, image processing by the image processing apparatus 1 of FIG. 36 will be described. Here, the basic image processing is the same as the processing described with reference to the flowchart of FIG. 2), and therefore the description of the flowchart of FIG. 2 is omitted. The image processing of the image processing apparatus 1 of FIG. 36 differs from the image processing by the image processing apparatus 1 of FIG. 1 is error processing among the processes in the flowchart of FIG. 2. Therefore, error processing by the image processing apparatus 1 in FIG. 36 will be described with reference to the flowchart shown in FIG. 38. Here, since the process of step S675-S681 in the flowchart of FIG. 38 is the same as the process of step S72-S78 demonstrated with reference to the flowchart of FIG. 8, the description is abbreviate | omitted. Here, the dsm average processed in the steps S676 and S678 S679 in the flowchart of FIG. 38 is substantially the same as the dsm average processed in the steps S72, S75 and S76 in the flowchart of FIG. Will be processed. Here, in the process of the flowchart of FIG. 38, the unit area is treated as one pixel.

That is, in step S671, the moving area determining unit 611a moves the pixels in which the movement is detected among the pixels of the input image according to the fixed block sum sds of the absolute absolute difference obtained for one pixel of the pixels in the input image. Determined by In particular, since the fixed block sum sds of the difference absolute value becomes a relatively large value in the moving area, if the fixed block sum sds of the difference absolute value of one pixel in the input image is larger than a predetermined threshold value, the moving area determining unit 611a determines the value. It is determined that the pixel is a moving area.

In step S672, the counter 611b counts the number of pixels constituting the area determined by the moving area determining unit 611a as the moving area, and supplies the number of the pixels to the average calculating unit 611d. .

In step S673, the moving area accumulating part 611c accumulates and adds the difference absolute value dsm to the pixels determined as the moving area by the moving area determining part 611a, and averages the dsm cumulative adding result. It supplies to 611d.

In step S674, the average calculating unit 611d uses one pixel in the moving area based on the number of pixels supplied from the counter 611b and the dsm cumulative addition result supplied from the moving area accumulating unit 611c. Compute the dsm mean that is the difference absolute value of for. Then, the dsm average is supplied to the MEL calculator 611e. The process then ends.

In the above processing, even if a predetermined ratio or more of the image becomes a fixed area, the presence of a motion vector having low reliability can be detected. Therefore, even in a scene in which a predetermined ratio or more of the image becomes a fixed area, distortion in the MC image can be suppressed.

Here, in the process of the flowchart of FIG. 38, one pixel was processed into a unit area. However, the unit area is not limited thereto, and for example, a plurality of pixels, that is, blocks of four pixels may be treated as the unit area.

As described above, structures for suppressing distortion in the MC image have been described. These structures can be combined.

39 is a diagram showing representative examples of another embodiment of the image processing apparatus in which the above structures are combined. Here, in the image processing apparatus 1 shown in FIG. 39, elements having substantially the same functions as those provided in the image processing apparatus 1 shown in FIG. It is indicated by a number. Description thereof is omitted as necessary.

That is, the image processing of the image processing apparatus 1 of FIG. 39 is different from the image processing by the image processing apparatus 1 of FIG. 1, instead of the motion vector estimating unit 12 and the error processing unit 14. The vector estimator 511 and the error processor 711 are provided. In addition, the motion vector estimating unit 511 of FIG. 39 functions substantially the same as the motion vector estimating unit 12 of FIG. 30. Therefore, his explanation is omitted.

The error processing unit 711 includes a movement ratio calculator 711a, an accumulator 711b, an area divider 711c, a multiplier 711d, a MEL calculator 711e, an FD unit 711f, 711g, and a change amount. A calculating section 711h, a change amount determining section 711i, and a combining section 711j are provided. Among these, the movement ratio calculation unit 711a and the accumulation unit 711b perform substantially the same processing as those executed by the movement ratio calculation unit 512a and the accumulation unit 512b in FIG. 30, respectively. The area divider 711c and the multiplier 711d perform substantially the same processing as those respectively executed by the area divider 211a and the multiplier 211c in FIG. MEL calculation part 711e, FD part 711f, 711g, the change amount calculation part 711h, the change amount determination part 711i, and the synthesis part 711j are MEL calculation part 411b and FD part 411c of FIG. 411d), the change amount calculating section 411e, the change amount determining section 411f, and the combining section 411g carry out substantially the same processing. Therefore, his explanation is omitted.

In the above structure, distortion of the MC image can be suppressed.

Here, the image processing executed by the image processing apparatus 1 of FIG. 39 is the same as the processes of the flowchart showing the processes performed by the elements constituting the image processing apparatus of FIG. 39. Therefore, description thereof is omitted.

40 is a diagram showing representative examples of another embodiment of the image processing apparatus in which the above structures are combined. Here, in the image processing apparatus 1 shown in FIG. 40, elements having substantially the same functions as those provided in the image processing apparatus 1 shown in FIG. It is indicated by a number. Description thereof is omitted as necessary.

That is, the image processing of the image processing apparatus 1 of FIG. 40 differs from the image processing by the image processing apparatus 1 of FIG. 1, instead of the motion vector estimating unit 12 and the error processing unit 14. The motion vector estimator 511 and the error processor 811 are provided. In addition, the motion vector estimating unit 511 of FIG. 40 functions substantially the same as the motion vector estimating unit 12 of FIG. 30. Therefore, his explanation is omitted.

The error processing unit 811 includes a moving area determining unit 811a, a counter 811b, a moving area accumulating unit 811c, an average calculating unit 811d, a moving ratio calculating unit 811e, an accumulating unit 811f, and a MEL. A calculation unit 811g, FD units 811h and 811i, a change amount calculation unit 811j, a change amount determination unit 811k, and a combining unit 811m are provided. Among them, the moving area determining unit 811a, the counter 811b, the moving area accumulating unit 811c, and the average calculating unit 811d include the moving area determining unit 611a, the counter 611b, and the moving area of FIG. 36. Processing that is substantially the same as the processing executed by the accumulation unit 611c and the average calculating unit 611d is performed. The movement ratio calculation unit 811e and the accumulation unit 811f perform processing substantially the same as the processing executed by the movement ratio calculation unit 512a and the accumulation unit 512b in FIG. 30. The MEL calculating unit 811g, the FD units 811h and 811i, the change amount calculating unit 811j, the change amount determining unit 811k and the combining unit 811m are the MEL calculating unit 411b and the FD unit (Fig. 28). 411c, 411d, the change amount calculating unit 411e, the change amount determining unit 411f, and the combining unit 411g carry out substantially the same processes as those executed. Therefore, description thereof is omitted.

In the above structure, distortion of the MC image can be suppressed.

Here, the image processing executed by the image processing apparatus 1 of FIG. 40 is the same as the processes of the flowchart showing the processes performed by the elements constituting the image processing apparatus of FIG. 40. Therefore, description thereof is omitted.

The above-described series of information processing can be executed by hardware, but can also be executed by software. When a series of processes are executed by software, a program constituting the software can be installed in a computer installed on dedicated hardware from a recording medium, or can execute various functions. It is installed in a computer.

41 shows an example of the configuration of a general-purpose personal computer. This personal computer has a CPU (Central Processing Unit) 1001 built in. The input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

The input / output interface 1005 includes an input unit 1006 composed of input devices such as a keyboard and a mouse for inputting an operation command by the user, an output unit 1007 for outputting an image of a processing operation screen or a processing result to a display device, and a program. Or a storage unit 1008 composed of a hard disk drive for storing various data, a local area network (LAN) adapter, or the like, and a communication unit 1009 for executing communication processing via a network represented by the Internet, have. In addition, a magnetic disk (including a flexible disk), an optical disk (including a Compact Disc-Read Only Memory (CD-ROM), a digital versatile disc (DVD)), and an optical magnetic disk (including a Mini Disc) Or a drive 1010 is read and written to a removable medium 1011 such as a semiconductor memory.

The CPU 1001 reads from a program stored in the ROM 1002 or a removable medium 1011 such as a magnetic disk, an optical disk, an optical magnetic disk, or a semiconductor memory, and is installed in the storage unit 1008 and stored therein. Various processes are executed in accordance with a program loaded from the unit 1008 into the RAM 1003. The RAM 1003 also stores data necessary for the CPU 1001 to execute various processes.

In this specification, the steps of describing a program recorded on the recording medium include processing performed in time series according to the described order, as well as processing executed in parallel or separately even if the processing is not necessarily performed in time series. .

It will be apparent to those skilled in the art that various modifications, combinations, subcombinations and modifications are possible within the scope of the appended claims or their equivalents.

1 is a block diagram showing a configuration example of an image processing apparatus to which the present invention is applied.

2 is a flowchart for explaining image processing.

FIG. 3 is a flowchart for describing motion vector estimation processing by the image processing apparatus of FIG. 1.

FIG. 4 is a diagram for explaining motion vector estimation processing by the image processing apparatus of FIG. 1.

FIG. 5 is a diagram illustrating a motion vector estimation process by the image processing apparatus of FIG. 1.

6 is a flowchart for explaining the movement compensation process.

7 is a diagram for explaining the movement compensation process.

FIG. 8 is a flowchart for describing error processing by the image processing apparatus of FIG. 1.

FIG. 9 is a diagram for describing error processing by the image processing apparatus of FIG. 1.

FIG. 10 is a diagram for explaining a dSm cumulative addition result in an image in which a scene change occurs or a subject moves rapidly. FIG.

FIG. 11 is a diagram illustrating a dSm cumulative addition result in an image in which there is no scene change and the movement of the subject is late. FIG.

12 is a block diagram showing a configuration example of another image processing apparatus to which the present invention is applied.

FIG. 13 is a flowchart for describing motion vector estimation processing by the image processing apparatus of FIG. 12.

FIG. 14 is a diagram for explaining motion vector estimation processing by the image processing apparatus of FIG. 12.

FIG. 15 is a flowchart for describing error processing by the image processing device of FIG. 12.

FIG. 16 is a diagram for describing error processing by the image processing device of FIG. 12.

17 is a block diagram showing a typical structure of another image processing apparatus to which the present invention is applied.

FIG. 18 is a flowchart for describing error processing by the image processing device of FIG. 17.

19 is a diagram illustrating a motion vector estimation process by the image processing device of FIG. 17.

20 is a diagram illustrating attenuation of the dsm cumulative addition result for an image including a scene change.

21 is a block diagram showing a typical structure of another image processing apparatus to which the present invention is applied.

FIG. 22 is a flowchart for describing error processing by the image processing device of FIG. 21.

Fig. 23 is a block diagram showing a typical structure of another image processing apparatus to which the present invention is applied.

24 is a flowchart for describing error processing by the image processing device of FIG. 23.

25 is a block diagram showing a typical structure of another image processing apparatus to which the present invention is applied.

FIG. 26 is a flowchart for describing error processing by the image processing device of FIG. 25.

FIG. 27 is a diagram for describing error processing by the image processing device of FIG. 25.

Fig. 28 is a block diagram showing a typical structure of another image processing apparatus to which the present invention is applied.

FIG. 29 is a flowchart for describing error processing by the image processing device of FIG. 28.

30 is a block diagram showing a typical structure of another image processing apparatus to which the present invention is applied.

FIG. 31 is a flowchart for describing motion vector estimation processing by the image processing apparatus of FIG. 30.

32 is a diagram illustrating a motion vector estimation process by the image processing device of FIG. 30.

33 is a flowchart for describing error processing by the image processing device of FIG. 30.

Fig. 34 is a diagram showing dsm cumulative addition results and sds accumulation addition results when an accurate motion vector is obtained.

Fig. 35 is a diagram showing the dsm cumulative addition result and the sds cumulative addition result when the exact motion vector is not obtained.

36 is a block diagram showing a typical structure of another image processing apparatus to which the present invention is applied.

Fig. 37 is a diagram showing the dsm average when it is difficult to track the movement of a subject in an image and it is impossible to obtain an accurate movement vector.

FIG. 38 is a flowchart for describing error processing by the image processing device of FIG. 36.

Fig. 39 is a block diagram showing a typical structure of another image processing apparatus to which the present invention is applied.

40 is a block diagram showing a typical structure of another image processing apparatus to which the present invention is applied.

41 is a view for explaining an example of the configuration of a personal computer.

Claims (17)

In the image processing apparatus, Pixel values of the pixels of the block of interest including a plurality of pixels corresponding to the pixels of interest in the first image, and reference blocks corresponding to the reference pixels of the second image whose display timing is different from that of the first image. Differential absolute value calculating means for calculating an absolute difference value between pixel values of a plurality of pixels of substantially the same arrangement as the block of interest; First comparing means for comparing the difference absolute values calculated by the difference absolute value calculating means to obtain a minimum difference absolute value; Motion vector estimating means for estimating a motion vector of the pixel of interest based on the reference pixel which is the minimum of the absolute difference values obtained by the first comparing means and the pixel of interest; Motion compensation image generating means for generating a motion compensation image by using a pixel value of each pixel of the first image as a pixel value of corresponding pixels in the motion compensation image based on the motion vector of each pixel of the first image and; First cumulative adding means for cumulatively adding the difference absolute value calculated when a motion vector corresponding to a pixel of the motion compensation image is obtained for each pixel of the first image; Interpolation image generating means for synthesizing each pixel in the first image and each pixel in the corresponding motion compensation image based on the cumulative addition result obtained by the cumulative addition means, to generate an interpolation image; , And each of the pixels of the first image corresponds to each of the pixels in the movement compensation image. The method of claim 1, Quantization means for quantizing the cumulative addition result using a ratio of the cumulative addition result to a predetermined maximum value set for the cumulative addition result, The interpolation image generating means synthesizes each pixel of the motion compensation image corresponding to each pixel in the first image based on the cumulative result quantized by the quantization means to generate an interpolation image. Image processing apparatus. The method of claim 2, The interpolation image generating means generates an interpolation image by using the pixels in the first image as it is for pixels whose accumulation result quantized by the quantization means is smaller than a first predetermined threshold amount, An image processing apparatus for generating an interpolation image, using each pixel in the movement compensation image as it is for pixels whose cumulative result quantized by the quantization means is larger than a second predetermined threshold amount. The method of claim 1, The reference pixel is in the second image, and includes an adjacent range corresponding to the pixel of interest in the first image. The method of claim 1, Subtraction means for subtracting a predetermined value from the cumulative addition result in the second image; The cumulative addition in the second image is compared with a value obtained by subtracting a predetermined value from the cumulative addition result in the second image by the subtraction means and the cumulative addition result in the first image. If the value from which the predetermined value is subtracted from the result is larger than the cumulative addition result in the first image, the value obtained by subtracting the predetermined value from the cumulative addition result in the second image is determined in the first image. Replace with the cumulative addition result of, When a value obtained by subtracting a predetermined value from the cumulative addition result in the second image is not larger than the cumulative addition result in the first image, the cumulative addition result in the first image is output as it is. And a second comparing means. The method of claim 1, The cumulative addition result in the second image is larger than the first threshold amount, and the amount increased as a result of the cumulative addition in the first image from the cumulative addition result in the second image is a predetermined amount. And an instruction means for transmitting a command for resetting the motion vector estimated by the motion vector estimating means to a still vector having a movement amount of zero, when larger than two thresholds. The method of claim 6, And resetting means for resetting the motion vector estimated by the motion vector estimating means to the stop vector having a movement amount of zero in accordance with the command transmitted from the command means. The method of claim 1, Pixel values of the pixels in the block of interest corresponding to the pixel of interest in the first image and including a plurality of pixels, and at the same position as the pixel of interest, disposed in the second image and of the same structure as the first image; In a block of the same structure corresponding to a pixel, fixing of a difference absolute value that yields a stationary-block sum of absolute differences between pixel values of pixels having the same structure as the pixels in the block of interest. Block sum calculation means; Calculating means for calculating a variable for each pixel of interest according to the fixed block sum of the difference absolute values calculated by the fixed block sum calculation means of the difference absolute values and the minimum value of the difference absolute values obtained by the first comparing means; Second cumulative adding means for cumulatively adding the variable calculated by the calculating means with respect to the pixel of interest; The interpolation image generating means synthesizes the pixels in the first image and the motion compensation image according to the cumulative addition result obtained by the second cumulative adding means to generate the compensation image, And each of the pixels in the first image corresponds to each of the pixels of the movement compensation image. The method of claim 1, Pixel values of the pixels in the block of interest corresponding to the pixel of interest in the first image and including a plurality of pixels, and at the same position as the pixel of interest, disposed in the second image and of the same structure as the first image; Fixed block sum calculation means for calculating a difference absolute value of the absolute absolute value in a block of the same structure corresponding to the pixel, the fixed block sum of the absolute difference value of the pixels having the same structure as the pixels in the block of interest; Of the fixed block sums of the absolute difference values calculated by the fixed block sum calculating means of the difference absolute values, the absolute absolute sums are accumulated in the region including the pixel of interest corresponding to the fixed block sums of the difference absolute values larger than a predetermined threshold amount. Further comprising a second cumulative adding means for adding, The interpolation image generating means synthesizes the pixels in the first image and the motion compensation image according to the cumulative addition result obtained by the second cumulative adding means to generate the compensation image, And each of the pixels in the first image corresponds to each of the pixels of the movement compensation image. The method of claim 9, And an average calculating means for calculating an average of the fixed block sums of the absolute difference values according to the cumulative addition result obtained by the second cumulative addition means. The method of claim 1, And a change amount calculating means for calculating a change amount according to time with respect to the cumulative addition result obtained by the first cumulative addition means, The interpolation image generating means synthesizes the pixels in the first image and the moving compensation image according to the amount of change obtained by the change amount calculating means to generate the compensation image, And each of the pixels in the first image corresponds to each of the pixels of the movement compensation image. The method of claim 1, Dividing means for dividing the first image into n regions; a multiplication means for multiplying the maximum value n times of the difference absolute values for the n regions, For each of the n areas divided by the dividing means, each of the difference absolute values obtained by the first cumulative adding means for accumulating and adding the difference absolute value is to each of the n areas. In response, The interpolation image generating means generates the compensation image by combining pixels in the first image and the motion compensation image according to the difference absolute value obtained by the multiplication means, And each of the pixels in the first image corresponds to each of the pixels of the movement compensation image. The method of claim 1, And said first cumulative adding means cumulatively adds the difference absolute values to an area except one area located at the end of the first image. In the image processing method, Pixel values of the pixels of the block of interest including a plurality of pixels corresponding to the pixels of interest in the first image, and reference blocks corresponding to the reference pixels of the second image whose display timing is different from that of the first image. A difference absolute value calculation step of calculating a difference absolute value difference between pixel values of a plurality of pixels of substantially the same arrangement as the block of interest; A comparison step of comparing the difference absolute value calculated by the difference absolute value calculation step to determine a minimum difference absolute value; A motion vector estimating step of estimating a motion vector of the pixel of interest based on the reference pixel which is the minimum of the difference absolute values obtained by the comparison step and the pixel of interest; A motion compensation image generation step of generating a motion compensation image by using a pixel value of each pixel of the first image as a pixel value of corresponding pixels in the motion compensation image based on the motion vector of each pixel of the first image and; A first cumulative adding step of cumulatively adding, to each pixel of the first image, a difference absolute value calculated when a motion vector corresponding to a pixel of the motion compensation image is obtained; An interpolation image generation step of generating an interpolation image by synthesizing each pixel in the first image and each pixel in the corresponding motion compensation image based on the cumulative addition result obtained in the cumulative addition step; , Wherein each of the pixels of the first image corresponds to each of the pixels in the motion compensation image. In a computer readable program for causing a computer to execute the following processing, The processing is Pixel values of the pixels of the block of interest including a plurality of pixels corresponding to the pixels of interest in the first image, and reference blocks corresponding to the reference pixels of the second image whose display timing is different from that of the first image. A difference absolute value calculation step of calculating a difference absolute value difference between pixel values of a plurality of pixels of substantially the same arrangement as the block of interest; A comparison step of comparing the difference absolute value calculated by the difference absolute value calculation step to determine a minimum difference absolute value; A motion vector estimating step of estimating a motion vector of the pixel of interest based on the reference pixel which is the minimum of the difference absolute values obtained by the comparison step and the pixel of interest; A motion compensation image generation step of generating a motion compensation image by using a pixel value of each pixel of the first image as a pixel value of corresponding pixels in the motion compensation image based on the motion vector of each pixel of the first image and; A first cumulative adding step of cumulatively adding, to each pixel of the first image, a difference absolute value calculated when a motion vector corresponding to a pixel of the motion compensation image is obtained; An interpolation image generation step of generating an interpolation image by synthesizing each pixel in the first image and each pixel in the corresponding motion compensation image based on the cumulative addition result obtained in the cumulative addition step; , Wherein each of the pixels of the first image corresponds to each of the pixels in the movement compensation image. A program recording medium having a computer readable program according to claim 15 therein. In the image processing apparatus, Pixel values of the pixels of the block of interest including a plurality of pixels corresponding to the pixels of interest in the first image, and reference blocks corresponding to the reference pixels of the second image whose display timing is different from that of the first image. A differential absolute value calculation unit for calculating an absolute difference value between pixel values of a plurality of pixels having substantially the same arrangement as the block of interest; A comparison unit for comparing the difference absolute values calculated by the difference absolute value calculation unit to obtain a minimum difference absolute value; A motion vector estimator for estimating a motion vector of the pixel of interest based on the reference pixel, which is the minimum of the difference absolute values obtained by the comparison unit, and the pixel of interest; Generating a motion compensation image using a pixel value of each pixel of the first image as a pixel value of corresponding pixels in the motion compensation image based on the movement vector of each pixel of the first image. Wealth; A cumulative adder which accumulatively adds the absolute difference value calculated when a motion vector corresponding to a pixel of the motion compensation image is obtained for each pixel of the first image; An interpolation image generation unit for generating an interpolation image by synthesizing each pixel in the first image with each pixel in the corresponding motion compensation image based on the cumulative addition result obtained by the cumulative addition unit; , And each of the pixels of the first image corresponds to each of the pixels in the movement compensation image.

Images