JP2007060192A - Interpolating frame generator, its method, image display system and program and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an interpolating frame generator which provides a technique allocating an accurate motion vector in an occlusion region, and displays a background image more distinctly. <P>SOLUTION: The whole image motion vector is detected from a plurality of reference frames approaching with time in an image, and the motion vector on an interpolating frame is generated from the reference frames. A background decision evaluation is conducted from a past background reference image, the reference frames, and the whole image motion vector from the reference frames. A new background image is synthesized from the result of the evaluation, the reference frames, and the background reference image, and stored while an interpolating-frame image is generated from the images of the reference frames and the background reference image by using the whole image motion vector, an interpolating-frame motion vector, and the result of the background decision evaluation. Consequently, an interpolating image is generated even in a region in which a correspondence is difficult only by the images of the reference frames. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an interpolation frame generation apparatus, a method thereof, an image display system, a program, and a recording medium. Specifically, at least one interpolation is performed between adjacent frames in order to shorten a display frame interval in reproducing a moving image. The present invention relates to a technique for interpolating a frame.

  In a hold-type image display device that keeps displaying the previous frame until a new image is written, such as a liquid crystal display or an electroluminescence display, the observer's eyes follow the movement of the moving object when displaying a moving image. There is a problem that blurring occurs and an unnatural movement occurs when a moving image with a small number of frames is displayed.

To solve this problem, there is a method of shortening the display frame interval. As a specific method, there is a television system conversion that takes the configuration as shown in FIG. 4, creates an interpolation frame using motion correction, and interpolates the interpolation frame between adjacent frames.
As a television system conversion device using this motion correction, techniques described in Patent Document 1, Patent Document 2, Non-Patent Document 1, Non-Patent Document 2, and the like have been proposed.

  In such a frame interpolation method using motion correction, when generating an interpolation frame by motion correction as shown in FIG. 5, first, the motion vector AB obtained between the first reference frame and the second reference frame is used as the interpolation frame plane. An operation called scale conversion for converting into a motion vector AP is performed.

Then, an interpolation frame is generated by performing motion correction using the motion vector obtained by the scale conversion. The interpolation frame is divided into several interpolation blocks S, and the generation of the interpolation frame is performed in units of these blocks.
Specifically, a motion vector passing through the interpolation target block on the interpolation frame is searched, and the image data of the image block indicated by the motion vector on the first reference frame and the second reference frame is interpolated on the interpolation frame surface. Composite and copy to block position.

  In this method, the motion vector obtained by the scale conversion does not necessarily pass through all the interpolation blocks on the interpolation frame plane. Overlapping areas are created.

Therefore, as a pre-stage of image interpolation, after searching for motion vectors that pass through all the interpolation blocks, a zero vector was assigned to an area that was not assigned a passing motion vector, or multiple motion vectors were assigned Processing such as selecting a motion vector for the region is required.
For example, in selecting a motion vector, a DFD (directed frame difference) or the like is used as an index of the certainty of the motion vector. The DFD is obtained by the following equation (1).

Here, (x, y) is the coordinates of the interpolation block, (dx, dy) is the motion vector before scale conversion, f () is the frame luminance data, n is the frame number, and Sh and Sv are the sizes of the interpolation block. In this constant, Sh is the horizontal size, and Sv is the vertical size. Here, the first reference frame corresponds to fn-1 (x, y), and the second reference frame corresponds to fn (x, y).
In vector selection using DFD, a value that minimizes this value is selected. When this is expressed by an equation, equation (2) is obtained.

Here, v and (dx _v , dy _v ) represent selected motion vectors, and V represents a motion vector group to be selected.
Japanese Laid-Open Patent Publication No. 01-309597 Japanese Patent Laid-Open No. 03-280681 The Television Society Journal, Vol. 45, No. 12, published in 1991, p. 1534-1543 Proceedings at the annual conference of the Institute of Television Engineers of Japan, 20-5, 1989

  However, in the above method, in the case of an image including occlusion, there is a problem in that an image is distorted because a motion vector cannot be accurately obtained in that region. For example, the occlusion area often occurs between the background and the object, and the camera is often fixed, especially in a television studio video, and so on. In many cases, a resolution occurs.

  The occlusion refers to an area in which a background or the like hidden behind an object in the first reference frame as shown in FIG. 6 becomes visible due to movement of the object in the second reference frame. On the contrary, the same is true when the visible background is hidden in the next frame, but the former is particularly problematic here.

  In such a region, there is no correlation between the first reference frame and the second reference frame, and as a result, a motion vector cannot be obtained. Then, a motion vector cannot be set on the interpolation frame, and a zero vector or an abnormal vector is selected. As a result, distortion occurs in an image around the object.

  Further, in the case of an interpolated frame generation method in which the number of frames is converted, the interpolated frame is sandwiched between the original frames, so that the single image distortion is difficult to visually recognize. For this reason, it is particularly necessary to solve the problem of continuous occlusion. For example, continuous occlusion includes telops such as end rolls.

  The present invention has been made in consideration of the above-described circumstances, and provides a method for assigning an accurate motion vector to an occlusion region, and an interpolation frame generation device capable of displaying a background image more clearly, It is an object to provide a method, an image display system, a program, and a recording medium.

In order to solve the above problems, the present invention has the following configuration.
In an interpolated frame generation apparatus that generates a new interpolated frame that does not exist from information of existing frames using an image composed of a plurality of temporally continuous frames, a background image is generated from some existing frames. An occlusion area in two or more reference frames extracted from existing frames, having background image extraction / synthesis means for extracting and synthesizing a background image and background image storage means for storing the generated background image Is to be detected.

  In addition, in an interpolated frame generation apparatus that generates a new interpolated frame that does not exist from information of existing frames using an image composed of a plurality of temporally continuous frames, a background from some existing frames A background image extracting / synthesizing means for extracting an image and synthesizing a background image; and a background image accumulating means for accumulating the generated background image. A new image is extracted from two or more reference frames extracted from existing frames. When generating an interpolated frame, an accurate motion vector is obtained even for a portion in which it is difficult to detect a motion vector from the reference frame due to occlusion or the like.

  Alternatively, in an interpolated frame generation apparatus that generates a new interpolated frame that does not exist from the information of existing frames using an image composed of a plurality of temporally continuous frames, the background from some existing frames A background image extracting / synthesizing means for extracting an image and synthesizing a background image; and a background image accumulating means for accumulating the generated background image. A new image is extracted from two or more reference frames extracted from existing frames. When generating an interpolated frame, a background image that is difficult to interpolate from the reference frame for reasons such as occlusion is interpolated using the background image accumulating means.

  Here, when generating a new interpolation frame from two or more reference frames taken out from existing frames, the background image stored in the background image storage means is used to convert the image of the reference frame as a background image. The image is divided into other moving objects.

  Furthermore, in an interpolated frame generation apparatus that generates an interpolated frame from a plurality of reference frames that are temporally close to each other in the image, the entire screen motion that detects a motion vector of the entire screen (hereinafter referred to as an all-image motion vector) from the reference frame Detecting means; interpolated frame motion vector generating means for generating a motion vector on the interpolated frame from the reference frame; background image accumulating means for accumulatively storing past background images from the reference frame; Background determination means for performing background determination evaluation from the background reference image, the reference frame, and the whole image motion vector possessed by the background image storage means, a result of the background determination evaluation, the reference frame, and a new background image from the background reference image A background image extracting / synthesizing means for synthesizing the whole image motion vector and the interpolation frame It comprises interpolation frame generation means for generating an interpolation frame image from the reference frame image and the background reference image using the motion vector and the result of the background determination evaluation of the background determination means. This makes it possible to generate an interpolated image even in a region where it is difficult.

Here, it is also possible to use a static region determination means as shown in FIG.
The static motion area determination means uses a background reference image and a reference frame image that the background image storage means has, and compares the brightness for each pixel or interpolation block, etc. And the other moving regions can be more accurately distinguished, and the processing of the background determination unit and the background image extraction / combination unit can be more accurately realized using the result.

The background image extraction / combination means generates a new background image each time from the background reference image and the images of the plurality of reference frames. Furthermore, it is also possible to generate a high-definition high-resolution background image using a technique for increasing the resolution of the image.
Alternatively, the background image extracting / synthesizing means stores the background image when a scene change (such as when the difference between the reference frames is large) is detected or when a large difference occurs between the background reference image and the reference frame. You may make it clear the content of a means.

The above problem can also be solved by an image display system that displays the interpolated frame and the reference frame generated by the interpolated frame generating apparatus so as to be temporally continuous.
The present invention also solves the above-described problems by a program for realizing the functions of the interpolation frame generating apparatus or the image display system having the above-described configuration on a computer, or a computer-readable recording medium storing the program. Can do.

According to the present invention, a background reference image is prepared by separating a video into a background image and a moving image of a moving object, and storing the background image in a video stream, and comparing this with a plurality of reference frames. Thus, the occlusion motion vector can be accurately obtained. As a result, the background image can be displayed more clearly.
Further, by making the background reference image a finer image than the actual image, it is possible to bring about an effect in improving the image quality of the still image region.

  DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram showing an overall configuration of an interpolation frame generation device according to an embodiment of the present invention. FIG. 2 is a block diagram showing a functional configuration of the interpolation frame generation device according to an embodiment of the present invention. In the figure, the interpolation frame generation device includes a first frame buffer unit 1, a frame delay 2, a second frame buffer unit 3, a whole screen motion detection unit 4, a static region determination unit 5, an interpolation frame motion vector generation unit 6, The background determination unit 7, the interpolation frame generation unit 8, the background image extraction / synthesis unit 9, and the background image storage unit 10 are configured.

  The first frame buffer unit 1 and the second frame buffer unit 3 are buffers for storing the first reference frame and the second reference frame of the input image, respectively. The interpolated frame generates a frame between them. The frame delay 2 is a circuit that delays the first reference frame by a frame that is temporally close, and the first reference frame represents a reference frame that is a frame that is temporally close to the second reference frame.

The entire screen motion detection unit 4 is a motion detection unit that calculates a motion vector (g _x , g _y ) of the entire screen, and there are several algorithms for realizing this, and will be described later. Basically, the motion vector of the entire screen is made to substantially coincide with the motion vector of the static region with little motion.

The static motion area determination unit 5 calculates a static area with less motion and other motion areas between the first reference frame and the second reference frame. This calculation may be performed with the size of the interpolation block or may be performed for each pixel.
There are several methods for determining the static region, but the following introduces a calculation method using the gradient method. According to the gradient method, the following relational expression holds for continuous image data.

Here, v _n (x, y) represents a motion vector between the first reference frame and the second reference frame at coordinates (x, y).
Converting this formula by focusing on the magnitude of the motion vector

It becomes. here,

Represents the horizontal and vertical gradients of the first reference frame,
f _n (x, y) −f _n−1 (x, y)
Is the frame difference between the first reference frame and the second reference frame. These values can be obtained by a simple calculation. That is, since it is sufficiently possible to process the equation (3) by the static region determination unit 5, this equation (3) can be adopted for the determination of the static region.
Here, when the motion vector (g _x , g _y ) of the entire screen is further taken into consideration, the equation (3) can be rewritten as the following equation (4).

  As a result, the static motion region determination unit 5 determines that the value is the motion region if the value of the expression (4) is larger than a certain threshold value, and the static region if the value is smaller. It is empirically known that the threshold value needs to be adjusted while actually creating a circuit and viewing the state of the image. These can be expressed by the following equation (5).

Here, h _n (x, y) is a static motion determination value at the coordinates (x, y) and corresponds to the output of the static motion region determination unit 5. h _n (x, y) has a null value in the static region and a normal value in the dynamic region. v _th represents a threshold value for determining static motion. When the motion of the image is larger than this threshold, it is determined as a moving region, and otherwise it is determined as a static region. K () is a conversion function, and is a function that replaces the result of Expression (4) with a simple numerical value.
The contents of the conversion function k () are not directly related to the present invention and will not be described in detail. However, if the value of | v | itself is output from the static region determination unit 5, the amount of data increases. Therefore, this value is used to convert the value into, for example, a 16-bit 4-bit representation.

  The interpolation frame motion vector generation unit 6 is a processing unit that obtains a motion vector for each interpolation block on the interpolation frame, and allocates the motion vector obtained between the first reference frame and the second frame on the interpolation frame by scale conversion. . Furthermore, in order to select a more accurate motion block from a plurality of assigned interpolation blocks, processing represented by Expression (1) and Expression (2) is performed.

  The background determination unit 7 evaluates between the background reference image and the first reference frame or the second reference frame, and calculates a determination value DUDF for the first reference frame and a determination value DUDB for the second reference frame. Output as background judgment value. The evaluation with the background reference image uses, for example, the following formulas (6) and (7).

These equations (6) and (7) are based on the full-screen motion vector (g _x , g _y ), but may not be considered because the motion is fine.
Here, U _n (x, y) represents a background reference image. “ux” and “uy” are values representing the offset of the background reference image, and are conversion coefficients for correction when the positions of the first reference frame and the background reference image are shifted due to panning or the like. Here, for the sake of simplicity, it is assumed that only parallel movement has been performed, and correction is performed using only ux and ui. Actually, there is a method of detecting a movement of several background images, obtaining a projective transformation matrix, and using it as a transformation coefficient.

  The conversion coefficients ux and ui are added for each frame. For example, assuming only parallel movement, ux and uy can be obtained as follows using n as a frame number.

ux _n = ux _n-1 + g _x
uy _n = uy _n-1 + g _y

  In the interpolated frame generation unit 8 of the present invention, the output of the background determination unit 7 is taken into account, and the value of the DFD based on the motion vector obtained by Equation (2) is compared with the values of Equation (6) and Equation (7). The smallest value is adopted as the interpolation image.

However, there are conditions for adopting the background determination values of Expressions (6) and (7), and only when all the background reference images U _n (x, y) referenced in the expressions exist. When there is no data, it is expressed as U _n (x, y) = null. In such a case, the background determination value is also null.

  In addition, when the values of Equation (6) and Equation (7) are selected as the smallest value, the background reference image is used for image interpolation. In the case of Expression (6), the background reference image and the first reference frame are used, and in the case of Expression (7), the background reference image and the second reference frame are used.

  Considering these processes in the occlusion region, the interpolated frame motion vector generation unit 6 in the prior art cannot calculate a motion vector from the first and second reference frames, and is used in the evaluation of Expression (2). Even when a correct motion vector does not exist among the motion vectors assigned to the interpolation block to be selected and a motion vector having the smallest value of the expression (2) is selected, it is an incorrect motion vector. Therefore, in the prior art, interpolation using the wrong motion vector has been performed.

  However, in the present invention, the comparison is performed using the background reference image accumulated in the frame before the second reference frame, and in the occlusion area, the expressions (6) and (7) are evaluated values of the expression (2). Since it is always lower, correct interpolation with the background reference image can be performed.

  The background image extraction / synthesis unit 9 reads out still area image data from the first reference frame in order to generate a background reference image, and synthesizes the image data with the background reference image already stored. Do. These processes can be expressed by, for example, the following formula (8).

Here, U _{n + 1} (x, y) indicates a background reference image used when the next interpolation frame is generated, and the data accumulated in the background image accumulation unit 10 from the background image extraction / synthesis unit 9 Represents. G (I, U) represents a synthesis function for synthesizing the image I of the first reference frame and the background reference image U. For example, when the luminance of the first reference frame is significantly different from the background reference image, The value of one reference frame is used as it is, and when the difference is within a certain range, the average value of the sum of both is taken. ux and ui are obtained from the background determination unit.
Further, the condition of I _n (x, y) = null indicates that the coordinate (x, y) of the first reference frame has been determined as a moving area by the static movement area determining unit 5, and Expression (8) In the case where the above condition is met, the background image extracting / synthesizing unit 9 does not perform processing, and the current value stored in the background image accumulating unit 10 is retained.

When the first reference frame is the first frame or when a scene change (no correlation between frames occurs), U _{n + 1} (x, y) is all null. As a result, all the static regions of the first reference frame and the second reference frame are newly registered in the background reference image in the next frame. Therefore, in the occlusion area immediately after the scene change, since the background reference image is null, the effect of the present invention is temporarily lost. However, since such a single phenomenon is difficult to visually recognize, it does not lead to deterioration of the moving image as a whole.

  In Expression (8), the output hn (x, y) from the static motion region determination unit 5 as to whether or not it is a static region has been described. This condition uses the motion vector Vn (x, y) directly. Also good. This is also clear from equation (5). In the block diagram in such a case, the static region determination unit 5 may be omitted as shown in FIG. Since the static region determination may be performed independently of the motion vector calculation, the block diagram as shown in FIG. 2 may be used.

The interpolation frame generation unit 8 generates interpolation frame images for all interpolation blocks from the first reference frame, the second reference frame, and the background reference image. The interpolation frame generation method is switched based on the static motion determination value h _n (x, y) that is the output of the static motion region determination unit 5 for each interpolation block.
First, when h _n (x, y) is not null, the motion vector v _{(x, y)} obtained from the interpolated frame motion vector from the images of the first reference frame and the second reference frame is used as a moving region as before. An image is generated by interpolation. When this is expressed by a simple linear interpolation formula, the following formula (9) is obtained.

Z _n (x, y) = m · f _n (x + v _x , y + v _y ) + n · f _n−1 (x, y) (9)

Here, Z _n (x, y) represents an interpolated frame image output from the interpolated frame generation unit 8.
M and n represent interpolation ratios of interpolation frames, and m + n = 1.
Next, when h _n (x, y) is null, interpolation is performed between the background image and the first reference frame or the second reference frame as a static region. For example, when frame interpolation is performed by linear interpolation, the following equations (10) to (12) are obtained.

Z _n (x, y) = m · U _n (ux + x + g _x , uy + y + g _y ) + n · f _n−1 (x, y) (10)
Z _n (x, y) = m · f _n (x, y) + n · U _n (ux + x, uy + y) (11)
Z _n (x, y) = m · f _n (x, y) + n · f _n−1 (x, y) (12)

Expression (10) is an example of linear interpolation when DUDF exists and DUDF is smaller than the motion vector determination value (DFD value of Expression (2)), Expression (11) has DUDB, and It is an example of linear interpolation when DUDB is smaller than the motion vector determination value (DFD value of Expression (2)). When DUDF and DUDB are similarly smaller than the motion vector determination value, the smaller of DUDF and DUDB is used.
Expression (12) is an example of linear interpolation when the above determination is lost. Expressions (10) and (11) are examples when the background moves in parallel. If other motions are considered, it is necessary to take the transformation matrix into consideration when obtaining values from the background reference image.

Practically, equations (9) to (12) need to be improved because luminance variations occur at block boundaries if these equations are left as they are. For example, it is conceivable that an image such as f _n (x, y) is weighted so that the influence of the (x, y) peripheral luminance is transmitted with respect to the reference location. That means

Devise such as. Where w (i, j) is a weighting function,

Is a function that holds. This is generally determined by using a filter design method or the like. Applying this to equation (10),

It becomes. The same applies to other equations.
When the interpolation blocks complemented by the equations (9) to (13) are combined, an interpolation frame output from the interpolation frame generation unit 8 is obtained.

There are several methods for detecting the movement of the entire image.
For example, “TELEVISION MOTION MEASUREMENT FOR DATV AND OTHER APPLICATIONS” of BBC RD 1987/11 performs motion vector detection using a phase correlation method. When this method is used for the entire screen, the motion vector of the largest region in the screen, that is, the portion considered to be the background, can be obtained from the coordinates of the highest peak on the correlation plane.

  Also, according to “Motion Compensated Frame Interpolation by new Block-based Motion Estimation Algorithm” of IEEE Transaction on CE, Vol.50, No.2, MAY 2004, the motion vector of the entire screen can be calculated from .

Here, g _x and g _y are the x component and y component of the motion vector, respectively, H _n (h) is the luminance average of the pixels existing at the coordinate x = h, that is, the luminance average of the pixels arranged vertically, V _n (v ) Represents the average luminance of horizontal pixels existing at the coordinate y = v, and S _h and S _v represent the search range.

  As described above, various methods for obtaining the motion vector of the entire screen have been proposed. In the present invention, motion detection of the entire screen is performed using these techniques.

As another embodiment, when determining which of Equation (9) and Equations (10) to (12) is used in the static region determination of the interpolation frame generation unit, the static and static region determination value h _n (x, y) Not only DUDF or DUDB is used for determination, but if DUDF or DUDB is below a certain threshold, an interpolation image is forcibly generated as a static region using equations (10) to (12) Also good.

  Furthermore, as another embodiment, the following expression can be used in the generation of the background reference image.

  That is, when a background reference image is generated, not only the first reference frame but also a portion of pixels corresponding to the same background image in the second reference frame is synthesized. More complicatedly, there is a method of generating by filtering both reference frames.

Further, according to the present invention, an image display system can be configured using the interpolation frame generation device described in the above embodiment.
In this image display system, an input image signal is input to an interpolation frame generation device, and an interpolation frame image signal is created by the method described so far. Control is performed to output either the interpolated frame image signal or the input image signal, and the image is output to the hold-type image display device. In the display device, an image is displayed by changing the refresh rate in accordance with the synchronization signal included in the output image signal.

Note that the present invention can also be configured as a computer program for realizing the functions of the interpolation frame generation apparatus and the image display system of the present invention. This program is stored in a computer-readable recording medium, and each process is realized by the program.
As the recording medium, the main memory itself may be a program medium, or a program medium provided with a program reading device as an external storage device and readable by inserting the recording medium therein.

  Here, the program medium includes a tape system such as a magnetic tape and a cassette tape, a disk system such as a magnetic disk such as a flexible disk and a hard disk, an optical disk such as a CD-ROM / MO / MD / DVD, and an IC card (memory card). Or a card system such as an optical card, or a medium carrying a fixed program including a semiconductor memory such as a mask ROM, FPROM, EEPROM, flash ROM, or the like.

Further, in the present embodiment, since the configuration is connectable to a network, the program stored in the storage device may be directly supplied from the server computer via the network. In this case, the storage device of this server computer is also included in the recording medium of the present invention.
As described above, by programming and distributing the functions of the above-described embodiments, the cost can be reduced, and the portability and versatility can be improved.

It is a block diagram which shows the whole structure of the frame interpolation production | generation apparatus which concerns on embodiment of this invention. It is a block diagram which shows the function structure of the frame interpolation production | generation apparatus which concerns on embodiment of this invention. It is a block diagram which shows the other function structure of the frame interpolation production | generation apparatus which concerns on embodiment of this invention. It is a block diagram which shows the structure of the frame interpolation production | generation apparatus of a prior art. It is a figure for demonstrating the relationship between an interpolation frame and a motion vector. It is a figure for demonstrating an occlusion area | region.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... 1st frame buffer part, 2 ... Frame delay, 3 ... 2nd frame buffer part, 4 ... Whole screen motion detection part, 5 ... Static motion area determination part, 6 ... Interpolation frame motion vector generation part, 7 ... Background determination 8 is an interpolation frame generation unit, 9 is a background image extraction / synthesis unit, 10 is a background image storage unit.

Claims (20)

  In an interpolated frame generation apparatus that generates a new interpolated frame that does not exist from information of existing frames using an image composed of a plurality of temporally continuous frames, a background image is generated from some existing frames. An occlusion area in two or more reference frames extracted from existing frames, having background image extraction / synthesis means for extracting and synthesizing a background image and background image storage means for storing the generated background image An interpolated frame generating device characterized by   In an interpolation frame generation method that generates a new interpolation frame that does not exist from information of existing frames using an image composed of a plurality of temporally continuous frames, a background image is obtained from some existing frames. Extracting and synthesizing the background image, and having a background image extraction / synthesizing process for accumulating the background image generated in the background image accumulating means, the occlusion area in two or more reference frames extracted from existing frames An interpolation frame generation method characterized by detecting.   In an interpolated frame generation apparatus that generates a new interpolated frame that does not exist from information of existing frames using an image composed of a plurality of temporally continuous frames, a background image is generated from some existing frames. A new interpolated frame from two or more reference frames extracted from existing frames, having a background image extracting / synthesizing unit for extracting and synthesizing a background image and a background image accumulating unit for accumulating the generated background image An interpolated frame generation apparatus characterized in that an accurate motion vector is obtained even for a portion in which motion vector detection is difficult from the reference frame due to occlusion or the like.   In an interpolation frame generation method that generates a new interpolation frame that does not exist from information of existing frames using an image composed of a plurality of temporally continuous frames, a background image is obtained from some existing frames. It has a background image extraction and synthesis process that extracts and synthesizes the background image and accumulates the background image generated in the background image accumulation means, and creates a new interpolation frame from two or more reference frames extracted from existing frames. An interpolation frame generation method characterized in that, when generating, an accurate motion vector is obtained even for a portion where it is difficult to detect a motion vector from the reference frame for reasons such as occlusion.   In an interpolated frame generation apparatus that generates a new interpolated frame that does not exist from information of existing frames using an image composed of a plurality of temporally continuous frames, a background image is generated from some existing frames. A new interpolated frame from two or more reference frames extracted from existing frames, having a background image extracting / synthesizing unit for extracting and synthesizing a background image and a background image accumulating unit for accumulating the generated background image An interpolated frame generating apparatus that interpolates a background image that is difficult to interpolate from the reference frame for reasons such as occlusion using the background image storage means.   6. The interpolated frame generating apparatus according to claim 5, wherein when a new interpolated frame is generated from two or more reference frames extracted from existing frames, the background image stored in the background image storing means is used. An interpolated frame generation apparatus that divides an image of a reference frame into a background image and an image of a moving object other than the background image.   In an interpolation frame generation method that generates a new interpolation frame that does not exist from information of existing frames using an image composed of a plurality of temporally continuous frames, a background image is obtained from some existing frames. It has a background image extraction / synthesis unit that extracts and synthesizes the background image and stores the generated background image in the background image storage unit, and creates a new interpolation frame from two or more reference frames extracted from existing frames. A method of generating an interpolated frame, comprising: interpolating a background image, which is difficult to interpolate from the reference frame for reasons such as occlusion, using the background image storage means.   8. The interpolation frame generation method according to claim 7, wherein when a new interpolation frame is generated from two or more reference frames extracted from existing frames, the background image stored in the background image storage means is used. Then, an interpolation frame generation method characterized by dividing an image of a reference frame into a background image and an image of a moving object other than the background image.   In an interpolation frame generation apparatus for generating an interpolation frame from a plurality of reference frames that are temporally close to each other in an image, an entire screen motion detection unit that detects a motion vector of the entire screen (hereinafter referred to as an all image motion vector) from the reference frame Interpolated frame motion vector generating means for generating a motion vector on the interpolated frame from the reference frame, background image accumulating means for accumulatively storing background images past the reference frame, and the background image Background determination means for performing background determination evaluation from the background reference image, the reference frame, and the whole image motion vector of the storage means, and a new background image is synthesized from the result of the background determination evaluation, the reference frame, and the background reference image Background image extraction / combination means, the whole image motion vector and the interpolated frame motion vector And an interpolated frame generating means for generating an interpolated frame image from the reference frame image and the background reference image using the result of the background determination evaluation of the background determining means. An interpolation frame generation device characterized in that an interpolation image can be generated even in a difficult region.   The interpolation frame generation device according to claim 9, wherein the background reference image and the reference frame image of the background image storage unit are used to compare the luminance for each pixel or each interpolation block. It further has a static area determination means that can more accurately distinguish a static area with less movement from other dynamic areas, and using the result, the processing of the background determination means and the background image extraction / synthesis means is more accurately performed. An interpolated frame generation device characterized by being realized.   10. The interpolated frame generation apparatus according to claim 9, wherein the background image extraction / combination means uses a background resolution image and a plurality of reference frame images to increase the resolution of the high-resolution background using an image resolution enhancement technique. An interpolation frame generation apparatus characterized by generating an image.   10. The interpolated frame generation apparatus according to claim 9, wherein the background image extraction / combination means includes a background image storage means when a scene change is detected or when a large difference occurs between the background reference image and the reference frame. An interpolated frame generation apparatus characterized by clearing the contents.   In the interpolation frame generation method for generating an interpolation frame from a plurality of reference frames that are temporally close to each other in an image, an entire screen motion detection step of detecting a motion vector of the entire screen (hereinafter referred to as an all image motion vector) from the reference frame An interpolated frame motion vector generating step for generating a motion vector on the interpolated frame from the reference frame, and a background reference image possessed by a background image accumulating means for accumulatively storing past background images from the reference frame A background determination step of performing a background determination evaluation from the reference frame and the whole image motion vector, and a background image extraction / synthesis step of combining a new background image from the result of the background determination evaluation, the reference frame, and the background reference image The whole image motion vector, the interpolated frame motion vector, and the background image. An interpolation frame generation step of generating an interpolated frame image from the reference frame image and the background reference image using the result of the background judgment evaluation of the process, and interpolation is performed even in a region that is difficult to associate with the reference frame image alone An interpolation frame generation method characterized by enabling generation of an image.   14. The method of generating an interpolation frame according to claim 13, wherein brightness of each pixel or interpolation block is compared by using a background reference image and a reference frame image of the background image storage means, for example. It further has a static motion region determination step that can more accurately distinguish a small number of static regions and other motion regions, and using the result, the background determination step and the background image extraction / synthesis process are more accurately realized. An interpolation frame generation method characterized by the above.   14. The interpolation frame generation method according to claim 13, wherein the background image extraction / synthesis step uses a background reference image and images of a plurality of reference frames to obtain a high-definition high-resolution background using an image resolution enhancement technique. An interpolation frame generation method characterized by generating an image.   14. The method of generating an interpolated frame according to claim 13, wherein the background image extracting / synthesizing step is performed by the background image accumulating unit when a scene change is detected or when a large difference occurs between the background reference image and the reference frame. An interpolation frame generation method characterized by clearing the contents.   An image display system for displaying an interpolated frame and a reference frame generated by the interpolated frame generating apparatus according to claim 1, 3, 5, 6, 9, 10, 11, or 12 so as to be temporally continuous.   The program which implement | achieves the function of the interpolation frame production | generation apparatus of Claim 1, 3, 5, 6, 9, 10, 11 or 12 in a computer, or the function of the image display system of Claim 17.   A program for causing a computer to execute the processing steps of the interpolation frame generation method according to claim 2, 4, 7, 8, 13, 14, 15 or 16.   A computer-readable recording medium on which the program according to claim 18 or 19 is recorded.

Images