JP3197801B2 - 2D display image generation method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for generating an image for two-dimensional display, and more particularly to a method for generating a new image by processing input monocular or multiview images. INDUSTRIAL APPLICABILITY The present invention can be mainly used for changing the viewpoint of an image, partially enlarging or reducing an image, and cutting out an image area.

[0002]

2. Description of the Related Art From the viewpoint of the number of cameras that shoot images, images are roughly classified into monocular images and multi-view images. A monocular image is obtained when a subject is photographed from one camera, and a multi-eye image is obtained when a subject is photographed from a plurality of cameras. The latter is also called a stereo image.

On the other hand, when an image is output by performing predetermined processing on the video, the output is roughly classified into a two-dimensional display image and a three-dimensional display image from the viewpoint of a display format. The three-dimensional display image is an image given to both the left and right eyes for pseudo three-dimensional display. In the case of a still image, a three-dimensional display image usually includes a pair of frames. In the case of a moving image, the set of frames is given to the viewer one after another. The two-dimensional display image refers to a normal image other than the three-dimensional display image. That is, the two-dimensional display still image refers to an image such as one photograph, and the two-dimensional display moving image refers to an image such as a normal television image. Hereinafter, in this specification, “video” mainly refers to an input to be subjected to image processing (except for a viewpoint change video described later), and “image” mainly refers to an output after processing.

[0004] Among the cases classified according to the input / output type described above, the present invention assumes a two-dimensional display image as an output. The following two combinations of inputs and outputs are assumed in the present invention.

(1) Input is a monocular image, output is a two-dimensional display image. (2) Input is a multi-eye image, output is a two-dimensional display image. Of these, in (1), it is general to output an input image as it is. It is. In (2), in many cases, one of the multi-view images is selected and output as it is. The merit of capturing a multi-view video is mainly exhibited in generating a three-dimensional display image. The present invention relates to these (1) and (2)
In this field, a completely new image processing method is provided, and there is no corresponding conventional technology.

[0006]

SUMMARY OF THE INVENTION An object of the present invention is to perform the following processing on an input video and generate the following output image. At this time, through the derivation and use of three-dimensional position information including information on the distance between the camera position (that is, the viewpoint of the observer, hereinafter simply referred to as “viewpoint”) and the subject (hereinafter, simply referred to as “depth information”), It is characterized in that the processing of

[0007] 1. Viewpoint change When the viewpoint is virtually changed, the image should change. The present invention automatically generates an image to be obtained after changing a viewpoint (hereinafter, referred to as a “changed viewpoint image”) while fixing the actual camera position.

[0008] 2. Partial enlargement / reduction of an image In image processing, partial enlargement / reduction of an image is often performed, but the present invention uses depth information to automatically generate the most natural and effective image.

3. Extraction of Image Area When extracting a desired image area, it is necessary to recognize the image area. Several techniques, such as a clustering technique, have been proposed for recognizing image regions, but have not been very successful. The present invention extracts a region with high accuracy from a completely different viewpoint of depth information.

[0010]

According to the present invention, depth information of an image to be processed is extracted from an original image to be processed, and an image for two-dimensional display is generated in accordance with this information. Here, the “original video” may be a video captured with a single eye or a stereo video captured with multiple eyes. In addition, not only a video taken by a camera but also an animation video or computer graphics may be used.

At this time, the present invention detects a two-dimensional displacement of each part of the image between a plurality of image frames,
The depth information is extracted from this information. “Each part of an image” refers to each area and each pixel constituting an image. The “video frame” is a processing unit of video, and includes an image field, an MPEG picture, and the like in addition to a normal image frame. The “plurality of video frames” refers to a plurality of frames photographed at different times in the case of a single eye (hereinafter, “different time frames”). In the case of a multi-view camera, a plurality of frames photographed at the same time (hereinafter, “simultaneous frame”) may be used, or a different time frame by one camera constituting the multi-view camera may be used. “Two-dimensional” means that the video frame is a planar video. Even in the case of multi-view, the image of each camera is a plane image. “Two-dimensional displacement of position” refers to displacement of a position on a plane. A different time frame indicates a change in position (ie, movement) with the passage of time, and a same time frame indicates a position shift between a plurality of frames.

In one embodiment of the present invention, the plurality of video frames are different time frames, and these different time frames are selected based on the two-dimensional position displacement. If the displacement amount is too large, for example, a method of selecting a plurality of frames by reducing the time difference can be considered.

In the present invention, the displacement of the two-dimensional position of each part of the image may be statistically processed, and a plurality of image frames may be selected based on the processing result. As a statistical process, the variance of the motion vector of each part of the video may be derived, and a plurality of video frames may be selected so that this value is larger than a predetermined value. The derivation of the variance is performed by a known calculation. If the movement of each part of the image is too small, the calculation error increases, so that it is preferable that the variance is large to some extent.

Therefore, according to an aspect of the present invention, when it is not possible to select a plurality of video frames whose variance is larger than a predetermined value, the generation of the two-dimensional display image is stopped. This is to avoid a situation in which an image is generated unnaturally due to the size of the error.

The present invention derives a relative positional relationship occupied by each part of the image in the actual three-dimensional space from the two-dimensional displacement of each part of the image thus obtained. The depth is determined according to the result. When the input is a monocular image, the absolute position is not always obtained. This is a problem of the scale factor described later. Therefore, at least the relative position is obtained.

At this time, a three-dimensional movement of each part of the image is calculated from the displacement of the two-dimensional position, and a position coordinate of each part of the image in a three-dimensional space is calculated from the movement according to the principle of triangulation. Then, the depth may be determined according to the result. In the case of a single eye, the principle of triangulation is applied assuming the movement of the camera. In the case of multiple eyes, this principle is applied to, for example, left and right images.

In the present invention, it is necessary to grasp the correspondence of the subject between a plurality of video frames. Therefore, a representative point is set in a reference video frame (hereinafter, referred to as a “reference frame”), and a corresponding point of the representative point in another video frame (hereinafter, referred to as a “target frame”) is determined. And the two-dimensional displacement of each part of the image may be recognized. This is a kind of image recognition technology. The “corresponding point” here is a point corresponding to the representative point. The corresponding points are "true corresponding points"
And "computationally corresponding points". Corresponding points should originally exist uniquely for each representative point, and it is impossible to imagine a state where a point other than the existence point is a corresponding point. This ideal corresponding point is a true corresponding point.

On the other hand, the corresponding points determined from the image processing calculation are:
It does not always match the true corresponding point. This is the corresponding point in the calculation. The corresponding points in the calculation may not only be present in places other than where the true corresponding points exist, but their positions may be changed as appropriate. The latter occurs when, for example, a process of improving the corresponding point accuracy described below is performed.

In this specification, unless otherwise required,
The word “corresponding point” is used without distinction for the two concepts of “true corresponding point” and “calculated corresponding point”.

Here, when the reference frame and the target frame are different time frames, the position of the corresponding point of another target frame is predicted from the positional relationship between the representative point and the corresponding point.
The area for searching for the corresponding point may be limited. In the simplest example, the position of the subject in another target frame may be predicted on the assumption that the motion of the subject detected between the reference frame and the target frame is maintained at the same speed.

At this time, for the representative points related to the image region having a geometric feature, the geometric characteristics are maintained so that the image region related to the corresponding point of the representative point is also maintained. The position of the corresponding point may be adjusted. Examples of video parts having “geometric features” are the corners of the roof of a house and the horizon. The geometric feature may be constituted not only by the geometric feature of the subject but also by a color change point in the video. It is based on an empirical rule that if geometric features are maintained, correspondences can be grasped more accurately.

At this time, an area including a straight line may be adopted as a geometrically characteristic image area. It is assumed that a portion that forms a straight line in the reference frame also forms a straight line in the target frame. The calculation load for finely adjusting the position of the corresponding point and putting it on a straight line is relatively light.

The similarity of the video is evaluated between the video area near the specific point in the target frame and the video area near the representative point of the reference frame. If the evaluation result is good, the specific point corresponds to the representative point. You may decide a point. The similarity of the video is used to grasp the correspondence. The criterion of “similarity” is closeness of colors, closeness of shapes, and the like.

Here, the "specific point" refers to a candidate of a corresponding point to be input into the evaluation. There is no limitation on how to determine candidates. For example, a plurality of specific points may be taken at equal intervals, and a point having the best evaluation result among these points may be set as a corresponding point.

Further, not only the similarity but also the validity of the relative position between the specific points is evaluated, and if the result of both the evaluations is good, the specific point may be determined as the corresponding point of the representative point. . For example, if it is determined that point B, which was on the right of point A in the reference frame, exists on the left of point A in the target frame as a result of the similarity evaluation, such a reversal phenomenon may also occur in the surrounding correspondence. For example, the position of point B can be considered valid. As another example, the position may be evaluated by assuming that the center of gravity of a certain region taken in the reference frame is also near the center of gravity of the region in the target frame.

Here, the corresponding point may be determined after the two evaluations have been completed, or the corresponding point may be temporarily determined at the stage when one of the two evaluations has been completed, and the validity thereof may be determined by the other evaluation. .

When a corresponding point is determined through these two evaluations, in one embodiment of the present invention, the results of each evaluation are quantified and integrated, and the numerical value is recalculated while changing the position of the corresponding point, and repeated. The position accuracy of the corresponding point may be improved through the calculation. That is, the two types of evaluation results are weighted and added, and a corresponding point for optimizing the added results is obtained.

At this time, further, once the positions of all the corresponding points are fixed, a point where the result of each evaluation is the best is searched while moving only one corresponding point, and the position of the searched best point is determined. The new position of the one corresponding point may be set as a new position, and the search and the change of the position may be sequentially performed on all the corresponding points.

When moving a corresponding point X, two evaluations are performed after fixing the other corresponding point, and a point at which the result is the best is set as a new position of the corresponding point X. This completes one position improvement of the corresponding point X. Subsequently, similar search and movement processing are performed for the next corresponding point. At this time, other points are fixed. The processing is performed for all corresponding points, and is repeated twice or more as necessary. Hereinafter, this improved method will be referred to as a “fixed search method”.

After the fixed search processing is completed, the position accuracy of the corresponding point may be improved using Euler's equation. The Euler equation represents a condition under which the numerical value after integration of the two evaluations has a maximum value or a minimum value. This numerical value reflects the evaluation results of all corresponding points. According to the Euler equation, it is possible to improve the position of the corresponding point with higher accuracy than the minimum unit (for example, pixel) of the image.

In the present invention, the similarity evaluation may be performed by block matching. According to this method, both color similarity and shape similarity are evaluated simultaneously.

More specifically, this block matching may be performed by calculating the sum of the n-th error of the color density between the video areas to be compared. n is 1 or 2. When n = 1, the sum of the absolute values of the color density differences is obtained.

Here, block matching may be performed on the color density in consideration of a predetermined color deflection constant (hereinafter referred to as "biased block matching"). In the case of the same time frame, a certain deviation in color density is likely to occur due to the characteristics of a plurality of cameras. The same phenomenon occurs when the weather changes with the time (that is, the brightness of the image changes) even in the case of different time frames by the same camera. A color deflection constant is given so as to cancel this.

Here, the color deflection constant is determined so that the sum of the n-th power errors is minimized. This is because it is considered that the determination of the color density is most effectively eliminated.

More specifically, the color deflection constant may be an average value of the color density difference of each pixel between the image areas. At this time, the most general sum of squared errors is minimized. In the block matching, the sum of square errors may be calculated after subtracting the color deflection constant from the color density difference of each pixel between image areas.

In one embodiment of the present invention, points included in an image are classified into feature points and non-feature points, and a corresponding point is preferentially determined for a representative point which is a feature point. The corresponding points that are non-feature points can be determined by interpolating the corresponding points that are feature points. Block matching may be performed between the reference frame and the target frame, and as a result, a corresponding point having a good correspondence with the representative point may be determined as a feature point. Block matching is biased
Block matching may be used.

In the present invention, the point where the position is stably changed in the different time frame may be set as a feature point. In other words, a point where the motion vector is stable is defined as a feature point. This is because such points are considered to be accurately tracked.

A feature point may be that a point at which the displacement of the position between the same time frames (that is, the displacement of the position) is substantially constant or changes constantly between the same time frames photographed at the near time. This is also because tracking accuracy is high.

In another aspect of the present invention, a viewpoint-change video is generated according to the depth information. For this purpose, the displacement of the position of each part of the image due to the assumed viewpoint change is calculated back from the depth information, and the image is reconstructed according to the displacement of the position. This is a kind of image processing technology. For example, when changing the height of the viewpoint, the displacement (translation amount and rotation amount) of the subject (each part of the image) can be calculated from the moving distance of the camera and the depth information. An intended video may be constructed according to the calculation result.

When the original video was captured by a twin-lens camera, a virtual camera is assumed to be located sufficiently close to these two eyes, and a video to be captured from this virtual camera is generated as a viewpoint change video. Alternatively, a multi-viewpoint image may be generated from the viewpoint-changed image and a real image captured by the twin-lens camera. Generally, when obtaining a multi-view video, it is necessary to shoot the video with three or more cameras. Therefore, in this embodiment, when a video image using a twin-lens camera originally exists, a video image captured by a virtual camera is generated as a viewpoint change video image, and a multi-view video image is generated using these three or more video images.

As another mode, while changing the viewpoint of a video virtually from the viewpoint at which a certain video frame is captured to the viewpoint at which another video frame is captured, a viewpoint change is performed at an arbitrary point on the movement route. An image may be generated. For example, the viewpoint at which the video frame 1 is photographed is the viewpoint 1,
If the viewpoint at which another video frame 2 is photographed is defined as viewpoint 2, the present invention virtually shifts the viewpoint of the image from viewpoint 1 to viewpoint 2 while moving the image to be photographed at any moving viewpoint. Generate as change video. When the video frames 1 and 2 form different time frames, if the viewpoint change video is generated more, a video that smoothly changes between the video frames 1 and 2 can be obtained. This can be applied, for example, when obtaining a smooth slow motion image. Of course, the video frames 1 and 2 may be the same time frame.

According to another aspect, an image is generated after changing the size of a partial region of a video image based on depth information. A region having a small depth may be selected as a partial region of a video, and the region may be enlarged before generating an image.
In general, highlighting is possible by making a subject closer to the viewpoint look closer. Therefore, an area having a small depth is selected and expanded.

As a reverse mode, an area having a large depth may be selected, the area may be reduced, and an image may be generated. This is because, by making a subject that is originally far from the line of sight look farther away, the importance of such an area is reduced and displayed.

It should be noted that images from different viewpoints,
For the video after the enlargement / reduction, the occlusion relationship between the video regions may be considered. For example, a subject having a large depth may be shielded by a subject having a small depth by moving the viewpoint, or a background may be blocked by enlarging a certain subject. In such a case, a more natural image can be obtained by generating an image by deleting the image of the portion to be occluded.

Undesired steps may appear in the image frame due to the image processing such as the viewpoint change image and the enlargement / reduction of a partial area. In the present invention, this step may be corrected. This is because, for example, as a result of the image processing, when the image edge is displaced such that it moves toward the center of the image, that portion appears to be depressed toward the center. In this portion, a smooth end portion may be reproduced as in the case of the original video by a method such as interpolation.

In still another embodiment, according to the depth information,
A desired video area is cut out to generate an image. The cutout is performed, for example, by selecting a part having a depth within a predetermined range from each part of the image. For example, when selecting only an image part with a depth of 10 meters or less, only a subject within 10 meters from the viewpoint is cut out. This is a kind of image recognition technology.

At this time, a new image may be generated by superimposing the clipped video area on another video. In the above example, a virtual image can be obtained by cutting out a subject existing within 10 meters and pasting it to a separately prepared scenery, another scenery shot at the same time, a CG image generated at the same time, or the like. .

[0048]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiment 1 FIG . Preferred embodiments of the present invention will be described with reference to the drawings as appropriate. In the first and second embodiments, it is assumed that a video shot with a single eye is input. The case of multi-view (stereo video) will be described in a third embodiment.

FIG. 1 is a view showing main steps for generating a two-dimensional display image according to the present embodiment. As shown in the figure, according to the present embodiment, an image for two-dimensional display is generated from an original input video according to the following steps. Here, steps 1 to
3 relates to derivation of depth information by analyzing a video, and step 4 relates to generation of a desired image.
First, the outline of each step will be described.

[Step 1] Extraction of two-dimensional motion information The motion information of the subject contained in the video is extracted. The motion information is two-dimensional information, and the display screen is set on a coordinate plane, and the motion of the subject on the screen is described in two-dimensional coordinates.

In this step, the correspondence between video frames is detected in order to grasp the movement of the subject. A plurality of representative points are set in advance in a video frame at time t (hereinafter referred to as “frame t”) which is a reference frame, and a different time t ′
The corresponding points of the respective representative points in the target frame (hereinafter referred to as “frame t ′”) are tracked. Frames t and t '
Form inter-time frames, which need not be temporally adjacent frames. The feature of this step is that two-dimensional motion information can be extracted from the motion of the subject in an arbitrary direction.

In the present specification, the term “frame” is hereinafter referred to as “frame”.
Broadly refers to the overall video composition unit such as the field,
For example, it does not refer to only one frame of a television receiver composed of 525 scanning lines or one screen of a personal computer composed of 640 × 480 pixels. The representative point may be set not only in the frame t but also in both the frames t and t ′.

[Step 2] Calculation of Three-Dimensional Motion Information When the two-dimensional motion of the subject is determined, actual motion information (three-dimensional motion information) of the subject in the three-dimensional space is calculated. At this time, by taking a large number of pairs of representative points and corresponding points, the motion actually caused by the subject is described by both translational and rotational motion components.

[Step 3] Acquisition of Depth Information If the actual state of movement of the subject is known, the relative positional relationship of the subject at each time can be determined. If this relationship is known, depth information of the subject or each part thereof (hereinafter, also simply referred to as “each part of the image”) is obtained.

[Step 4] Generation of Image The desired image is automatically generated according to the three-dimensional information including the depth information. Here, the viewpoint change video, part of the image
An image generated by reduction and an image generated by cutting out a certain image area will be described.

The above is an outline. Hereinafter, each step will be described in detail.

[Step 1] Extraction of Two-Dimensional Motion Information FIG. 2 is a flowchart for detecting the correspondence between video frames. Each step shown in the figure will be described.

(S10) Setting a representative point in the frame t As shown in FIG. 3, first, a representative point is set in the reference frame t. In the figure, a frame t is divided by a mesh every 8 × 8 pixels, and each representative point is placed at a grid point. Here, the i-th representative point from the left and the j-th representative point from the top are Pt
(I, j), and the time t 'with respect to Pt (i, j).
Is described as Pt '(i, j). Also, if necessary, the x and y coordinates of Pt (i, j)
(I, j) x and Pt (i, j) y.

In this step, the representative points are not limited to the lattice points, but may be arranged at arbitrary positions. In extreme cases, all pixels can be used as representative points.

(S11) Setting of Corresponding Point Candidate Area For example, in consideration of a representative point Pt (6, 4) shown in FIG. 3, an area where Pt ′ (6, 4) can exist is set in advance. This is based on the assumption that Pt '(6,4) is in the vicinity of Pt (6,4), except when the motion of the image is more abrupt than a certain limit. In the present embodiment, for example, Pt (6,
It is assumed that Pt ′ (6, 4) is included in the area of 100 × 60 pixels in the vicinity of 4), and the amount of calculation when Pt ′ (6, 4) is detected is reduced.

The following application is possible for this step.

1. When the image is moving relatively violently, t 'is determined so that frames t and t' are adjacent to each other. The change in the position of the representative point is suppressed to a minimum, and the possibility that the corresponding point does not enter the area is also minimized. However, as a matter of course, a method in which the candidate area is set to the entire screen in advance is also conceivable. In that case,
Although the amount of calculation increases, the possibility that a corresponding point is missed due to a large motion of the image is reduced.

2. In the present embodiment, it is simply assumed that Pt ′ (6, 4) is near Pt (6, 4).
When the movement trajectory in a plurality of frames of t (6, 4) is found, it is possible to determine a candidate area on an extension of this trajectory. If the motion of the video is constant to some extent, narrowing down the corresponding point candidate areas by this method is very effective.

(S12) Calculation of Dissimilarity in Corresponding Point Candidate Area Next, the position of the corresponding point is specifically determined from the candidate area. However, in this step, contrary to the previous step, a problem occurs when the motion of the video is too slow. This is because if the movement is small, it becomes difficult to extract the motion information, and the extracted information may include a large error.

In such a case, t 'is determined in advance so that the frames t and t' are separated to some extent. On this occasion,
The amount of change in each part of the image is statistically processed, and for example, t 'may be determined so that the variance of the amount of change exceeds a predetermined value. If t ′ that satisfies the condition is not found, the generation of the two-dimensional display image is stopped. In this case, the input video may be output as it is.

In this step, the dissimilarity is calculated by block matching between the frames t and t 'in order to determine the position of the corresponding point. The sum (dissimilarity) of density square errors between a neighboring block centered on a certain point in the corresponding point candidate area and a neighboring block of the representative point is obtained, and a point at which this is minimized is calculated. Determine the point.

FIG. 4 is a diagram showing the state of block matching. In this embodiment, nine pixels are defined as one block, and the central pixel is used as a representative point. Referring to FIG.
A block 1 containing Pt (i, j) is taken above, and a block 2 containing a corresponding point candidate Pt '(i, j) is taken on the frame t' side. Here, generally, if the pixel value of the pixel (x, y) at the time t is described as It (x, y), the dissimilarity (E1) is as follows.

E1 = ΣΣ ｛It (Pt (i, j) x + u, Pt (i, j) y + v) -It '(Pt' (i, j) x + u, Pt '(i, j) y + v)｝ ² (Equation 1) Here, two Σs relate to u and v. These take values u = -1,0,1 v = -1,0,1 respectively, and the sum of square errors for a total of 9 pixels can be calculated for the temporary Pt '(i, j). Then Pt '(i,
j) may be moved little by little in the candidate area, and a point at which E1 becomes minimum may be determined as a corresponding point.

FIG. 5 is a schematic diagram showing the value of E1 in the vertical direction for each Pt '(i, j). In this case, the point Q at which the dissimilarity has a steep peak is determined as the corresponding point. . Thereafter, corresponding points are similarly determined for other representative points.

This step has the following applications or modifications.

1. Here, the density 2
Although the squared error has been calculated, in the case of a color image, the sum of squared errors of each density of RGB, that is, E1 _R + E1 _G + E1 _B may be used as the dissimilarity. This is the density from other color spaces,
For example, it may be an HVC concentration. Instead of the square error, a simple absolute value of the error (sum of residuals) may be employed. 2. In this step, one block has 9 pixels, but it is usually desirable to define a block by a relatively large number of pixels. For example, assuming a high resolution screen of a normal personal computer or workstation,
As a result of the experiment, good results have been obtained with blocks of about 16 × 16 pixels.

(S13) Determination of Initial Position of Corresponding Point In the previous step, the corresponding point could be determined. However, at this stage, the position of the corresponding point is not always optimal. Although the corresponding points are relatively correctly determined for the boundaries and edges of the subject, for example, in an image portion with little change,
The position of the corresponding point should be considered to contain a considerable error. In FIG. 5, it may be said that E1 does not take a clear peak. FIG. 6 is a diagram showing the relationship between the corresponding points and the representative points obtained as a result of the previous step. As shown in FIG. 6, the corresponding points are satisfactorily obtained for characteristic points such as a house and a tree, especially their contours. However, the error is large for the sky and the ground.

Therefore, in this step and the next step, the position of the corresponding point is corrected. In this step, first, the concept of the initial position of the corresponding point is introduced, and the initial position is actually determined. In the next step, the position accuracy is improved by repeated calculation. The following policy can be considered for determining the corresponding point initial position in this step.

1. Treat all the corresponding points found in the previous step equally. The positions of all the corresponding points are input to the next step as their initial positions.

2. Provide a difference in handling of corresponding points The positions of corresponding points (hereinafter referred to as “feature points”) that seem to be in a correct position to some extent from the beginning are used as initial positions as they are, and corresponding points that are not (hereinafter referred to as “non-feature points”) Is determined based on that of the feature points. Here, the following points can be assumed as the feature points, but these points often coincide in reality.

(1) Corresponding point where E1 in the previous step shows a clear peak This is because the positional accuracy of such a corresponding point is generally high.

(2) Corresponding point at a position where many orthogonal edge components exist This is because the position of the corresponding point seems to be fairly correct in a portion such as a corner of a building. (3) Corresponding points whose positions are stably changing in frames t, t ',... Here, the stability of the change can be said to be the uniformity of the motion vector. Here, a corresponding point whose moving direction and moving distance are constant as the frame progresses is selected. Specifically, for example, a corresponding point whose variation of the motion vector is equal to or less than a predetermined value is selected. This is because such corresponding points should be accurately tracked, and it can be determined that they have a correct correspondence with the representative points. However, for example, when the camera that shoots the video moves irregularly, the determination is made in consideration of the influence.

When feature points are selected in this way, they can be used as initial positions as they are, and the initial positions of non-feature points can be determined by interpolating the positions of feature points or sequentially starting from the vicinity of feature points. That is, since the position accuracy of the non-feature point in the previous step is low, the initial position of the non-feature point is given geometrically from the highly accurate feature point. Of course, the method of the previous step can also be used effectively when finding the feature point of (3).

Although the method of determining the corresponding point initial position based on the selection of the feature point has been described above, the initial value of the corresponding point may be obtained by using a dynamic programming method (dynamic programming).

(S14) Corresponding Point Improvement Process An equation is introduced to evaluate the validity of the position of the corresponding point, and the position accuracy is improved by iterative calculation. In step S12, equation 1 for evaluating the degree of dissimilarity was introduced, but here, an equation for evaluating the validity of the relative positional relationship between corresponding points is introduced, and these two evaluation results are integrated to improve the position. Plan.

FIG. 7 is a diagram for explaining the principle of evaluating the relative position. In the figure, each point represents a corresponding point. Considering Pt '(i, j) in the figure, the following four corresponding points

## EQU2 ## Pt '(i-1, j), Pt' (i + 1, j), Pt '(i, j
-1) and Pt '(i, j + 1) are adjacent. Pt '(i, j)
Is usually near the center of gravity of these four points. This is based on an empirical rule that, even when each part of the image moves, the relative positional relationship is substantially maintained if viewed microscopically in pixel units. In mathematical terms, this property is nothing but that the second derivative of the function Pt '(i, j) of (i, j) is almost zero.

Therefore, the center of gravity of the above four points is defined as (St '(i, j) x, S
t '(i, j) y)

Equation 3] E2 = {Pt '(i, j) x-St' (i, j) x} 2 + {Pt '(i, j) y-St' (i, j) y} 2 ( Formula 2 ) Is the formula for evaluating the validity of the relative position. Considering only this equation, the position of the corresponding point becomes the most desirable state when E2 becomes the minimum value.

In this step, the evaluation results of Equations 1 and 2 are added with an appropriate coupling constant k, and E expressed by E = E1 / N + kE2 (Equation 3) is used as the final evaluation equation ( N is the number of pixels included in one block defined at the time of block matching). That is, first, E is calculated for each corresponding point, then the sum ΣE of E of all corresponding points is calculated, and the position of each corresponding point is changed little by little so that ΣE becomes the minimum value. The improvement processing is performed until the value of ΣE converges or until the repetition calculation reaches a predetermined upper limit number. More specifically, when changing the position of each corresponding point, one of the following methods may be performed.

(1) Method of Solving Euler's Equation Corresponding points are obtained by numerically solving the Euler's equation showing the condition that ΣE takes an extreme value (here, a minimal value). This technique itself is known. This is because the direction to be improved is found from the image inclination information in the block including each representative point and the pixel difference information between the corresponding blocks, and based on this, the position of the corresponding point is gradually moved from the initial position. Find a solution.

(2) Fixed Search Method First, in the corresponding point candidate area, the E of the corresponding point to be improved is
Find a point where is minimum, and use this as a new corresponding point. At this time, it is characterized in that the search is performed by regarding the positions of other points as immovable. This process is sequentially performed on all corresponding points.

(3) Mixing Method According to the method (2), the position of the corresponding point can be determined with an accuracy of a pixel unit. On the other hand, according to (1), the position can be theoretically obtained with an accuracy of a pixel unit or less. Therefore, it is also possible to first obtain the correspondence with the accuracy of the pixel unit by the method (2), and thereafter to improve the accuracy by applying the Euler equation.

According to the experiment, when compared with the same accuracy, a preferable solution was obtained in a shorter time than the method (1) by the method (2).

FIG. 8 is a diagram showing the result of performing the improvement processing of this step on the corresponding point candidates of FIG. According to an experiment, in the case of a color image, k was about 5 to 200, and a good result was obtained. FIG. 6 and FIG. 8 are schematic diagrams, but as a result of the experiment, improvements close to those in these figures were actually observed.

The above is the details of the step 1. This step has the following applications or modifications. 1. When deriving E2, consider not only the four points in the up, down, left, and right, but also the center of gravity of a total of eight points including four points in the oblique direction. Which combination is optimal depends on the type of video, so it is desirable to appropriately determine the combination by experiment.

2. The evaluation based on Expression 3 is preferentially performed from the corresponding point where the evaluation result based on only E1 is not good. This is because a corresponding point having a poor result of E1 is generally considered to have a large position error, and it is desirable to improve the position of such a corresponding point early and significantly.

3. Geometric information is also used for position improvement.
For a region having a geometric characteristic in the frame t, for example, a plurality of representative points that have formed a straight line, the positions are corrected so that their corresponding points also form a straight line. This is because a portion that appears to be a straight line on an image is likely to be a straight line even in the actual three-dimensional space, while a straight line in the three-dimensional space should be a straight line even in the frame t ′. Originally, the depth changes uniformly along the straight line, and the change along the straight line can be easily grasped visually, so that the improvement effect by this method is great. If such improvement is not made, the depth of the finally generated image will be uneven along the straight line, which may result in an unnatural image.

4. A corresponding point is obtained for another frame. In this step, the frame t 'with respect to the frame t
Are obtained, the corresponding points in the third frame t '' are also obtained, and the averaged motion of each part of the video can be obtained. This method does not improve the position of the corresponding point in the frame t '. By taking the corresponding points in many frames, the movement of each part of the video is statistically determined from the positions of the corresponding points and the time at which the frames were taken.

[Step 2] Calculation of Three-Dimensional Motion Information In step 1, two-dimensional motion of each part of the image on the screen was determined. In step 2, a three-dimensional movement of each part is calculated from this information. The image is obtained by projecting the actual movement of the subject on a plane, and in this step, the original movement is derived from the positional relationship between the representative point and the corresponding point.

In general, the motion of a subject in a three-dimensional space can be described as a combination of a translational motion and a rotational motion. Here, first, a calculation method in the case where the motion is composed of only translational motion will be described, and then a generalized method will be outlined later.

1. FIG. 9 is a diagram showing the correspondence between the movement of a certain point P on the screen and the actual movement in a three-dimensional space. In the figure, the two-dimensional coordinates on the screen are represented by capital letters X and the like, and the actual three-dimensional coordinates are represented by small letters x and the like. The x- and y-axes of the three-dimensional coordinates are represented on the screen, and the z-axis is represented in the depth direction. I have. Further, the distance from the viewpoint to the screen is set to 1.

As shown in this figure, P (X, Y) moves on the screen to P ′ (X ′, Y ′), and during this time, this point is S (x, y, z) in the three-dimensional space. To S (x ', y',
z '). here,

If (x ′, y ′, z ′) = (x, y, z) + (a, b, c), the distance to the screen is 1, so that X = x / z, Y = y / z X ′ = x ′ / z ′ and Y ′ = y ′ / z ′. If this is solved, X '= (Xz + a) / (z + c) Y' = (Yz + b) / (z + c), so z is eliminated and the following equation is obtained.

[0096]

(A−X ′ c) (Y′−Y) = (b−Y ′ c) (X′−X) (Equation 4) Since Equation 4 is represented by motion information on the screen, Step 1
The unknowns a, b, and c can be determined based on the information obtained in (1). However, in this case, in reality, in the case where an object having a size of k times moves at a speed of k times at a position separated by k times, the value of k (scale factor) cannot be determined. , c, it is possible to determine only their ratio. Mathematically speaking, even if three sets of (X, Y) and (X ', Y') are given, the rank (order) of the coefficient matrix when this simultaneous equation is represented as a matrix is at most 2; a, b, and c can be determined only as relative values. Therefore, in this step, a and b are temporarily represented by normalizing c = 1. This is because the process in the next step can be performed using only the ratio.

As another solution for the translational motion, the error e from equation 4 is given by

E = e (a-X′c) (Y′-Y) − (b-Y′c) (X′-X)｝ ² = ｛(Y′-Y) a- (X′- X) b− (XY′−X′Y) c｝ ² (Equation 5), and the sum Σe of e is obtained for all the correspondences between the representative points and the corresponding points, and a, b, c may be calculated from the following equation.

D (Σe) / da = 0 (formula 6) d (Σe) / db = 0 (formula 7) d (Σe) / dc = 0 (formula 8) More specifically, formulas 6 to 8 are Each is expanded to the following form.

[0099]

７ (Y′-Y) ² -bΣ (X'-X) (Y'-Y) -cΣ (Y'-Y) (XY'-X'Y) = 0 (Equation 9)- a Σ (X'-X) (Y'-Y) + bΣ (X'-X) ² + cΣ (X'-X) (XY'-X'Y) = 0 (Equation 10) -a Σ (Y '-Y) (XY'-X'Y) + b Σ (X'-X) (XY'-X'Y) + c Σ (XY'-X'Y) ² = 0 (Equation 11) It is an example of the calculation method regarding exercise.

2. When Motion Includes Rotational Motion Rotational motion can be described by three displacements in the x, y, and z directions and three angles of rotation about each axis, for example, α, β, γ. The rotation angle can be expressed by an Euler angle or a roll pitch method.

Here, the total of six variables may be determined. However, since the scale factor is not determined as described above, a certain variable is set to 1 and the ratio of each variable is determined.
Theoretically, the movement can be described if there are five sets of representative points and corresponding points.

It should be noted here that, depending on how the pairs are set, the state of the movement may not be obtained by the linear solution method. When such a case is considered, it is known that the number of sets should be eight or more. For the grounds that can describe the rotational motion by the linear solution from the eight sets of changes, see, for example, “About the linear solution of shape recognition by monocular stereoscopic vision from motion” (Exit / Akiba, Transactions of the Society of Instrument and Control Engineers vol.26,
No. 6, 714/720 (1990)).

[Step 3] The relative amount of the three-dimensional movement of each part of the image was found in step 2 of acquiring depth information. In step 3, depth information of each part is derived from the relative amount. In this process, for the sake of explanation, the subject is stationary,
Assume that the side of the camera that shoots it moves. Since the relative motion between the subject and the camera becomes a problem during image processing, a good result can be obtained by this assumption.

The motion of a certain part of the image is represented by (x ′, y ′, z ′) = R (x, y, z) + (a, b, c) using the rotation matrix R and the translation vector (a, b, c). c) this inverse transformation,

(X, y, z) = R ⁻¹ {(x ′, y ′, z ′) − (a, b, c)} (Equation 12) is considered as the motion of the camera.

FIG. 10 is a diagram for explaining the principle of deriving the three-dimensional coordinates of the point P from the three-dimensional movement of the camera and the movement of the point P on the screen. As can be seen from the figure, this principle is generally known as the principle of triangulation,
When the direction of the point P is viewed from the point, the actual position of the point P (point S in the figure) exists at the intersection of these two lines of sight.

In the figure, it is assumed that the camera has moved in accordance with Equation 12 as shown by the arrow between times t and t '. In frame t, point S is projected on point Pt, and in t ', point S is projected on point Pt'. Point S is located at the intersection of two straight lines Lt and Lt 'in the figure.

Under these conditions, the direction of the camera and Lt, Lt '
Since the angles θt and θt ′ are known and the moving direction and distance of the camera are known, the three-dimensional coordinates of the point S can be obtained. From these coordinates, the depth information of each part of the image is determined.

It should be noted that c = 1 as described above.
For this normalization, the obtained coordinates are also expanded or compressed at a constant rate. However, also in this case, the depth information is correct because the depth information is uniformly expanded and compressed.

The above is the outline of the present step. In this step, it is necessary to consider errors up to the previous step. Due to the error,
This is because Lt and Lt 'do not normally intersect in calculation. In consideration of such circumstances, in this step, the z coordinate of the midpoint between the closest points of both straight lines is approximated to the depth value of the point S. This will be described using mathematical expressions.

The direction vectors of Lt and Lt ′ are (u, v, w) and (u ′, v ′, w ′), respectively. Here, both straight lines are expressed by the real number parameters α and β as follows: Lt: (x, y, z) + α (u, v, w) Lt ′: (x ′, y ′, z ′) + β (u ′, v ′, w ′) (Equation 13). Therefore,

E = {(x + βu)-(x ′ + αu ′)} ² + {(y + βv)-(y ′ + αv ′)} ² + {(z + βw)-(z ′ + αw ′)} ^2, and α and β that minimize e are de / dα = 0, de /
It is determined from dβ = 0. That is,

(U ² + v ² + w ² ) α- (uu '+ vv' + ww ') β + (x-x') u + (y-y ') v + (z-z') w = 0 (u ' ² + v' ² + w ' ² ) β- (uu' + vv '+ ww') α + (x-x ') u' + (y-y ') v' + (z-z ') w' = 0 to obtain α and β, and finally the depth value of the point S,

[Equation 11] {(z + αw) + (z ′ + βw ′)} / 2 Here, if the error is 0, this coordinate coincides with the z coordinate of the intersection of both straight lines.

As another method, these two straight lines are once perspectively projected on the screen of frame t, and the projected closest point z is projected.
Coordinates can also be determined. Here, Lt is projected to one point which is a representative point, while Lt 'is generally projected to a straight line. If Lt ′ is expressed by Equation 13, the x, y coordinates of each point after projection can be obtained by dividing those of each point on Lt ′ by its z coordinate, as x = f (x ′ + βu ′). ) / (z ′ + βw ′) (Equation 14) y = f (y ′ + βv ′) / (z ′ + βw ′) (Equation 15) Here, f is the distance from the viewpoint to the screen of the frame t, and may be handled as f = 1 in practice. If β is eliminated from Expressions 14 and 15, a straight line after projection (hereinafter referred to as Li) is obtained as follows.

Kx + my + f n = 0 where k = v'z'-w'y ', m = w'x'-u'z', and n = u'y'-v'x '.

The closest point to be obtained is L from the representative point Pt.
It is the intersection (hereinafter referred to as D) of the perpendicular drawn to i and Li, and its coordinates are x = (m ² X-kn-kmY) / (k ² + m ² ) (Equation 16) y = (k ² Y-mn-kmX) / (k ² + m ² ). Here, a point on Lt ′ corresponding to point T is represented by E
If (x '', y '', z ''), the point E is obtained by substituting equation 16 into equation 14 to obtain β, and substituting this into the equation Lt '. Since β is β = (xz′−fx ′) / (fu′−xw ′), this is substituted into Expression 13 to obtain the z coordinate z ″ of the point E.
Is obtained as z '' = z '+ w'(xz'-fx') / (fu'-xw'). This may be used as the depth value of the point S.

When the depth becomes a negative value due to an error in image processing (when the point S is located behind the camera), the calculation result cannot be trusted. At this time, processing such as interpolation from a nearby representative point having a positive depth value is performed.

As described above, aside from which method is taken,
The obtained depth of each part of the image may be given as a numerical value for each representative point, for example. FIG. 11 is a diagram showing a state in which a numerical value is given to each representative point in a frame t.
t (2, 3) and Pt (4, 3) each have a depth of 10
0 and 200, indicating that the actual position of the latter is twice as far as the former.

[Step 4] Generation of Image An expected image is generated according to the depth information obtained in Step 3. In the steps so far, necessary information has been extracted from at least two video frames. However, in this step, a desired image can be obtained based on one video frame.

(1) Viewpoint change video FIGS. 12 and 13 are diagrams showing the correspondence between the original video and the viewpoint change video. FIG. 12 shows an original image, which includes image areas of “tree”, “house”, and “person” in order of decreasing depth. On the other hand, FIG. 13 shows a viewpoint change video. Here, a case where the viewpoint is virtually moved to the upper right is taken as an example.

As can be seen from these figures, according to the present invention, it is possible to obtain images from different viewpoints without actually moving the position of the camera. This is because the three-dimensional position coordinates including the depth information for each part of the image are grasped in step 3. Here, it is assumed that the viewpoint moves to the upper right, which is equivalent to thinking that the subject moves to the lower left. This movement to the lower left can be represented by rotation and translation, as described in step 3. Therefore, by following steps 1 to 3 of the present embodiment in the opposite direction, it is possible to reversely calculate the two-dimensional movement on the image from the virtual three-dimensional movement. As a result, an image shown in FIG. 13 is obtained. There is no room for human beings to arbitrarily place the images in the steps 1 to 4 and the obtained image is extremely natural.

In this step, it is desirable to generate an image in consideration of the shielding relationship. For example, in the case of FIG. 13, the lower end of the "tree" is hidden by the roof of the "house" as the viewpoint moves. Therefore, a natural image is generated by painting the lower end of the “tree” with the image data of the “house”. In the actual software processing, the images after the movement of the viewpoint may be generated in order from the video part having the largest depth. When it is desired to grasp the occlusion relationship by calculation, it is sufficient to determine whether or not the gaze vectors that look at each part of the video from a new viewpoint match. In the case where there are two parts A and B having the same gaze vector, if the part A is closer to the viewpoint than the part B, the part B is hidden by the part A. An image may be generated according to this information.

When a viewpoint-changed video is generated, it is natural that the shorter the virtual moving distance of the viewpoint is, the more accurate an image can be obtained. Utilizing this fact, the following applications can be considered for the viewpoint change video.

1.2 Generation of Multi-View Image from Eye Image When a stereo image is obtained by a two-eye camera,
A third camera is virtually provided to generate a multi-view video.
Since the interval between the twin-lens cameras is generally small, for example, a viewpoint-changed video is generated with a point forming a small triangle with them. This image is relatively accurate, and a good multi-view image can be formed by combining the image with the image obtained by the original twin-lens camera.

2. Generation of Slow Motion Video For example, two different time frames that are closest in time are frames t and t ′. The viewpoints at which these frames are photographed are referred to as viewpoints t and t ', respectively. The viewpoint is frame t, t '
Is actually moving from the viewpoint t to the viewpoint t 'during this period, but no image exists during this period. Therefore, a virtual viewpoint is provided between the viewpoints t and t ', and a viewpoint change video is generated. If a large number of viewpoint change images are provided and sequentially displayed, a slow motion image can be obtained. At this time, the following effects are obtained.

A. The motion of the video, which should be a frame-by-frame video, becomes extremely smooth. B. Very good images can be obtained since the viewpoint is generally not moved between frames that are close in time. C. By changing the movement path of the viewpoint between viewpoints t and t ',
Slow motion images of different images can be obtained.

Note that this technique can naturally be applied between frames at the same time.

(2) Partially Enlarged / Reduced Image One of the image display techniques is an emphasis display. As an example of the highlighting, there is a case where a closer subject is closer and a farther subject is farther away. This is for adding sharpness to the image.

In order to respond to such a request, the present invention performs partial enlargement / reduction of an image based on depth information. FIG.
Reference numeral 4 denotes an image generated by enlarging a part of the image in FIG. Here, the target of the enlargement processing is an area having a small depth, that is, a “person”. As a result, the "person" closest to the viewer is perceived further to the front, and effective highlighting can be performed. Here, as in (1), it is desirable to consider the shielding relationship.

It should be noted here that, as long as the region having the minimum depth is enlarged, the enlargement ratio is not limited in principle. This is because the region may be theoretically perceived at a distance of zero depth. On the other hand, when an area having an intermediate depth, for example, “house” in FIG. 14 is enlarged, it should not be perceived in front of “person”, so that a restriction naturally occurs. Failure to observe this restriction will result in very unnatural images. As in the present invention, when performing enlargement based on depth information, for example, it is possible to provide a rule that "only expands the area having the minimum depth and reduces only the area having the maximum depth". It is possible to obtain a natural image (an image according to the law of nature) that matches reality.

Although the generation of a natural image has been described here, there is also a demand for creating an unnatural image. For example, in an image of a game, an application in which a distant place is enlarged to create a kind of incongruous feeling can be considered. In any case, the feature of the present embodiment is that the selection of such a natural feeling or an unnatural feeling can be made consciously. In the conventional method, a situation in which a natural image is accidentally obtained when a certain area is enlarged tends to occur, but the present embodiment solves such a situation.

Assuming that a natural image is obtained as a result of this process, when further processing (1) or (3) described below is performed on this image, first, the depth of the enlarged or reduced area is changed. It is desirable. For example, the depth of a region that is doubled in length is reduced to about 1/2. Conversely, the depth of a region reduced to half the size is approximately doubled. This is due to the fact that the size and the depth of the region are almost inversely proportional. With this consideration, a natural image can be obtained even in the subsequent processing.

In the image processing of (1) and (2), a final image may be generated by correcting a step appearing in an image frame. For example, when generating the viewpoint change image of FIG. 13 from the original image of FIG. 12, generally, not all of the image parts included in FIG. 13 and those of FIG. 12 correspond completely one-to-one. The upper right end of FIG. 13 is farther from the upper right end of the “house” than the case of FIG. 12, and thus the image that is the source of the image near the upper right end of FIG. 13 is not included in FIG.
Therefore, when the video of FIG. 13 is simply generated from the video information included in FIG. 12, the vicinity of the upper right end is missing. This portion is depressed inward from the ideal image frame shown in FIG. For the same reason, the surplus video information included in FIG. 12 jumps out of the image frame of FIG.

Here, it is assumed here that the image frame (the outer shape of the image area, which is a rectangle in this case) is maintained as it is by allocating the pixels to the concave portions and cutting off the surplus pixels from the convex portions. The allocation is performed by, for example, pixels of the same color as the neighboring pixels. In the case of (2) partial enlargement / reduction, similar processing is performed because it is considered that each of the irregularities occurs. Through such processing, display of an unnatural image frame can be avoided.

(3) Image by cutting out An intended area is cut out to generate an image. For example, FIG.
, The depth of “person”, “house”, and “tree” is 3,
Let it be 10, 20 meters. Here, temporarily, "person"
When cutting out only, for example, "within 5 meters in depth"
In this case, the depth of each part may be searched and determined. In the case of "house", "5-15 meters in depth"
And so on.

FIG. 15 shows an image generated by cutting out “house” from the image of FIG. After the desired area is cut out, the other area can be blanked or the cut out area can be pasted on another image.

As described above, the present invention also provides a kind of image recognition / processing technique. Conventionally, an image area is cut out by a technique such as manual clustering or color clustering. However, the present invention provides a highly accurate area recognition technique from a new viewpoint of depth information.

The two-dimensional display image generation method according to the first embodiment has been described. According to the present invention, a new image processing technique is disclosed by deriving and using accurate depth information. Since the series of processes can be fully automated, for example, by software, the use of the present invention is wide.

Embodiment 2 FIG . In a second embodiment, an apparatus most suitable for implementing the first embodiment will be described.

FIG. 16 is a diagram showing an example of a hardware configuration for implementing the first embodiment.

In the figure, a video to be processed is input from a video input circuit 20, where it undergoes A / D conversion. The converted video is stored in the frame memory 24 by the frame memory control circuit 22. Frame memory 2
Subsequent to 4, a corresponding point detection circuit 26 that reads a plurality of video frames and detects corresponding points is provided. This circuit implements step 1 of the first embodiment by hardware, and for block matching, for example, an MPEG encoder circuit can be employed.

The coordinates of the corresponding points detected by the corresponding point detecting circuit 26 are temporarily stored in the corresponding point coordinate memory 28 and read out by the motion detecting circuit 30 as appropriate. The motion detection circuit 30 performs steps 2 and 3 of the first embodiment, and calculates a three-dimensional relative position of a subject from translation / rotational motion.

Subsequently, the three-dimensional position information is stored in the image generation circuit 3.
2 is input. This circuit reads an original video from the frame memory 24 and performs predetermined processing to generate an image. An instruction input unit 34 for receiving various instructions from the outside is provided at a stage preceding the image generation circuit 32.

The image thus generated is subjected to D / A conversion by the image output circuit 36 and output to a display device (not shown).

The operation of the present apparatus having the above configuration will be described.

First, an image captured by a camera or an image reproduced by a video reproducing apparatus is input from the image input circuit 20. This video is frame memory 2
4 is stored. With respect to a plurality of video frames read from the frame memory 24, depth information of the subject is obtained by the corresponding point detection circuit 26 and the motion detection circuit 30. Subsequently, the image generation circuit 32 generates an image such as a viewpoint change video according to the depth information. At this time, the viewpoint is changed, enlarged / reduced, and cut out based on the input from the instruction input unit 34 described above. These instructions can be realized by a known user interface.

The results of an experiment in which the present apparatus is incorporated into a workstation to determine depth information of a video will be described with reference to the drawings.

FIGS. 17 to 22 show a process of generating an image by the present apparatus, all of which are photographs of a halftone image displayed on a display, and are composed of about 640 × 480 pixel areas.

FIGS. 17 and 18 are images of frame t and frame t ', respectively, and the image slightly moves due to the movement of the camera. FIG. 19 shows a state in which a representative point is provided by dividing the frame t into meshes, and FIG. 20 shows an initial position of a corresponding point in the frame t '. Here, feature points are preferentially treated, and the best point obtained by performing block matching in a 16 × 16 pixel area centered on the representative point is set as the initial position.

FIG. 21 is a diagram showing Expression 3 of the first embodiment, that is, a result of improving the position of the corresponding point in consideration of the positional relationship of the corresponding points. The position is greatly improved as compared with FIG.

FIG. 22 shows the obtained depth information in light and shade, and the thinner the pixel, the smaller the depth. As can be seen from the figure, the depth information is obtained fairly accurately.

As described above, according to the present apparatus, the first embodiment can be carried out smoothly. At this time, if the execution time of the software for block matching is taken into consideration, the effect of improving the processing speed by performing this processing by hardware is significant.

As a mode of using the present apparatus as a product,
For example, a method of mounting the add-on card having the configuration of FIG. 16 in a personal computer or a workstation, a method of previously incorporating the configuration of FIG. 16 in a television receiver or a video playback device, and the like are effective. When this apparatus is combined with a camera, depth measurement using laser, infrared rays, ultrasonic waves, or the like, which is conventionally required, becomes unnecessary.

Embodiment 3 In the first and second embodiments, the input video is taken by the monocular camera. Here, a method of generating a two-dimensional display image when a stereo image by a multi-lens camera is used as an input image will be described focusing on differences from the first embodiment.

FIG. 23 shows main steps for generating a two-dimensional display image according to the third embodiment. FIG. 1 and Embodiment 1
The main differences from FIG. 1 are as follows.

1. "Motion information" in step 1 is "displacement information"
The first embodiment deals with a different time frame, but the third embodiment
So basically we deal with the same time frame. In the case of the same time, since movement cannot be defined for the subject, information on the displacement (displacement) of the position of the subject between frames at the same time is extracted instead.

[0154] 2. Step 2 becomes unnecessary There is no step corresponding to step 2 “calculation of three-dimensional motion information” in FIG. This is because, in the case of a multi-view, since the shooting is performed in the state of FIG. 10 from the beginning, depth information can be directly obtained by the principle of triangulation.

When using a multi-lens camera system in which the relative positional relationship between a plurality of cameras may be deviated, it is better to perform a self-calibration to correct the deviation. In this case, step 2 is a self-calibration step. For the self-calibration method,
For example, Tomita and Takahashi "Self-calibration of stereo camera" (Information Processing Vol. 31, No. 5 (1990) 650-65)
9), JP-A-63-293038, JP-A-63-293039 and the like.

Hereinafter, Steps 1 to 3 of Embodiment 3 will be described.

[Step 1] Extraction of Two-Dimensional Displacement Information In the description of the first embodiment, in addition to replacing “movement” with “displacement”, a set of frames t and t ′ may be replaced with frames 1 and 2. Frames 1 and 2 are camera 1,
2 indicates a video taken, and the shooting time is fixed at t. In the third embodiment, the final image can be obtained from at least these two frames. That is, in the case of multi-view photography, the input may be a still image. In addition, the differences from Step 1 of Embodiment 1 are as follows.

(1) In S11 (setting of corresponding point candidate area) in the first embodiment, selection of a different time frame or narrowing down the corresponding point candidate area is performed based on the intensity of the motion of the image or the movement trajectory of each part. The calculation amount of point detection processing was reduced. In the third embodiment, the narrowing-down method is changed as described below, and similarly, an effective calculation amount is reduced.

First, it is assumed that the multi-lens camera is installed horizontally as usual. At this time, the y-coordinates (vertical coordinates) of the corresponding points are substantially equal. Considering this assumption, an error associated with image processing, and a camera installation error, the corresponding point candidate area is limited to a horizontally long band-like area. Further, the frame t '
If the difference between the positions of the corresponding representative points at (t ′ = t−1) is x, the corresponding point search area in the frame t can also be limited to the neighborhood where the difference is also x.

(2) In S12 of the first embodiment (calculation of dissimilarity in the corresponding point candidate area), statistical processing is introduced when the movement of the video is too slow. However, in the third embodiment, this operation is unnecessary. .

(3) As in S12 of the first embodiment, in the third embodiment, block matching is performed in order to determine the position of the corresponding point. Here, it may be better to employ biased block matching. Biased block matching works effectively when the cameras constituting the multi-view camera have different characteristics. For example, if camera 2 transmits an image bluish than the camera 1, minus (i.e. predetermined color deflection constants alpha _B after subtracting a certain amount of blue components (B) from the color density of Frame 2 After), block matching should be performed. If such processing is not performed, the best matching may be missed. However, actually, for example, when the color density is represented by RGB, the color deflection constants α _R and α _G should be subtracted not only for the blue (B), but also for the red (R) and the green (G).

The biased block matching will be described using equations based on FIG. 4 and equation 1. Here, Pt (i, j) used in the first embodiment is simply expressed as P1 and P2 corresponding to frames 1 and 2, and It (i, j) is similarly expressed as I1 and I2. At this time, Equation 1 is

[Equation 12] E1 = {I1 (P1x + u, P1y + v) -I2 (P2x + u, P2y + v)} ² (Equation 18) This expression represents normal block matching for a grayscale image.

On the other hand, in biased block matching, Expression 18 is

E1 = {I1 (P1x + u, P1y + v) -I2 (P2x + u, P2y + v) -α} ² (Equation 19) In the case of a color image, α is α _R , α _G or α _B , and E obtained in each of the RGB images is
Matching is performed with the sum of 1, that is, E1 _R + E1 _G + E1 _B. Further considering the legibility, if I1 (P1x + u, P1y + v) is simply expressed as I1, and I2 (P2x + u, P2y + v) is simply expressed as I2,
Equation 19 is given by E1 = ΣΣ (I1-I2-α) ² (Equation 20). I1 and I2 are functions of u and v, while α is a constant.

Consider the optimal value of α. Since the cameras 1 and 2 should have photographed the same subject, the images of the frames 1 and 2 include substantially the same contents except for the displacement of each part of the image. That is, as the characteristics of the camera get closer, E
The value of 1 becomes smaller. Conversely from this fact, α should be a value that minimizes E1. Equation 20 is

Equation 14] is E1 = ΣΣ {(I1-I2 ) 2 -2 α (I1-I2) + α 2} = ΣΣ (I1-I2) 2 -2 αΣΣ (I1-I2) + ΣΣα 2 ( Equation 21), Assuming that the total number of pixels in the region is N, ΣΣ1 = N
So Equation 21 is

[Equation 15] E1 = {(I1-I2) ² −2 α} (I1-I2) + Nα ² (Equation 22) Therefore, since dE1 / dα = −2ΣΣ (I1−I2) + 2Nα, when α = {(I1−I2)} / N (Equation 23), E1 becomes the minimum. This α is rephrased as the average value of the color density difference of each pixel between the two regions to be subjected to block matching. Substituting Equation 23 into Equation 22 and calculating,

## EQU16 ## Since E1 = {(I1-I2) ² -{(I1-I2)} ² / N (Equation 24) In the end, Expression 24 may be calculated in biased block matching. After that, the best matching may be searched through the same processing as in the first embodiment.

Here, naturally, RG such as HVC concentration is used.
A density in a color space other than the B density may be adopted. Instead of the square error, block matching may be performed based on a general square error, that is, a color residual.

(4) In S13 of the first embodiment (determination of the initial position of the corresponding point), the different time frame t,
At t ',..., a point whose position is stably changed is selected, but the selection criterion is weighted here.

FIG. 24 is a diagram showing criteria for selecting feature points introduced in the third embodiment. In the figure, F10 to F1
3 are different time frames taken by the camera 1, and F20 to F22 are different time frames taken by the camera 2. Each set of two right and left pieces indicates the same time frame. Here, focusing on a certain point P, the movement of the position of the point P between different time frames is represented by
Are represented by a vector Bn (n: natural number).

Under the above settings, in Embodiment 3, points satisfying the following criteria are selected as feature points.

(A) Whether the vector Bn is substantially constant,
Alternatively, a criterion that (b) the vector An is almost constant or changes almost constant is added, and a point satisfying both (a) and (b) is selected as a feature point. Is also good.

(B) corresponds to the condition introduced in the first embodiment. As described above, in multi-view imaging, depth information can be obtained only from the same time frame. However, accurate understanding of the correspondence between videos as a premise is another problem, and information between frames at different times should be actively used. Points satisfying the above two conditions at the same time are considered to be tracked quite accurately, and thus provide important clues to extraction of two-dimensional displacement information. However, if the input is a still image, the corresponding point can be obtained by a known dynamic programming method (dynamic programming).

[Step 2] Acquisition of Depth Information Depth information of each part is derived from the displacement of each part of the video obtained in Step 1. In the case of multi-view, since the state of FIG. 10 is realized at a certain time t, depth information may be obtained by the method of step 3 of the first embodiment.

It should be noted here that since the positional relationship between the photographing cameras is fixed, if this relationship and the magnification (focal length) of the camera are known, the scale factor c, which is not determined in the first embodiment, is also determined. In the third embodiment, depth information is correctly obtained.

[Step 3] Image Generation Processing equivalent to step 4 (image generation) of the first embodiment may be performed.

The above is the outline of the third embodiment. In the case of the third embodiment, generally, depth information can be obtained with very high accuracy, and the accuracy of the final image, that is, the viewpoint change video, etc., also increases.

[0175]

According to the present invention, an image for two-dimensional display can be generated from the depth information of the original video, and the functional and technical restrictions which have been a problem can be solved. The input image may be either monocular or multi-view, and has a wide application range. In the case of a single eye, the system configuration can be simplified.

According to the present invention, depth information can be extracted from a two-dimensional displacement of each part of an image between a plurality of image frames.

In the case where a plurality of video frames are selected based on the two-dimensional displacement of the position, errors during the calculation are reduced.

When the two-dimensional position displacement is statistically processed, the selection can be optimized and the object can be made objective.

When a plurality of video frames are selected so that the variance of the motion vector of each part of the video becomes larger than a predetermined value, it is possible to guarantee the calculation accuracy in addition to the objectivity of the selection. If you can't choose to increase the variance,
Other displays, such as a normal video display, can be substituted.

When the relative positional relationship occupied by each part of the image in the actual three-dimensional space is derived from the two-dimensional position change, the depth can be obtained correctly.

When the position coordinates of each part of the image in the three-dimensional space are calculated from the three-dimensional movement of each part of the image by the principle of triangulation, the depth can be determined easily and accurately.

In the present invention, the concept of representative points and corresponding points is introduced, so that it is easy to grasp the motion information.

When the position of the corresponding point in another different time frame is predicted from the positional relationship between the representative point and the corresponding point, the calculation amount of the corresponding point search is reduced.

[0184] Of the representative points, those related to the image region having a geometric feature are maintained in the image region related to the corresponding point, so that the position accuracy of the corresponding point is high.

When a region including a straight line is employed as a geometrically characteristic image region, the positional accuracy of the corresponding point is improved.

When a point having a high degree of video similarity between a plurality of video frames is set as a corresponding point, the corresponding point can be correctly obtained.

When evaluating not only the similarity of the images but also the validity of the relative position between the corresponding points, the position accuracy of the corresponding points can be improved from the results of both evaluations.

When the results of the above evaluations are integrated and repeated calculations are performed, the positional accuracy of the corresponding points can be gradually improved.

When the similarity is evaluated by block matching, easy and appropriate evaluation can be performed.

In the block matching, since the n-th error of the color density is calculated, optimal conditions can be set by experiments or the like according to the video.

When biased block matching is performed, it is possible to generate a good two-dimensional display image especially for an input video image captured with multiple eyes.

In the case where the color deflection constant is determined so that the sum of the n-th power errors is minimized, the reliability of the evaluation result increases.

When the average value of the color density difference of each pixel between the image areas is used as the color deflection constant, the sum of the square errors is minimized, and the reliability of the evaluation result is improved.

When preferentially determining a corresponding point of a feature point,
The position accuracy of the corresponding point increases.

When a corresponding point that is a non-characteristic point is determined by interpolating a corresponding point that is a characteristic point, the positional accuracy of the corresponding point of the non-characteristic point necessarily increases.

When a feature point is determined by performing block matching between a plurality of video frames, it is possible to make the feature point selection objective and improve the validity of the selection result.

When a point at which a change in position occurs stably in a different time frame is set as a feature point, such point is tracked with high accuracy, and thus the position accuracy is high.

The displacement of the position between the same time frames is
In the case where a point is substantially constant or a feature point between frames at the same time captured at the near time, a two-dimensional display image can be satisfactorily generated from the multiview video.

When a viewpoint-changed image is generated according to the depth information, it is not necessary to actually move the camera. The generated image is also natural. It is also possible to generate a multi-viewpoint video from a twin-lens video and to create a smooth slow motion video.

When a part of an image is enlarged / reduced based on the depth information, the naturalness of the image is not lost.
Highlighting and the like can be performed. This is particularly effective when a region with a small depth is enlarged or a region with a large depth is reduced.

When correcting a step at an end of an image caused by these image processing, a natural image frame can be reproduced similarly to the original video.

In the case where a video area is cut out according to depth information, it is possible to cut out a considerably accurate cutout from a completely different point of view. When cutting out a part having a predetermined range of depth from each part of the image, image processing such as pasting of an image can be performed.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating main steps for generating a two-dimensional display image according to a first embodiment.

FIG. 2 is a flowchart for detecting a correspondence between video frames.

FIG. 3 is a diagram showing how a representative point is set in a reference frame t.

FIG. 4 is a diagram showing a state of block matching.

FIG. 5 is a schematic diagram showing the value of E in the vertical direction for each provisional corresponding point Pt ′ (i, j).

FIG. 6 is a diagram showing a relationship between corresponding points and representative points obtained as a result of step S12.

FIG. 7 is a diagram illustrating a principle of evaluating a relative position of a corresponding point.

8 is a diagram illustrating a result of performing the improvement processing of this step on the corresponding point candidates in FIG. 6;

FIG. 9 is a diagram showing a correspondence between a movement of a certain point P on the screen and a movement in a three-dimensional space.

FIG. 10 is a diagram illustrating the principle of deriving the three-dimensional coordinates of a point P from the three-dimensional movement of the camera and the movement of a point P on the screen.

FIG. 11 is a diagram showing a state where numerical values are given to respective representative points in a frame t.

FIG. 12 is a diagram showing a correspondence relationship between an original video and a viewpoint-changed video.

FIG. 13 is a diagram showing a correspondence relationship between an original video and a viewpoint-changed video.

FIG. 14 is a diagram illustrating an image generated by enlarging a part of the image.

FIG. 15 is a diagram showing an image generated by cutting out “house” from the image of FIG. 12;

FIG. 16 is a diagram illustrating an example of a hardware configuration for implementing the first embodiment.

FIG. 17 is a photograph of a halftone image in which a video image of a frame t is displayed on a display.

FIG. 18 is a photograph of a halftone image in which an image of a frame t ′ is displayed on a display.

FIG. 19 is a photograph of a halftone image in which a state in which a frame t is divided into meshes and representative points are provided is displayed on a display.

FIG. 20 is a photograph of a halftone image in which an initial position of a corresponding point in a frame t ′ is displayed on a display.

FIG. 21 is a photograph of a halftone image showing the result of improving the corresponding point position on a display.

FIG. 22 is a photograph of a halftone image in which depth information is displayed on a display in light and shade.

FIG. 23 is a diagram illustrating main steps for generating a two-dimensional display image according to the third embodiment.

FIG. 24 is a diagram showing a feature point selection criterion introduced in the third embodiment.

[Explanation of symbols]

20 video input circuit, 22 frame memory control circuit,
24 frame memory, 26 corresponding point detection circuit, 28
Corresponding point coordinate memory, 30 motion detection circuit, 32 image generation circuit, 34 instruction input unit, 36 image output circuit.

──────────────────────────────────────────────────続 き Continuation of the front page (72) Inventor Tsutomu Arakawa 2-5-5 Keihanhondori, Moriguchi-shi, Osaka Sanyo Electric Co., Ltd. (56) References JP-A-7-162744 (JP, A) JP-A-5-250459 (JP, A) JP-A-6-203163 (JP, A) JP-A-4-4476 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G06T 1 / 00 G06T 3/00-3/60 G06T 7 / 00,7 / 20

Claims (35)

(57) [Claims] 1. A two-dimensional display image generating method for extracting depth information of a video from an original video to be processed, and generating a two-dimensional display image according to the information. A two-dimensional position change detection step of detecting a two-dimensional position displacement in each other; a plurality of frame selection steps of selecting a plurality of predetermined frames equal to or less than the plurality of video frames; and the selected plurality of selected frames Deriving the relative positional relationship occupied in the actual three-dimensional space by each part of the image from the displacement of the two-dimensional position of each part of the image between each other,
A depth determination step of determining the depth according to the result, wherein the plurality of video frames are shot at different times, and in the plurality of frame selection step, the plurality of video frames are detected by the two-dimensional position change detection step. The two-dimensional position displacement between a plurality of predetermined frames is further obtained based on the position change amount between the video frames, and the change amount is compared with a predetermined value to specify the displacement amount from the predetermined plurality of frames. A two-dimensional display image generating method, wherein a plurality of video frames are selected. 2. A two-dimensional display image generating method for extracting depth information of an image from an original image to be processed and generating an image for two-dimensional display according to the information, comprising: A two-dimensional position change detection step of detecting a two-dimensional position displacement in each other; a plurality of frame selection steps of selecting a plurality of predetermined frames equal to or less than the plurality of video frames; and the selected plurality of selected frames Deriving the relative positional relationship occupied in the actual three-dimensional space by each part of the image from the displacement of the two-dimensional position of each part of the image between each other,
A depth determination step of determining the depth according to the result, wherein the plurality of video frames are shot at different times, and in the plurality of frame selection step, the plurality of video frames are detected by the two-dimensional position change detection step. Based on the amount of position change between the video frames, the amount of displacement of the two-dimensional position between a plurality of predetermined frames is further obtained, from the predetermined plurality of frames such that the average value of the amount of change is larger than a predetermined value. A two-dimensional display image generation method, wherein a plurality of specific video frames are selected. 3. A two-dimensional display image generating method for extracting depth information of an image from an original image to be processed and generating an image for two-dimensional display according to the information, the method comprising: A two-dimensional position change detection step of detecting a two-dimensional position displacement in each other; a plurality of frame selection steps of selecting a plurality of predetermined frames equal to or less than the plurality of video frames; and the selected plurality of selected frames Deriving the relative positional relationship occupied in the actual three-dimensional space by each part of the image from the displacement of the two-dimensional position of each part of the image between each other,
A depth determination step of determining the depth according to the result, wherein the plurality of video frames are shot at different times, and in the plurality of frame selection step, the plurality of video frames are detected by the two-dimensional position change detection step. The variance of a motion vector between a plurality of predetermined frames is determined based on the amount of position change between the plurality of video frames. A two-dimensional display image generation method, characterized by selecting: 4. The two-dimensional display image generating method according to claim 3, wherein when it is not possible to select a plurality of predetermined video frames such that the variance is larger than a predetermined value, the two-dimensional display image generation method is used. A two-dimensional display image generation method, which stops generating an image. 5. The two-dimensional display image generation method according to claim 1, wherein a three-dimensional movement of each part of the image is calculated from the displacement of the two-dimensional position. A two-dimensional display image generation method, wherein position coordinates of each part of the image in a three-dimensional space are calculated according to the principle of triangulation, and the depth is determined according to the result. 6. A two-dimensional display image generating method for extracting depth information of a video from an original video to be processed and generating an image for two-dimensional display according to the information. A two-dimensional position change detection step of detecting a two-dimensional position displacement in each other; a plurality of frame selection steps of selecting a plurality of predetermined frames equal to or less than the plurality of video frames; and the selected plurality of selected frames Deriving a relative positional relationship occupied in the actual three-dimensional space by each part of the image from the displacement of the two-dimensional position of each part of the image between each other,
A depth determination step of determining the depth according to the result, wherein the plurality of video frames are shot at different times, and in the plurality of frame selection step, the plurality of video frames are detected by the two-dimensional position change detection step. Based on the position change amount between the video frames, a displacement amount of a two-dimensional position between predetermined frames is further obtained, and the change amount is compared with a predetermined value to specify the displacement amount from the predetermined plurality of frames. A plurality of video frames, and in the depth determination step, among the predetermined plurality of frames, a representative point is set in a reference video frame, and a corresponding point of the representative point in another video frame is obtained. A secondary characteristic characterized by recognizing a two-dimensional position change of each part of the image by obtaining a positional relationship between a representative point and a corresponding point. Display image generation method. 7. The two-dimensional display image generation method according to claim 6, wherein the reference video frame and the other video frame are captured at different times.
A two-dimensional display image generation method, wherein a position of a corresponding point in another video frame is predicted from a positional relationship between the representative point and the corresponding point, and an area in which the corresponding point is searched is limited. 8. The two-dimensional display image generation method according to claim 6, wherein, among the representative points, those corresponding to a geometrically characteristic image area correspond to the representative points. A two-dimensional display image generation method, comprising: adjusting a position of the corresponding point so that the geometric feature is maintained even in a video area related to the point. 9. The two-dimensional display image generation method according to claim 8, wherein the geometrically distinctive video region is a region including a straight line. 10. The two-dimensional display image generation method according to claim 6, wherein a video area between a video area near a specific point in the another video frame and a video area near a representative point of the reference video frame is displayed. A two-dimensional display image generation method, wherein similarity is evaluated, and when the evaluation result is good, the specific point is determined as a corresponding point of the representative point. 11. The two-dimensional display image generation method according to claim 6, wherein a video area between a video area near a specific point in the other video frame and a video area near a representative point of the reference video frame is displayed. Two-dimensional display, wherein the similarity is evaluated, the validity of the relative position between the specific points is evaluated, and when the results of these two evaluations are good, the specific point is determined as the corresponding point of the representative point. Image generation method. 12. The two-dimensional display image generating method according to claim 11, wherein the results of the respective evaluations are digitized and integrated, and the numerical values are recalculated while changing the positions of the corresponding points, and are repeatedly calculated. A two-dimensional display image generation method, wherein the position accuracy of the corresponding point is improved. 13. The two-dimensional display image generation method according to claim 12, wherein the positions of all corresponding points are fixed once,
While moving only one corresponding point, a point where the result of each evaluation is the best is searched for, the position of the searched best point is set as a new position of the one corresponding point, and the search and change of the position are performed. A two-dimensional display image generation method, characterized in that the method is sequentially performed for all corresponding points. 14. The two-dimensional display image generating method according to claim 13, wherein after the search and the change of the position are performed for all the corresponding points, a numerical value of each integrated evaluation indicates an extreme value. A two-dimensional display image generation method, characterized by improving the positional accuracy of the corresponding point by numerically solving an equation. 15. The two-dimensional display image generation method according to claim 10, wherein the similarity is evaluated by block matching. 16. The two-dimensional display image generating method according to claim 15, wherein, in the block matching, a sum of n-th errors (n is 1 or 2) of color densities between video regions to be compared is calculated. A method for generating a two-dimensional display image, the method comprising: 17. The two-dimensional display image generation method according to claim 16, wherein the block matching is performed on the color density in consideration of a predetermined color deflection constant. Method. 18. The two-dimensional display image generation method according to claim 17, wherein the color deflection constant is determined so that the sum of the n-th power errors is minimized. 19. The two-dimensional display image generation method according to claim 18, wherein the color deflection constant is an average value of a color density difference of each pixel between the image areas, and the block matching is performed on the image area. A two-dimensional display image generating method, wherein the sum of squared errors is calculated after subtracting the color deflection constant from the color density difference of each pixel. 20. The two-dimensional display image generation method according to claim 10, wherein points included in the video are classified into feature points and non-feature points, and a representative point which is a feature point is preferentially given. A two-dimensional display image generation method characterized by determining corresponding points. 21. The two-dimensional display image generation method according to claim 20, wherein the corresponding points as the non-characteristic points are determined by interpolating the corresponding points as the characteristic points. Method. 22. The two-dimensional display image generation method according to claim 20, wherein block matching is performed between the reference video frame and the other video frame, and as a result, the correspondence with the representative point is good. A two-dimensional display image generation method, wherein a certain corresponding point is set as the feature point. 23. The two-dimensional display image generation method according to claim 20, wherein the characteristic point is a point whose position is stably changing in a plurality of video frames having different shooting times. 2D display image generation method. 24. The two-dimensional display image generation method according to claim 20, wherein the feature point is such that the displacement of the position between the video frames shot simultaneously is also determined between the video frames shot simultaneously at a time near the same. A two-dimensional display image generation method, wherein the two-dimensional display image generation point is substantially constant or a point that constantly changes. 25. The two-dimensional display image generation method according to claim 1, wherein a viewpoint-changed image to be obtained when a viewpoint of the image is virtually changed according to the depth information. A two-dimensional display image generation method characterized by the following. 26. The two-dimensional display image generating method according to claim 25, wherein the displacement of the position of each part of the image due to the assumed change of the viewpoint is calculated back from the depth information, and the image is reproduced according to the displacement of the position. A two-dimensional display image generation method, wherein the two-dimensional display image generation method comprises: 27. The two-dimensional display image generating method according to claim 25, wherein when the original video is captured by a twin-lens camera, the virtual image is located at a position sufficiently close to the two eyes. Assuming a camera, generate a video to be shot from this virtual camera as a viewpoint change video,
A two-dimensional display image generating method, wherein a multi-viewpoint image is generated from the viewpoint-changed image and a real image taken by the twin-lens camera. 28. The two-dimensional display image generation method according to claim 25, wherein a viewpoint of the video is virtually moved from a viewpoint at which a certain video frame is captured to a viewpoint at which another video frame is captured. A two-dimensional display image generation method, wherein a viewpoint-change video is generated with an arbitrary point on a moving route as a viewpoint. 29. A two-dimensional display image generating method for extracting depth information of an original video to be processed and generating an image for two-dimensional display according to the information, the method comprising: A two-dimensional position change detection step of detecting a two-dimensional position displacement in each other; a plurality of frame selection steps of selecting a plurality of predetermined frames equal to or less than the plurality of video frames; and the selected plurality of selected frames Deriving a relative positional relationship occupied in the actual three-dimensional space by each part of the image from the displacement of the two-dimensional position of each part of the image between each other,
And a depth determination step of determining the depth according to the result. Based on the position change amount between the video frames, the amount of displacement of the two-dimensional position between a plurality of predetermined frames is further obtained, and the predetermined amount of frames is selected by comparing the change amount with a predetermined value. A two-dimensional display image generating method. 30. The two-dimensional display image generation method according to claim 29, wherein an area having a small depth is selected as a partial area of the original video, and an image is generated after enlarging the area. Two-dimensional display image generation method. 31. The two-dimensional display image generation method according to claim 29, wherein a region having a large depth is selected as a partial region of the original video, and the region is reduced before generating an image. Two-dimensional display image generation method. 32. The two-dimensional display image generation method according to claim 25, wherein a step appearing in an image frame is corrected with image processing. 33. A two-dimensional display image generation method according to claim 1, wherein a desired video area is cut out to generate an image in accordance with the depth information. Method. 34. The two-dimensional display image generation method according to claim 33, wherein the clipping is performed by selecting a part having a predetermined range of depth from each part of the video. Method. 35. The two-dimensional display image generation method according to claim 33, wherein after the clipping,
A two-dimensional display image generation method, wherein a new image is generated by superimposing a cut-out video area on another video.