JP2002324249A - Image display system and its method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image display system and method capable of stereoscopically displaying the image of an object from an arbitrary point of view on a computer. SOLUTION: This image display system for automatically preparing the interposed image of an object from a point of view desired by a user is provided with a server having a means for fetching a two-dimensional image from a plurality of points of view of the object and a user terminal for transmitting and receiving data through a network with the server. The user terminal is provided with a means for receiving the two-dimensional image from the server, and for automatically preparing the interposed image of the object from the point of view desired by the user.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image display system and method capable of displaying an image of an object three-dimensionally from an arbitrary viewpoint on a computer.

[0002]

2. Description of the Related Art Conventionally, a method of displaying an image of an object (object) three-dimensionally on a computer is as follows: (1) A moving image of an object is photographed in advance and the moving image data is displayed. (2) By inputting three-dimensional data of an object in advance and specifying a viewpoint by a user, an object viewed from the viewpoint is calculated and displayed based on the three-dimensional data. (3) Automatically create and display an interpolated image at a viewpoint designated by the user from two images. Etc. exist as its main method.

[0003]

In the method (1), the surroundings, the whole view, and the like of an object are photographed in advance using a video camera or the like, and the image (moving image data) is recorded in an MPEG (Movi- ture) image.
In this method, data is compressed by a known compression method such as NG Pictures Experts Group, and the image of the target object is displayed on a computer by reproducing the data using software corresponding to the moving image data. With this method, a person who wants to view an image of the object (hereinafter, user) cannot arbitrarily change the viewpoint of the object. This is because, in this method, an image of the object photographed in advance is simply reproduced on a terminal (hereinafter, a user terminal) of the user. That is, the user cannot browse images from viewpoints that have not been photographed in advance.

Further, there is a problem in that if images from all viewpoints are to be stored as moving image data, the amount of data becomes large. This means that a great deal of communication time is required when a user tries to browse this moving image data via a network. That is, the communication cost is expended in proportion to the data amount, which imposes a great burden on the user. Furthermore, it is necessary to transfer a large amount of data from a computer storing moving image data to a user terminal, thereby increasing network traffic. In other words, traffic is increased for other users who do not browse the moving image data but use some of the same network, and there is a problem that response is reduced.

As one method for solving the problem in (1), the method (2) has been devised. In this method, a point in a three-dimensional space where an object is present (not necessarily on the object) is set as an origin in the three-dimensional space, and a distance from the origin or three-dimensional coordinates (x-coordinate, By inputting (y-coordinate, z-coordinate), three-dimensional shape data of the object is created. After that, the distance from the viewpoint designated by the user, coordinates, etc. are calculated, and projected and displayed on a two-dimensional plane, so that the image of the object from the viewpoint designated by the user can be browsed. It is.

[0006] However, this method is very complicated, for example, in order to create three-dimensional shape data of an object, the length, width, height, etc. of the object must be accurately measured and input. This imposes a great burden on the data creator. Furthermore, there is a problem that it is very difficult to create three-dimensional shape data from originally two-dimensional plane data (for example, a photograph).

In order to further solve these problems, two-dimensional plane data of an object from two viewpoints (that is, plane data (for example, photographs) of an object from two directions) is input in advance, A method of automatically creating an interpolated image between the viewpoints (refers to an image created automatically to interpolate the viewpoint from the original image) from the two-dimensional plane data of the viewpoint and displaying it. It has been devised.

In this method, eight or more sets of corresponding points in two-dimensional plane data (hereinafter, referred to as an image) from two viewpoints and an arbitrary number of line segments (hereinafter, referred to as feature lines) representing features of the image are designated. And interpolating the viewpoint between the two images to display the interpolated image, an example of which is disclosed in Japanese Patent Application No. 2000-24.
3032 publication. Using this method, unlike conventional image interpolation, it is possible to create an interpolated image based on projective geometric constraints of the viewpoint.

However, the above-mentioned invention has the following problems. (1) When creating an interpolated image, the number of combinations of feature lines and all pixels in the image becomes enormous, and a great deal of calculation time is required. (2) It is difficult to set the parameter of the influence range of the characteristic line, and an unnatural interpolated image may be created. (3) As disclosed in Japanese Patent Application No. 2000-243032, in image interpolation from two viewpoints, even if a plurality of combinations of two images are prepared, the viewpoint movement can be realized only in one dimension. And the viewpoint cannot be freely moved in the space.

[0010]

SUMMARY OF THE INVENTION In view of the above problems, the present inventor has solved the problem by employing the following method. (1) A common plane is defined for a set of images, only the vertices of the plane are interpolated by a parallelized image (described later), and the pixels inside the plane are displayed by mapping the texture of the linearly interpolated plane. (2) It is possible to change the viewpoint from the conventional two viewpoints to the three viewpoints. In this case, the viewpoint can be moved two-dimensionally. That is, if the set of three viewpoints is set to all directions, the image of the object can be viewed from any viewpoint desired by the user.

According to a first aspect of the present invention, there is provided an image display system for automatically creating an interpolated image of an object from a viewpoint desired by a user, wherein the image capturing system captures a two-dimensional image of the object from a plurality of viewpoints. In an image display system having a user terminal capable of transmitting and receiving data via a network with a server having means, the user terminal receives the two-dimensional image from the server and receives an object from a viewpoint desired by the user. An image display system having an interpolated image creating means for automatically creating an interpolated image of an object.

The invention according to claim 33 is an image display method for automatically creating an interpolated image of an object from a viewpoint desired by a user in a user terminal capable of transmitting and receiving data via a server and a network. And the user terminal
An image display method for automatically creating an interpolated image of an object from a viewpoint desired by the user by receiving the two-dimensional image from the server.

An invention according to claim 65 is an image display system for automatically creating an interpolated image on a server for automatically creating and browsing an interpolated image of an object from a viewpoint desired by a user, An image display system comprising: an image capturing unit that captures a two-dimensional image of the object from a plurality of viewpoints; and an interpolated image creating unit that automatically creates an interpolated image of the object from the desired viewpoint. It is.

An invention according to claim 94 is an image display method for automatically creating an interpolated image on a server for automatically creating and browsing an interpolated image of an object from a viewpoint desired by a user, The server is an image display method for acquiring a two-dimensional image of the object from a plurality of viewpoints and automatically creating an interpolated image of the object from the desired viewpoint.

According to the first and third aspects of the present invention,
From the two-dimensional image captured in advance on the server, an interpolated image of the viewpoint desired by the user can be automatically created on the user terminal.

According to the inventions of claims 65 and 94, the two-dimensional image previously captured on the server is transferred to the same server (however, logically the same terminal,
The terminals need not be physically the same. In the same manner, the interpolated image of the desired viewpoint can be automatically created.

According to a second aspect of the present invention, there is provided an image display system for automatically creating an interpolated image of an object from a viewpoint desired by a user, wherein the interpolated image of the object from a viewpoint desired by the user is automatically generated. In an image display system having a server capable of transmitting and receiving data via a network and a user terminal having an interpolated image creating means for creating, the server captures a two-dimensional image of the object from a plurality of viewpoints. An image display system having image capturing means for transmitting a two-dimensional image to the user terminal.

According to a thirty-fourth aspect of the present invention, in a server capable of transmitting and receiving data to and from a user terminal via a network, an interpolated image of an object from a viewpoint desired by the user is automatically created at the user terminal. An image display method in which the server acquires a two-dimensional image of the object from a plurality of viewpoints and transmits the two-dimensional image to the user terminal.

According to the invention of claims 2 and 34,
It is possible to transmit a two-dimensional image captured in advance on the server to the user terminal and automatically create an interpolated image of a viewpoint desired by the user on the user terminal.

According to a third aspect of the present invention, in the image display device, the image capturing means defines a common vertex and a plane connecting the vertex with a line for the captured two-dimensional image set. System.

According to a thirty-fifth aspect of the present invention, the server captures a plurality of the two-dimensional images before transmitting the two-dimensional image to the user terminal, and common to the set of the captured two-dimensional images. This is an image display method for defining vertices and a plane connecting the vertices with a line.

The invention according to claim 95 is an image display method in which the server defines a common vertex and a plane connecting the vertex with a line for the set of the captured two-dimensional images.

According to the invention of claims 3 and 35,
By defining a surface, it is possible to create an interpolated image by mapping the texture of the surface without performing the image morphing conventionally performed. According to the inventions of claims 66 and 95, it is possible to define these aspects on the same server.

According to a fourth aspect of the present invention, the image capturing means defines a common vertex and a plane connecting the vertex with a line for the set of the captured two-dimensional images. Extracting the edge endpoint of the object, designating a Delaunay triangle designation point in the captured two-dimensional image set, and extracting a Delaunay triangle in one two-dimensional image of the two-dimensional image set. Generate and correct the Epipole constraint from the Delaunay triangle specified point, and perform the corrected linear search to obtain a common vertex and a surface connecting the vertex with a line for the set of two-dimensional images. This is an image display system that is automatically defined.

According to a thirty-sixth aspect of the present invention, in the server, when defining a common vertex and a plane connecting the vertex with a line for the captured two-dimensional image set, Extracting an edge end point of the object, designating a Delaunay triangle designation point in the set of the captured two-dimensional images, and generating a Delaunay triangle in one two-dimensional image of the set of the two-dimensional images; By performing a correction that satisfies the epipole constraint from the Delaunay triangle designation point and performing the corrected linear search, a common vertex and a surface connecting the vertex with a line are automatically set for the set of the two-dimensional images. This is the image display method to be defined.

According to the inventions of claims 4, 36, 67, and 96, the human (image creator) himself / herself can be provided as in the inventions of claims 3, 35, 66, and 95. Does not define both common vertices and the surface connecting the vertices, if they are specified in one image, they will be automatically defined in the other image, Can be reduced.

According to a fifth and a 68th aspect of the present invention, the image capturing means assumes a virtual three-dimensional space coordinate system on the server and polar coordinates of a sphere centered on the virtual three-dimensional space coordinate system origin. An image display system that sets a viewpoint of the two-dimensional image by associating a movement amount of an input device with the server.

According to the inventions of claims 37 and 97, when the server takes in the two-dimensional image, the server assumes a virtual three-dimensional space coordinate system on the server and sets the virtual three-dimensional space coordinate system origin. By associating the polar coordinates of the center sphere with the amount of movement of the input device in the server,
This is an image display method for setting a viewpoint of the two-dimensional image.

According to the fifth, 37th, 68th, and 97th aspects of the present invention, the three-dimensional
A two-dimensional viewpoint can be set only by moving an input device such as a mouse two-dimensionally, so that the burden of capturing a two-dimensional image can be reduced.

According to a sixth aspect of the present invention, the interpolated image creating means receives the two-dimensional image from the server and performs a predetermined operation on the received two-dimensional image, An image display system comprising: a parallelized image creating unit that automatically creates a parallelized image based on the calculation result; and a texture mapping unit that performs texture mapping on the parallelized image.

According to a thirty-eighth aspect of the present invention, the user terminal comprises:
When automatically creating an interpolated image of the object from the viewpoint desired by the user, the user receives the two-dimensional image from the server and performs a predetermined operation on the received two-dimensional image. This is an image display method for automatically creating a parallelized image based on a result and performing texture mapping on the parallelized image.

The invention of claim 69 is characterized in that the interpolation image creation means performs initial information calculation means for performing a predetermined calculation on the two-dimensional image, and automatically creates a parallelized image based on the calculation result. An image display system comprising: a parallelized image generating unit that performs texture mapping; and a texture mapping unit that performs texture mapping on the parallelized image.

The invention according to claim 98, wherein the server performs a predetermined operation on the two-dimensional image when automatically creating an interpolated image of the object from a viewpoint desired by the user, An image display method for automatically creating a parallelized image based on the calculation result and performing texture mapping on the parallelized image.

According to the invention of claims 6 and 38,
Interpolated images can be created by mapping surface textures without performing conventional image morphing.
According to the inventions of claims 69 and 98, the creation of an interpolated image based on texture mapping can be performed on the same server as the server that imported the original two-dimensional image. .

The invention according to claim 7 and claim 70 is an image display system wherein the initial information calculation means includes a calculation of a basic matrix and a calculation of epipole for the set of two-dimensional images as the predetermined calculation. is there.

The invention according to claims 39 and 99 is an image display method including a basic matrix operation and an epipole operation on the two-dimensional image set as the predetermined operation.

In a preferred embodiment of the present invention, the texture mapping means interpolates vertices of the surface defined in the server into the automatically created parallelized image, performs linear interpolation, and calculates the linearly interpolated surface. This is an image display system that texture-maps pixels inside a surface inside vertices.

According to a 40th aspect of the present invention, when the texture mapping is performed, the vertices of the surface defined in the server are interpolated into the automatically created parallelized image, linear interpolation is performed, and the linear interpolation is performed. This is an image display method in which pixels inside the surface are texture-mapped inside the vertices of the surface.

The invention according to claim 71, wherein the texture mapping means interpolates the vertices of the defined surface into the automatically created parallelized image, performs linear interpolation, and sets the surface inside the vertices of the linearly interpolated surface. This is an image display system that texture maps internal pixels.

According to a 100th aspect of the present invention, when the texture mapping is performed, the vertices of the defined surface are interpolated into the automatically created parallelized image, and linear interpolation is performed. This is an image display method for texture mapping pixels inside the surface.

Conventionally, at the time of image morphing, calculation with a characteristic line is performed for all pixels, and it is desired to spend a great deal of time on the processing time. Interpolated images can be created. Also, according to the inventions of claims 71 and 100, it is possible to perform these on the same server as the server that has captured the two-dimensional image.

A ninth aspect of the present invention is the image display system, wherein the texture mapping means further includes a face selecting means for selecting a texture image for all or some of the defined faces.

The invention according to claim 41 is an image display method which further includes, when performing the texture mapping, a process of selecting a texture image for all or some of the defined surfaces.

The invention of claim 72 is an image display system wherein said texture mapping means further comprises a face selecting means for selecting a texture image for all or some of the defined faces.

The invention according to claim 101 is an image display method for further performing a process of selecting a texture image for all or some of the defined surfaces when performing the texture mapping.

In a tenth aspect and a thirty-third aspect of the invention, when only one surface of the surface is facing up to a viewpoint designated by the user, the surface selecting means sets the surface as a texture image. If two or more faces are facing up, the area of the facing face is calculated and the face having the larger area is selected as a texture image, and / or image A
Is A, the brightness of the image B is B, and the ratio of the distance from the image B to the viewpoint at the viewpoint between the image A and the image B is d (0 ≦ d ≦ 1). , S = A × d + B
This is an image display system that selects a luminance S of an image calculated by (1-d) as a texture image.

The invention of claims 42 and 102 is characterized in that, in the process of selecting the texture image, if only one of the surfaces is facing up with respect to the viewpoint specified by the user, Select the face as the texture image,
When the above-mentioned surface is facing up, the area of the facing surface is calculated and the surface having the larger area is selected as a texture image, and / or the brightness of the image A is A, the brightness of the image B is B, When the ratio of the distance from the image B to the viewpoint at the viewpoint between the image A and the image B is d (0 ≦ d ≦ 1), S = A × d + B (1-d). This is an image display method in which the luminance S of the image to be selected is selected as a texture image.

According to the inventions of claims 10, 42, 73 and 102, the image quality of a texture image can be improved, and a more natural interpolated image can be used as a texture.

The eleventh aspect and the seventy-fourth aspect are the image display system in which the plane selecting means draws the image in order from a plane far from the epipole.

The invention according to claims 43 and 103 is an image display method in which, in the process of selecting the texture image, the image is drawn in order from the surface farther from the epipole.

In the inventions of claims 10, 42, 73 and 102, an image to be pasted as a texture is always selected from one of the images. For this reason, when the viewpoint is moved, the image pasted as the texture may be frequently switched, which is not preferable in appearance. However, according to the invention of claim 11, claim 43, claim 74, and claim 103, Switching between images during movement has been eliminated, and more natural and simple rendering of interpolated images has become possible.

The invention according to claims 12 and 75 is an image display system wherein said texture mapping means further comprises contour processing means for performing contour processing on said surface.

The invention according to claims 44 and 104 is an image display method for further performing contour processing of the surface when performing the texture mapping.

According to a thirteenth aspect of the present invention, the contour processing means sets a point at which a vertex coordinate of the surface is averaged as a center of gravity of the surface and passes through the center of gravity of the surface and is perpendicular to a ridge line which is a connection between surface vertices. Calculate the vertex offset vector, which is the average unit vector of the boundary ridge offset vector including the vertex, by calculating a ridge line offset vector, which is a unit outward vector, for each surface, and averaging the calculated ridge line offset vectors. The contour processing is performed by displaying an offset surface composed of the vertices at both ends of the boundary ridge line and vertices moved in the offset vector direction from the created interpolated image, and then displaying a normal surface. An image display system.

The invention according to claims 45 and 105 is characterized in that, in the contour processing, a point at which the coordinates of the vertices of the surface are averaged is set as the center of gravity of the surface, and the ridge line which passes through the center of gravity of the surface and is a connection between the surface vertices Calculate a vertex offset vector, which is an average unit vector of a boundary ridge offset vector including a vertex, by calculating a ridge offset vector that is a vertical outward unit vector for each face, and averaging the calculated ridge offset vectors. Then, for the created interpolated image, an offset surface composed of both end vertices of the boundary ridge line and vertices moved in the direction of the offset vector therefrom is displayed, and thereafter, a normal surface is displayed, thereby performing contour processing. This is an image display method to be performed.

According to the inventions of claims 13, 45, 76 and 105, the contour of the texture image can be sharpened. For example, it is difficult to show the contour of a stuffed toy with a straight line. The contour of a certain object can be sharpened.

The invention according to claims 14 and 77 is an image display system wherein the texture mapping means further comprises a color correction means for performing a color correction process on the surface.

The invention according to claims 46 and 106 is an image display method for further performing a color correction process on said surface when performing said texture mapping.

According to a fifteenth aspect and a thirty-eighth aspect of the present invention, the color correction means includes the steps of:
Assuming that the color of No. 1 is C ₀ (p) and C ₁ (p) and the weight of the ridge line is w, the color to be corrected is calculated by calculating the color C (p) to be corrected by Equation 1 to correct the color. An image display system.

The invention of claim 47 and claim 107 is
When performing the color correction process, the text corresponding to the pixel point p
Color of image 0,1 is C ₀(P), C₁(P)
Assuming that the weight is w, the color C (p) to be corrected is calculated by Equation 1.
Image table to calculate the color to be corrected by performing
It is a method of showing.

According to the inventions of claims 15, 47, 78 and 107, a color difference may occur when a different texture image is selected for each adjacent face, but this can be solved. Becomes

The invention according to claims 16 and 79 is characterized in that, in the case where three sets of the two-dimensional images are set as one set in the image capturing means, the final interpolated image from the viewpoint desired by the user. Automatically creating a primary interpolated image by using a combination of the two images in the set of the captured two-dimensional images, and automatically creating a primary interpolated image in the set. Automatically creating another primary interpolation image using two combinations other than the combination of the two images obtained from the viewpoint from the viewpoint desired by the user using the two primary interpolation images created automatically. This is an image display system that automatically creates a final interpolated image.

In the invention according to claim 48, when three sets of the two-dimensional images are set in the server, when the user automatically creates the final interpolated image from the viewpoint desired by the user, The user terminal automatically creates a primary interpolation image using the combination of the two images in the set of the captured two-dimensional images, and outputs two images other than the combination of the two images in the set. Image display for automatically creating another primary interpolation image using a combination of the above, and automatically creating a final interpolation image from a viewpoint desired by the user using the two primary interpolation images created automatically. Is the way.

The present invention according to claim 108, in the case where the three two-dimensional images are set as one set in the server, when the user automatically creates a final interpolated image from a viewpoint desired by the user, A primary interpolated image is automatically created using a combination of two images in the set of the captured two-dimensional images, and a combination of two images other than the combination of the two images in the set is used. Automatically create another primary interpolation image by using
This is an image display method for automatically creating a final interpolated image from a viewpoint desired by the user using one primary interpolated image.

According to the sixteenth and forty-eighth aspects, two-dimensional viewpoint movement is possible. Claim 79
In addition, according to the invention of Claim 108, it is possible to create a final interpolated image on the same server as the server that has imported the two-dimensional image, and it is possible to move the viewpoint in two dimensions.

According to a seventeenth aspect of the present invention, in the case where three sets of the two-dimensional images are set as one set in the image capturing means, the interpolated image from the viewpoint desired by the user is converted to a set. When automatically creating, the interpolated image creating means calculates a rotation matrix using the three captured two-dimensional images, performs parallelization rotation conversion using the rotation matrix, and performs the parallelization rotation conversion. Performs a horizontal rotation conversion based on the horizontal rotation conversion, performs a scale conversion based on the horizontal rotation rotation conversion, interpolates the homography matrix based on the parallelization rotation conversion, the horizontal rotation conversion, and the scale conversion, and automatically interpolates the image. This is an image display system to be created.

According to the invention of claims 49 and 109, when three sets of the two-dimensional images are set as one set in the server, an interpolated image from a viewpoint desired by the user is automatically created. At this time, the user terminal calculates a rotation matrix using the three captured two-dimensional images, performs a parallelization rotation conversion using the rotation matrix, and performs a horizontal rotation conversion based on the parallelization rotation conversion. Perform the scale conversion based on the horizontal rotation conversion, the parallelization rotation conversion, the horizontal rotation conversion, interpolate the homography matrix based on the scale conversion, the image display method to automatically create an interpolation image is there.

In the inventions of claims 16, 48, 80 and 109, in order to enable two-dimensional viewpoint movement, two of the three are set as one set of three. Two sets of sheets were created, a primary interpolation image was created from them, and then a final interpolation image was created from the two primary interpolation images. Therefore, when creating from three images, a two-step process is performed. However, going through the two-step process results in an increase in calculation time, image distortion, and the like. However, according to the seventeenth and forty-ninth aspects of the present invention, it is possible to create an interpolated image from three images at a time while maintaining the viewpoint movement in two dimensions, thereby shortening the calculation time and distorting the image. Can be prevented.

[0069] The invention of claim 18 is a communication system, wherein the user terminal comprises:
An image display system further comprising a shadow creating means for creating a shadow on the created interpolated image.

The invention according to claim 50 is characterized in that the user terminal comprises:
An image display method for performing a process of creating a shadow on the created interpolation image after creating the interpolation image.

The invention according to claim 81 is an image display system wherein said server further comprises a shadow creating means for creating a shadow on said created interpolated image.

The invention according to claim 110 is an image display method in which the server creates a shadow on the created interpolation image after creating the interpolation image.

According to the inventions of claims 18 and 50, it is possible to add a shadow to the created interpolation image, and to create a more realistic interpolation image on the user terminal. Become. According to the inventions of claims 81 and 110, it is possible to add a shadow to the created interpolated image, and a more realistic interpolated image is placed on the same server as the server that imported the two-dimensional image. It can be created at

The invention according to claims 19 and 82 is an image display system wherein the shadow creating means creates a shadow for an object using an affine transformation matrix.

The invention of claims 51 and 111 is an image display method in which the process of creating a shadow creates a shadow on an object using an affine transformation matrix.

According to a twentieth aspect, the user terminal comprises:
An image display system further comprising reflection creating means for creating a reflection on the created interpolation image.

[0077] The invention according to claim 52 is that the user terminal comprises:
This is an image display method that further performs a process of creating a reflection on the created interpolation image after creating the interpolation image.

The invention according to claim 83 is an image display system in which the server further comprises reflection creating means for creating a reflection on the created interpolation image.

The invention according to claim 112 is an image display method in which the server further performs a process of creating a reflection on the created interpolation image after creating the interpolation image.

According to the twentieth and fifty-second aspects of the present invention, it is possible to reflect the created interpolated image, and to create a more realistic interpolated image on the user terminal. Become. Further, according to the inventions of claims 83 and 112, it is possible to add reflection to the created interpolation image, so that the interpolation image with more reality is placed on the same server as the server that imported the two-dimensional image. It can be created at

The invention according to claims 21 and 84 is an image display system wherein the reflection creating means creates a reflection for an object using an affine transformation matrix.

The invention of claims 53 and 113 is an image display method in which the processing of creating a reflection creates an reflection on an object using an affine transformation matrix.

According to a twenty-second aspect of the present invention, the user terminal includes:
Combination display means for combining at least one or more objects with one or more images from the same viewpoint by restoring the three-dimensional information from at least one or more interpolated images, shadows, reflections, and / or two-dimensional images. This is an image display system further provided.

The invention of claim 54 is that the user terminal comprises:
By restoring the three-dimensional information from at least one or more interpolated images, shadows, reflections, and / or two-dimensional images, a process of combining at least one or more objects with one or more images from the same viewpoint is further performed. This is an image display method.

[0085] The server according to claim 85, wherein the server restores at least one or more objects by identifying the three-dimensional information from at least one or more interpolated images, shadows, reflections, and / or two-dimensional images. An image display system further comprising a combination display unit for combining one or more images from a viewpoint.

The invention according to claim 114, wherein the server restores at least one or more objects by identifying the three-dimensional information from at least one or more interpolated images, shadows, reflections, and / or two-dimensional images. This is an image display method that further performs a process of synthesizing one or more images from a viewpoint.

According to the inventions of claims 22 and 54, it is possible to combine an image of an object with another two-dimensional image. According to the inventions of claims 85 and 114, it is possible to combine an image of a target object with another two-dimensional image on the same server as the server that has captured the two-dimensional image.

The invention according to claims 23 and 86 is an image display system using projection restoration or Euclidean restoration as the restoration.

The invention of claims 55 and 115 is an image display method using projection restoration or Euclidean restoration as said restoration.

The invention according to claims 24 and 87, wherein the server further comprises an automatic continuation means for setting a viewpoint path and a moving speed when performing the automatic continuous display of the interpolated image. System.

The invention according to claims 56 and 116 is an image display method wherein said server further performs an automatic continuous process for setting a viewpoint path and a moving speed when performing said automatic continuous display of said interpolated image. .

According to the twenty-fourth, fifty-sixth, eighty-seventh, and sixteenth aspects of the present invention, it is possible to perform continuous display with more flexibility when performing automatic continuous display of interpolated images.

According to a twenty-fifth aspect of the present invention, in the automatic continuation means, a virtual three-dimensional space coordinate system is assumed on the server, and a polar coordinate of a sphere centered on the virtual three-dimensional space coordinate system origin. An image display system for setting a route by associating the amount of movement of an input device with the server.

According to the inventions of claims 57 and 117, in performing the automatic continuous processing, a virtual three-dimensional space coordinate system is assumed on the server, and the virtual three-dimensional space coordinate system origin is set as a center. This is an image display method for setting a route by associating the polar coordinates of a sphere with the movement amount of an input device in the server.

According to the invention of claims 25, 57, 88 and 117, it is possible to easily set the route of the viewpoint on the server by using an input device such as a mouse.

[0096] According to a twenty-sixth aspect, the user terminal comprises:
An image display system further comprising automatic continuation means for setting a path and a moving speed of a viewpoint when performing automatic continuous display of the interpolated image.

The invention according to claim 58 is characterized in that the user terminal comprises:
This is an image display method that further performs automatic continuous processing for setting a viewpoint route and a moving speed when performing the automatic continuous display of the interpolated image.

According to the twenty-sixth and fifty-eighth aspects, it is possible to perform continuous display with more flexibility when performing automatic continuous display of interpolated images.

The invention according to claim 27 is characterized in that said automatic continuation means assumes a virtual three-dimensional space coordinate system on said user terminal, and polar coordinates of a sphere centered on said virtual three-dimensional space coordinate system origin and said user terminal. Is an image display system for setting a route by associating the amount of movement of the input device in the above.

The invention of claim 59 is characterized in that when performing said automatic continuous processing, a virtual three-dimensional space coordinate system is assumed on said user terminal, and polar coordinates of a sphere centered on said virtual three-dimensional space coordinate system origin. And an amount of movement of the input device at the user terminal.

According to the twenty-seventh and fifty-ninth aspects, it is possible to easily set a viewpoint path using an input device such as a mouse on a user terminal.

The invention according to claims 28 and 89 is an image display system wherein said automatic continuation means sets a moving speed on said set path by associating a moving amount of said input device with time. is there.

In the invention of claims 60 and 118, when the automatic continuous processing is performed, the moving speed of the set path is set by associating the moving amount of the input device with time. This is an image display method.

According to the inventions of claims 28, 60, 89 and 118, the moving speed of the viewpoint can be set only by moving an input device such as a mouse without inputting the moving speed of the viewpoint. Becomes possible.

[0105] The invention of claims 29 and 90 is an image display system wherein said automatic continuation means sets information indicating a range of an object in said two-dimensional image.

The invention of claims 61 and 119 is an image display method for setting information indicating a range of a target in the two-dimensional image when performing the automatic continuous processing.

According to the inventions of claims 29, 61, 90 and 119, the position and size can be corrected.

[0108] According to a thirtieth aspect of the present invention, the user terminal comprises:
An image display system further comprising automatic continuous creation means for automatically creating and displaying an interpolated image continuously based on a route and a moving speed set on the server or the user terminal.

[0109] The invention of claim 62 is that the user terminal comprises:
An image display method that further performs an automatic continuous reproduction process for automatically creating and displaying an interpolated image continuously based on a route and a moving speed set on the server or the user terminal.

The invention according to claim 91 is an image display system wherein said server further comprises automatic continuous creation means for automatically creating and displaying an interpolated image continuously based on the set route and moving speed. is there.

The invention according to claim 120 is an image display method in which the server further performs an automatic continuous reproduction process for automatically creating and displaying an interpolated image continuously based on the set route and moving speed.

According to the inventions of claims 30 and 62, it is possible to perform automatic continuous display on the user terminal based on the setting of automatic continuous display set on the server / user terminal. Claims 91 and 120
According to the invention, automatic continuous display can be performed on the same server as the server on which automatic continuous display is set.

[0113] According to a thirty-first aspect, the user terminal comprises:
An image display system further comprising a correction processing unit that calculates an enlargement / reduction magnification and a parallel movement amount based on information indicating a range of the target object, and corrects a position and a size of the target object.

[0114] In the invention according to claim 63, the user terminal comprises:
An image display method for calculating an enlargement / reduction magnification and a translation amount based on information indicating a range of the object and performing a position and size correction process on the object.

In a ninth aspect of the present invention, the server calculates an enlargement / reduction magnification and a translation amount based on the information indicating the range of the object, and performs a position and size correction process on the object. The image display system further includes a correction processing unit for performing the correction.

The invention of claim 121, wherein the user terminal calculates an enlargement / reduction magnification and a translation amount based on the information indicating the range of the object, and corrects the position and size of the object with respect to the object. Is an image display method for performing the following.

The invention according to claims 32 and 93 is an image display system in which the correction processing means calculates a range of an interpolated image of the object by Equation 2 and corrects the interpolated image within the range. .

According to the invention of claims 64 and 122, when performing the correction processing, the image display for calculating the range of the interpolated image of the object by the formula 2 and correcting the interpolated image within the range is provided. Is the way.

[0119]

DESCRIPTION OF THE PREFERRED EMBODIMENTS An example of a system configuration of an image display system 1 according to the present invention will be described in detail with reference to the system configuration diagrams shown in FIGS.

A user terminal 15 and a server 14 of a user who wants to view an interpolated image of an object from an arbitrary viewpoint are an image display system 1 capable of transmitting and receiving data via a network 16. . Here, the network 16 is an open network such as the Internet, or a LAN (Local Area Network).
k) or an intranet which is a combination thereof. When data is transmitted and received between the server 14 and the user terminal 15 via the network 16, it is preferable to use a known data transfer technique. Further, in this embodiment, for convenience of explanation, one user terminal 15 and one server 14 are assumed and described, but it goes without saying that the functions may be distributed to a plurality of terminals. Further, all functions may be realized only by the server 14 (or the user terminal 15) without passing through the network 16.

The server 14 has the image capturing means 2, and the user terminal 15 has the interpolated image creating means 3, the shadow creating means 4,
It has a reflection creating means 5, a created image storing means 6, a combining display means 7, and a correction processing means 8.

The image capturing means 2 is a means for capturing a two-dimensional image (basic image) obtained by photographing an object from a plurality of viewpoints, and is a means for creating information for image display. It is also means for transmitting the fetched information to the user terminal 15 via the network 16.

The interpolated image creating means 3 is means for creating an interpolated image based on the basic image and the set viewpoint.
It has initial information calculation means 9, parallelized image creation means 10, and texture mapping means 19.

The initial information calculation means 9 is a means for calculating a basic matrix and an epipole in each basic image set required to create an interpolated image.

The parallelized image creating means 10 calculates a homography matrix by using the epipole calculated by the initial information calculating means 9 so that the images between the two viewpoints are parallel and the common points are arranged on the same scanning line. This is a means for mapping an image on a plane.

The texture mapping means 19 interpolates the vertices of the plane defined by the server 14 into the parallelized image created by the parallelized image creating means 10,
This is a means for performing linear interpolation and performing texture mapping of pixels inside the surface inside the vertices of the linearly interpolated surface, and includes a surface selecting means 11, a contour processing means 12, and a color correcting means 13.

The plane selecting means 11 selects an image to be used as a texture from two images, or selects one of the images and uses it as it is, but combines the two images to create a more natural image. Means used as a texture. The process of selecting an image to be used as a texture from two images and the process of creating a more natural image by combining the two images and using the image as a texture will be described later. Further, when occlusion occurs, it is not possible to determine the front and back of the surface, and therefore, drawing may be performed in order from a far image by using a reference (ie, epipole) corresponding to the distance from the viewpoint. Thereby, it is possible to more easily create an interpolated image. The process in this case will also be described later.

The contour processing means 12 is a means for reflecting an image of the details of a contour on an interpolated image when defining a plane for an image of an object. By this means, the object is displayed as a clearer image.

The color correcting means 13 is a means for correcting the color of the image applied to the interpolated image as a texture. By this means, the color difference between the textures of the object is eliminated.

The shadow creating means 4 is a means for adding a shadow to the object. By this means, the reality of the image is improved.

The reflection creating means 5 is a means for creating a state in which an object is reflected and reflected on another object. By this means, the reality of the image is improved.

The created image storage means 6 is a means for temporarily storing information created and calculated by the interpolated image creating means 3, the shadow creating means 4, and the reflection creating means 5.

The combining display means 7 is a means for combining the interpolated image, shadow, reflection, and the like stored in the created image storage means 6 on one image.

The correction processing means 8 is means for performing linear interpolation of the position and size of the object for each viewpoint. This is because, in the basic image, the distance from the camera to the object and the position of the object in the image may be different for each viewpoint. There is a problem that the displacement of the object other than the above occurs, the object appears to be enlarged / reduced, and the object moves up and down without rotating around a certain point. It is.

[0135]

Embodiment 1 An example of the process of the present invention is shown in FIGS.
This will be described in detail with reference to the flowchart shown in FIG. In the present embodiment, a process of creating an interpolated image (I _s ) at a viewpoint desired by the user from a two-dimensional image (basic image) (I ₀ , I ₁ ) from two viewpoints of the object will be described. I do.

The person who creates the image (image creator) performs a predetermined procedure from the server 14 of the image creator (for example, activation of a computer, activation of software, activation of a peripheral device, etc.), and acquires the image. From 2, a basic image from a plurality of viewpoints of the object is captured (S100). Assuming that the object is located in a virtual three-dimensional space, the position of the viewpoint at which each basic image is photographed is adjusted. The process of capturing the basic image will be described later.

In the basic images at a plurality of viewpoints of the object created in S100, two basic images are set as one set, and coordinates of positions common to the basic images in the set, that is, Eight or more common point coordinates are defined (S160).
An interpolation image is created for the selected coordinates in a subsequent process. However, the common point coordinates must be corresponding points such that any three points will not be aligned on a straight line. Further, it is preferable to define the common point coordinates using an input device (not shown) such as a mouse, but other means may be used.

[0138] Corresponding points are selected from the set of basic images, and these points are connected by a straight line to define a plane (ie, define plane definition coordinates). This is performed for the entire object (S170). FIG. 20 shows an example of a plane definition screen for defining a plane. At this time, the common point coordinates defined in S160 and the plane definition coordinates which are points defining the plane may be the same or different.

Further, when a corresponding point is selected from this basic image, a point can be selected in one of the basic images even if a human does not perform the operation as described above. And the corresponding point may be automatically selected. This process will be described later.

After the surface definition is completed, S100 to S170
Is transmitted from the image capturing means 2 of the server 14 to the interpolated image creating means 3 of the user terminal 15.

The user specifies a viewpoint desired to be browsed by using a mouse or the like (S176). The interpolated image creating means 3 creates an image from this viewpoint specified by the user.

Thereafter, an interpolated image is created based on the common point information defined by the image capturing means 2 and the plane definition coordinates (S180). When creating an interpolated image, first, initial information (basic matrix and epipole) is calculated by the initial information calculating means 9 of the interpolated image creating means 3. The process will be described below.

FIG. 36 shows the positional relationship of each coordinate system between the two viewpoints. The two viewpoints are C ₀ and C ₁ , and the basic images from the two viewpoints are I ₀ and I ₁ , respectively. 3 of basic images I ₀ and I ₁
Let the base matrix of × 3 be F ₀ and F ₁ respectively.

To calculate the basic matrix, a well-known method (for example, by RH Hartley,
"A defense on the 8-point algorithm" (In Defense of t)
heEight-point Algorithm), Vol.19, No.6, p.580-59
3, 1997.) (S19).
0).

After the basic matrix is calculated in S190, a known method (for example, ODFaugeras, Te Papadopro, etc.) is used based on the calculated basic matrix.
(T. Papadopoulo), A Nonlinear Method for Estimating the
Projective Geometry of Three Views), p.1-27, 199
The epipole of each basic image is calculated according to (7) (S20).
0). Base image _{I 0} and _{I 1 e 0} the epipole of the basic image _{I 0} for the epipole of the basic image _{I 1} and _{e 1.} Epipole e ₀ is an eigenvector corresponding to the smallest eigenvalue of F ₀ ^T F ₀ , and epipole e ₁ is F ₀ F ₀ ^T
Is the eigenvector corresponding to the smallest eigenvalue of

After calculating the basic matrices and epipoles in the basic images I ₀ and I ₁ , if the user requests viewing of the object from the specified viewpoint S from the user terminal 15, interpolation between the two viewpoints In order to create an image, a parallelized image (described later) is created based on the information captured by the image capturing unit 2 and the information calculated by the initial information calculating unit 9 (S210).

A conventional interpolation image creation method (for example, Japanese Patent Application No.
In the example shown in Japanese Patent Application Laid-Open No. 00-243032, an interpolated image is created through the following process. (1) A basic matrix and an epipole are calculated from given common points of eight or more. (2) A homography matrix H ₀ that performs parallel image conversion so that the epipole is at infinity and common points are arranged on the same scanning line,
Determine the H _1. (3) base image _I 0, the end point of the matrix in characteristic lines _{I 1} ^{H _-1 0, H} _-1 collimated by _first image _{I ^} 0, _I ^ ₁
To a point. (4) The end point of the converted feature line is interpolated into the ratio s between viewpoints. (5) Basic images I ₀ , I ₁ and parallelized images I ＾ ₀ , I ＾ ₁
Calculating a matrix H _s to convert the collimated image to the image coordinates from the interpolation of the four corners of the points. (6) Image morphing for mixing two basic images is performed based on the influence function defined for the interpolated feature line.

However, in the case of the method of creating an interpolated image performed through such a process, since the distance is calculated for all combinations of all pixels and all characteristic lines to calculate the influence function, the calculation time is reduced. Takes very much. In this case, the parallelized image is generally a greatly distorted image, and the influence range of the feature line greatly differs depending on the direction.

In the case of gazing at a single object as in online shopping, since the depth from the viewpoint is a close value, distortion inside the object is small and can be approximated almost linearly. Therefore, in the present invention, instead of defining a characteristic line, a surface is defined (a surface definition coordinate is defined), and only vertices of the surface are interpolated on the parallelized image, and then an image inside the surface is texture-mapped. indicate. That is, instead of performing image morphing, in (4), a homography matrix H _s (described later) is applied by interpolating a vertex of a surface instead of an end point of a characteristic line, and interpolation is performed by texture-mapping a basic image. Get an image. Since the surface is displayed in two dimensions, the texture mapping is not a mapping from a normal three-dimensional space to a two-dimensional space, but a mapping between two-dimensional spaces. Further, since the inside of the plane is linearly interpolated, unnatural distortion unlike the case of using image morphing does not occur.

In order to create an interpolated image from two basic images, first, the two basic images are imaged such that common points (surface definition coordinates determined in S170) are aligned on the same scanning line. This process is generally called parallelization, and the created image is called a parallelized image. In order to perform the parallelization, the two basic images are parallelized in the parallelized image creating unit 10 of the interpolated image creating unit 3 using the epipole calculated in S200 (that is, the parallelization is performed). A parallelized image is created) (S210). The process of parallelizing two images will be described later.

[0151] In S210, after creating the parallelizing two images of the basic image, by interpolating the only surface defining the coordinates are the vertices of the defined surface (S220), the basic image by applying the homography matrix H _s Is interpolated by texture mapping (S230). The texture mapping process will be described later.

In S200, the interpolation image creating means 3
An interpolated image (Is) is created from the two basic images (I ₀ and I ₁ ).

In this embodiment, since two viewpoints constitute one set, the created interpolated image is transmitted to the created image storage means 6 as the final interpolated image (final interpolated image) (Is). Store temporarily. 2 shown in FIGS. 34 (a) and 34 (b)
FIG. 35 shows an example of an interpolated image created from the basic images.

Next, in order to give the created final interpolated image a reality, a shadow for the object is created by the shadow creating means 4.
In step S240, the created shadow is transmitted to the created image storage unit 6 and temporarily stored. The process of creating a shadow will be described later. The reflection for the object is created by the reflection creation means 5 (S250), and the created reflection is transmitted to the created image storage means 6 and temporarily stored. The process of creating the reflection will be described later.
The order of creating shadows and reflections in S240 and S250 is random.

Created in S180 to S250,
A final interpolated image temporarily stored in the created image storage means 6,
The shadow and reflection are transmitted to the combination display means 7, and these are combined into one image (S260). The method of combining in S260 can be realized by combining and displaying a plurality of objects from the same viewpoint by "restoring" the three-dimensional information from a plurality of images of the objects. This is realized by using a known technique of "projection restoration" or "Euclidean restoration" as a technique of "restoration". By adding this combination process, for example, as shown in FIG. 29, a plurality of objects such as a table and a chair are combined and displayed, or a photograph of a room at home is taken,
Combine furniture images and visually check the harmony of size, color tone, etc., save data of your own full body image, and combine fashion product images and hairstyle images with it. To browse (virtual fitting system, virtual beauty salon).

After the shadows, reflections, and interpolated images are synthesized and displayed in S260, correction processing relating to the position and size is executed (S270). This is because in the basic image captured in S100, the distance from the camera to the object and the position of the object in the basic image generally differ from viewpoint to viewpoint. If interpolation images are continuously created while these are different from each other, displacement of the object other than the movement of the viewpoint occurs, and the object is apparently enlarged / reduced or the object is displayed at a fixed point. This is to solve the problem of moving up and down without rotating around.

In order to prevent this, the present invention solves this problem by adding a process for correcting the distance (= enlargement / reduction magnification) and position of the object in the basic image from the camera. . The position and size correction processing will be described later.

In step S270, after the position and size correction processing is completed, the created interpolated image is displayed. The correction of the shadow, reflection, combined display, position, and size in S240 to S270 may or may not be performed entirely or partially.

[0159]

Embodiment 2 Another embodiment of the present invention will be described in detail with reference to the system configuration diagrams of FIGS. In the present embodiment, an interpolated image (I _s ) at a viewpoint desired by the user is created from a two-dimensional image (basic image) (I ₀ , I ₁ , I ₂ ) of the object from three viewpoints. Describe the process. The description of the same parts as those of the above embodiment will be omitted.

The image creator fetches at least three or more basic images of the object from the image fetching means 2 on the server 14 (S100). In the captured basic image, three basic images are set as one set (in the present embodiment, one set of three images is taken as an example. Eight or more coordinates of positions common to the basic images in the group, that is, common point coordinates are defined (S160). However, the common point coordinates must be such that even if three points are arbitrarily taken, they do not line up on a straight line. Further, it is preferable to define the common point coordinates using an input device (not shown) such as a mouse, but other means may be used.

Here, one set of three basic images is
Conventionally, an interpolated image was created using two basic images as a set.
However, by making three or more basic images into one set,
Previously, the viewpoints at which interpolation images could be created were limited to one dimension.
The problem can be solved. That is, three basic images
By using one set, the viewpoint that can create an interpolated image is secondary
It can be enlarged to the original, and the set of these three basic images
By creating a number, an interpolated image can be created from any viewpoint.
It becomes possible. Interpolation in three viewpoints
To create an image, you first need to create two basic
Image (I₀, I ₁) Primary interpolation image (I_s0Create)
Then, another two basic images (I₀, I
₂) Primary interpolation image (I_s2) Created, two created
Interpolation image (I_s0, I_s2) To final viewpoint interpolation
Image (final interpolation image) (I_s) To create. Where primary
An interpolated image is temporarily used to create the final interpolated image.
5 shows an interpolation image to be created. As shown in FIG.
Inset image (I_s), The basic image (I₀, I₁)
Image (I_s0), Basic image (I₀, I₂)
Image (I_s2FIG.

The corresponding points are selected from the set of basic images, and these points are connected by a straight line to define a plane (ie, define plane definition coordinates). This is performed for the entire object (S170). FIG. 20 shows an example of a plane definition screen for defining a plane. Steps S160 to S170 are executed for at least one or more basic image combinations. At this time, the common point coordinates defined in S160 may be the same as or different from the plane definition coordinates that define the plane.

Further, in S170, as in the first embodiment, when selecting a corresponding point from the basic image, the point is selected in one of the basic images even if no human operation is performed as described above. If so, a point corresponding to one of the basic images may be automatically selected. This process will be described later.

After the surface definition for at least one or more basic image combinations is completed, the information in S100 to S170 is transmitted from the image capturing means 2 to the interpolated image generating means 3.

Next, the viewpoint which the user desires to browse is specified by using a mouse or the like (S176). Thereafter, an interpolated image is created based on the common point information defined by the image capturing means 2 and the plane definition coordinates (S180). First, initial information (basic matrix and epipole) necessary for creating an interpolated image is calculated by the initial information calculating means 9. The process will be described below.

FIG. 18 shows the positional relationship of each coordinate system among three viewpoints.
21. C from 3 viewpoints₀, C₁, C₂From three perspectives
Each basic image is I₀, I₁, I₂And Basic image I₀
And I ₁, Basic image I₁And I₂, Basic image I₂And I₀3
Let each of the × 3 basic matrices be F₀, F₁, F₂And

To calculate the basic matrix, the basic images I ₀ and I ₁ , the basic images I ₁ and I ₂ (or the basic images I ₀ and I 1)
_1, there is a need to calculate the two base matrix from the basic image I ₂ and I _0), which can be calculated by using a known method (S190).

Since the calculation of this basic matrix can be performed by the same process, in the example of the present embodiment in which three basic images are set as one set, it is necessary to repeat this process three times.

After calculating all the basic matrices in S190, the epipole of each basic image is calculated by a known method based on the calculated basic matrices (S200). Base image _{I 0} and _{I 1 e 00} a epipole of the basic image _{I 0} for the epipole of the basic image _{I 1} and _{e 11.} Similarly, for the basic images I ₁ and I ₂ , the epipole of the basic image I ₁ is e ₁₀ , the epipole of the basic image I ₂ is e ₂₁ , and for the basic images I ₂ and I ₀ , the epipole of the basic image I ₂ is e ₂₀ . Let the epipole of I _{0 be} e ₀₁ . Epipole _{e 00} is an eigenvector corresponding to the minimum eigenvalue of _{F ⁰} T _F ^0, epipole _{e 11} is an eigenvector corresponding to the smallest eigenvalue of the _F ⁰ _F 0 ^T. The same can be calculated for other epipoles. S190 and S2
The relationship between the basic matrix calculated in 00 and the epipole is not independent, and the constraint shown in Expression 3 holds.

Equation 3] ^{_{e 10 T F 0 e 01 =}} e 20 T F 1 e 11 = e
₀₀ ^T F ₂ e ₂₁ = 0

[0170] The basic image _{I 0, I 1,} after calculating the in base matrix and epipole in _{I 2,} in order to create an interpolation image _{I s,} first three basic images _{(I 0, I 1, I} 2 primary in interpolated image of two from) (I _s0, create a _{I s2),} the user from the primary in interpolated two images to create it to produce the final in interpolated image of the viewpoint to the desired (I _s). The interpolation image creation means 3
This is a means for creating one interpolated image from two images (basic image or interpolated image). This is because, as shown in FIG. 22, two interpolated images are created from three basic images, and a final interpolated image of a viewpoint desired by the user is re-created from the two interpolated images created. It is.

In order to create an interpolated image from two basic images, first, two basic images are imaged such that common points (surface definition coordinates determined in S170) are aligned on the same scanning line. This process is generally called parallelization, and the created image is called a parallelized image. In order to perform the parallelization, the two basic images are parallelized in the parallelized image creating unit 10 of the interpolated image creating unit 3 using the epipole calculated in S200 (that is, the parallelization is performed). A parallelized image is created) (S210). The process of parallelizing two images will be described later.

After creating a parallelized image of the two basic images in S210, the texture mapping means 19 interpolates only the surface definition coordinates which are vertices of the defined surface (S2).
20), it will create an interpolation image by texture mapping the basic image by applying the homography matrix H _s (S230). The texture mapping process will be described later.

In the present embodiment, since three viewpoints constitute one set, a primary interpolation image (I _s0 ) is created from two basic images (I ₀ , I ₁ ). In a similar process, two primary images (I ₀ , I ₂ ) are converted into another primary interpolation image (I _s2 ).
Create That is, S180 is performed twice.

When two primary interpolated images (I _s0 , I _s2 ) are created, a final interpolated image (final interpolated image) is formed from the two primary interpolated images by a process similar to S180. ) (I _s ) (ie, S18 at this stage)
The 0 interpolation image is created based on the two primary interpolation images created earlier). The created final interpolated image (I _s ) is transmitted to the created image storage means 6 and temporarily stored. That is, S180 is performed three times in total.

Next, in order to give the created final interpolated image a reality, a shadow on the object is created by the shadow creating means 4.
In step S240, the created shadow is transmitted to the created image storage unit 6 and temporarily stored. The process of creating a shadow will be described later. The reflection for the object is created by the reflection creation means 5 (S250), and the created reflection is transmitted to the created image storage means 6 and temporarily stored. The process of creating the reflection will be described later.
The order of creating shadows and reflections in S240 and S250 is random.

Created in S180 to S250,
A final interpolated image temporarily stored in the created image storage means 6,
The shadow and reflection are transmitted to the combination display means 7, and these are combined into one image (S260). The method of combining in S260 can be realized by combining and displaying a plurality of objects from the same viewpoint by "restoring" the three-dimensional information from a plurality of images of the objects. This is realized by using a known technique of "projection restoration" or "Euclidean restoration" as a technique of "restoration".

After the shadows, reflections, and interpolated images are synthesized and displayed in S260, correction processing relating to the position and size is executed (S270). This is because in the basic image captured in S100, the distance from the camera to the object and the position of the object in the basic image generally differ from viewpoint to viewpoint. If interpolation images are continuously created while these are different from each other, displacement of the object other than the movement of the viewpoint occurs, and the object is apparently enlarged / reduced or the object is displayed at a fixed point. This is to solve the problem of moving up and down without rotating around.

To prevent this, the present invention solves this problem by adding a process for correcting the distance (= enlargement / reduction magnification) and position of the object in the basic image from the camera. . The position and size correction processing will be described later.

In step S270, after the position and size correction processing is completed, the created interpolated image is displayed. The correction of the shadow, reflection, combined display, position, and size in S240 to S270 may or may not be performed entirely or partially.

[0180]

Embodiment 3 Another embodiment of the present invention will be described in detail with reference to the system configuration diagrams of FIGS. In this embodiment, an interpolated image (I _S ) at a viewpoint desired by the user is created from a two-dimensional image (basic image) (I ₀ , I ₁ , I ₂ ) of the object from three viewpoints. The process will be described (that is, a process that enables two-dimensional viewpoint movement). However, in the second embodiment, a primary interpolation image is created from two or more basic images in a set of three or more basic images, and a final interpolation is performed from a plurality of primary interpolation images. The case of passing through a two-step process of creating an image has been described.

However, in this case, since a two-stage process is performed, for example, when three basic images are set as one set, the process of creating an interpolated image needs to be performed three times. The processing time is delayed, and the final interpolated image is created from the primary interpolated image, so that the final interpolated image is easily distorted.

Therefore, in the present embodiment, a process for creating an interpolated image (final interpolated image) at a time from three basic images will be described. Note that, for the sake of simplicity, a description of the same parts as those of the first and second embodiments will be omitted.

The image creator takes in three basic images of the object from the image taking means 2 on the server 14 (S
100). In the captured basic images, three basic images are made into one set, and coordinates of positions common to the basic images in the set, that is, eight or more common point coordinates are defined (S16).
0). However, the common point coordinates must be such that even if three points are arbitrarily taken, they are not aligned on a straight line. or,
It is preferable to define the common point coordinates by using an input device (not shown) such as a mouse, but other means may be used.

[0184] Corresponding points are selected from the set of basic images, and these points are connected by a straight line to define a plane (ie, define plane definition coordinates). This is performed for the entire object (S170). FIG. 20 shows an example of a plane definition screen for defining a plane. The common point coordinates defined in S160 may be the same as or different from the surface definition coordinates that define the surface.

At the time of selecting corresponding points, only one basic image is selected as in the first and second embodiments, and the corresponding points of the remaining basic images are automatically selected. You may be able to. This process will be described later.

After the surface definition for the combination of the basic images is completed, the information in S100 to S170 is transmitted from the image capturing means 2 to the interpolated image creating means 3.

Next, the user specifies a viewpoint desired to be browsed using a mouse or the like (S176). Thereafter, an interpolated image is created based on the common point information defined by the image capturing means 2 and the plane definition coordinates (S180). First, initial information (basic matrix and epipole) required to create an interpolated image is calculated by the initial information calculating means 9. The process will be described below.

FIG. 40 shows the positional relationship of each coordinate system among the three viewpoints. The three viewpoints are C ₀ , C ₁ , and C ₂ , and the basic images from the three viewpoints are I ₀ , I ₁ , and I ₂ . The 3 × 3 basic matrices of the basic images I ₀ and I ₁ , I ₁ and I ₂ , and I ₂ and I ₀ are F ₀ , F ₁ , and F ₂ , respectively. The basic matrix can be calculated from eight or more corresponding points of each basic image set by using a known method such as an eight-point algorithm (S19).
0).

After calculating the basic matrix in S190, the epipole of each basic image is calculated by a known method based on the calculated basic matrix (S200). Basic images I ₀ and I
₁ , the epipole of the basic image I ₀ is e ₀₀ , the epipole of the basic image I ₁ is e ₀₁ , and similarly the basic images I ₁ and I ₁
Basic image epipole the _{e 10} of _{I 1} for _2, the base image _{I 2} of the epipole of _{e 11,} the base image _{I 2} and _{I 0} base image _{I 2} of the epipole the _{e 20} for the basic image _{I 0}
The epipoles and _{e 21.} Epipole e _{0 0} is F ₀
Is the eigenvector corresponding to the smallest eigenvalue of ^T F _0. The same can be calculated for other epipoles.

After calculating the basic matrices and epipoles in the basic images I ₀ , I ₁ , and I ₂ , in order to generate an interpolated image IS, an interpolated image is generated using the epipole calculated in _S 200. In the parallelized image creating means 10 of the means 3,
The three basic images are parallelized to create a parallelized image (S210). The process of parallelizing the three basic images will be described later.

After creating parallelized images of the three basic images in S210, the texture mapping means 19 interpolates only the surface definition coordinates which are vertices of the defined surface (S2).
20), to create the interpolated image by texture mapping the basic image by applying the homography matrix H _s (S
230). The texture mapping process will be described later.

In the interpolated image creating means 3, the interpolated image (I
_{When s} ) is created, it is transmitted to the created image storage means 6 and temporarily stored. In the present embodiment, a final interpolated image (I _s ) is created from three basic images in the same manner as in the second embodiment. The final interpolation image (I _s ) is created by performing this operation twice. However, in the present embodiment, since the final interpolated image (I _s ) is created at once from three basic images, 1
Time is good.

Next, in order to give the created interpolation image a reality, shadows and reflections are created by the shadow creating means 4 and the reflection creating means 5 (S240, S250), and the created image storage means 6 is created. Send to and store temporarily. The details of these processes are the same as those in the above-described embodiment, and thus the details are omitted.

The interpolated image, shadow, and reflection temporarily stored in the created image storage means 6 are synthesized by the synthesis display means 7 (S
260), a correction process relating to the position and size is performed (S2).
70). Since these processes are the same, the details are omitted. After the correction processing is completed in S270, this image is displayed. As a matter of course, the shadow, reflection, composite display, and correction processing may or may not be performed entirely or partially.

[0195]

[Embodiment 4] As another embodiment of the present invention, there will be described an automatic display in which a viewpoint is moved at a predetermined path and a moving speed, an interpolated image is automatically created and continuously displayed. The description of the same parts as in the first, second and third embodiments will be omitted. In the present embodiment, an automatic continuation unit 17 for setting a moving speed between a path and a viewpoint is provided on the server 14, and an automatic continuation creating unit for performing automatic continuous display based on the set path and the moving speed. An example in which 18 is provided on the user terminal 15 will be described. FIG.
FIG. 1 shows a system configuration diagram as an example of the system configuration of the image display system 1 at that time.

Conventionally, such automatic continuous display has been realized by inputting and executing three-dimensional data, realizing moving image data, and the like, but has the above-mentioned problems such as an increase in data amount. . Conventionally, a viewpoint section is divided by a specified number of frames, and each frame is sequentially displayed. In this case, however, there is a problem that a viewpoint moving speed is constant in the viewpoint section.

Therefore, in the present embodiment, an interpolated image is automatically created by specifying a path and a moving speed in advance in the server 14 and moving the viewpoint based on the path and the moving speed. A description will be given of an embodiment in which the information is displayed. One example of the process flow is shown in the flowcharts of FIGS.

First, when the automatic continuous display is desired, a plurality of basic images to be automatically and continuously displayed are captured by the image capturing means 2 (S100). The process of capturing the basic image will be described later. Two or more of the basic images captured in S100 are defined as one set, and coordinates of positions common to the basic images in the set, that is, eight or more common point coordinates are defined (S160). At this time, the set may be a pair of two as shown in the first embodiment, or a pair of three (or a plurality of sets of three) may be provided as shown in the second or third embodiment. good.

[0199] Corresponding points are selected from the set of basic images, and the points are connected by straight lines to define a surface (S1).
70) (ie, define the surface definition coordinates). This is performed for the entire object. At this time, the common point coordinates determined in S160 may be the same as the plane definition coordinates that define the plane, or they may be different points.

In the basic image captured in S100, the distance from the camera (viewpoint) to the target and the position of the target in the basic image often differ from viewpoint to viewpoint. If automatic continuous display is performed while these are different, displacement of the object other than movement of the viewpoint occurs, and the object appears to be enlarged / reduced, or the object does not rotate around a certain point Problems such as movement occur.

Therefore, when automatic continuous display is performed in the present invention, in order to correct the distance (enlargement / reduction magnification) and position of the object in each basic image from the camera (viewpoint), the object is corrected. Information indicating the range of the object (upper, left, lower, right position coordinates in the basic image, hereinafter referred to as target frame information) is set (S175). FIGS. 37A and 37B show an example of a basic image in which target frame information is set. In FIG. 37, the inner square indicates the target frame, and the outer square indicates the viewpoint frame of the basic image.

After the surface definition is completed, S100 to S175
Information from the image capturing means 2 to the automatic continuation means 17
Send to

First, a viewpoint from which automatic continuous display is started (start viewpoint), a viewpoint to stop (stop viewpoint), and a viewpoint to end (end viewpoint) are set (S500). When setting the viewpoint, it is preferable to use a mouse or the like which is an input device (not shown) provided in the server 14.

When setting the viewpoint, it is preferable to move the viewpoint in accordance with the amount of movement of the mouse, as described in the process of capturing the basic image in S100. The specific method can be calculated using Equations 3 and 4 (described later), similarly to the process in S150. When setting the stop viewpoint, it is preferable to measure the time during which the mouse is stopped at the viewpoint using a timer function (not shown) provided in the server 14 in advance. It is.

That is, by executing S500, a path for automatic continuous display can be set. By setting the moving speed for each pass, the moving speed for each pass can be changed. When setting the moving speed, the moving speed may be input using a keyboard, or the moving speed in the path may be set by associating the moving speed of the mouse with the timer.

After setting the viewpoint path and moving speed in S500, the automatic continuous display process is executed by the automatic continuous means 17 (S505).

Next, the user sends an automatic continuous display request from the user terminal 15 and sends it to the automatic continuous
7, the data is received from S100 to S175 and S5.
00 is transmitted from the server 14 to the automatic continuous creation means 18 of the user terminal 15.

In the automatic continuous display, a plurality of still images are displayed in a moving image by displaying them continuously. Here, one still image is called a frame. That is, a moving image is formed by continuous display of frames. A path between the viewpoints is defined as a viewpoint section. FIG. 38 shows a conceptual diagram.

First, it is determined whether or not the current drawing section position is the start or end viewpoint of the view section and the stop count (stop time) is 0 (S510). Since the image displayed first should be the start viewpoint, the stop time is subtracted (S515). During that time, the basic image of the starting viewpoint is displayed.

When the stop time becomes 0, the viewpoint movement is started based on the path defined by the automatic continuation means 17 of the server 14. That is, the process in the case where the drawing is not stopped in S510 is executed.

A next drawing section position is calculated from the current drawing section position (S520). Naturally, in this case, the position of the drawing section is increased or decreased depending on the direction of the path of the continuous display.

It is determined whether the position of the next drawing section is equal to or less than the start position or the end position of the viewpoint section (S53).
0). That is, it is determined whether or not the next drawing section position exceeds the viewpoint section.

If the position of the next drawing section does not exceed the viewpoint section in S530, information on the viewpoint section and the drawing section position is transmitted to the interpolated image creating means 3 to create an interpolated image (S1).
80). When an interpolated image is created, the interpolation image creation process in S180 may be used. If necessary, a shadow is created in S260, a reflection is created in S270, a combined display process is performed in S280, and a position and size correction process is performed in S290.

If the next drawing section position exceeds the viewpoint section in S530, the next viewpoint section is set to the next adjacent viewpoint section, and the next drawing section position is set to the initial value (start section position or end section position). (S540).

After setting in S540, it is determined whether or not the next viewpoint section exceeds the last viewpoint section of automatic continuous display (S550). If the next viewpoint section does not exceed the final viewpoint section, an interpolated image in S180 is created.

The next viewpoint section exceeds the last viewpoint section (S55).
0), and when the automatic continuous display is a loop display (ie, when the final viewpoint section is reached, the display is returned to the start viewpoint section and redisplay is performed) (S555), the viewpoint section set in the loop and the initial drawing section The position is set (S560). S
If the next viewpoint section does not exceed the final viewpoint section in 550, or if the loop display is not set in S555, an interpolated image is created (S180). The setting of the loop display may be specified in advance when setting a path or the like in the automatic continuation means 17 of the server 14 or may be set in the user terminal 15.

After creating the interpolated image in S180, the shadow,
A reflection is created (S240, S250) and a composite display thereof is performed (S260), and then the interpolated image is transmitted to the correction processing means 8 to execute the position / size correction processing (S240).
270). S270 about position / size correction processing
Perform the same process as.

After executing the correction processing in S270, the correction processing is displayed on the user terminal 15. Thereafter, by continuously repeating S505 until the final viewpoint is reached, a continuous frame can be created, and by displaying this on the user terminal 15, it becomes possible to display it as a moving image. (S570).

[0219]

Embodiment 5 Hereinafter, a process for setting a path on the user terminal 15 will be described. Also, overlapping processes are omitted. In the present embodiment, the user terminal 15 includes an automatic continuation unit 17 for setting a moving speed between a path and a viewpoint, and an automatic continuation creating unit 18 for performing automatic continuous display based on the set path and the moving speed. The image display system 1 in the case where there is is described. FIG. 4 shows an example of a system configuration diagram of the image display system 1 in this case.

On the server 14, the basic image of the object is fetched (S100), common points and planes are defined for it (S160 and S170), and target frame information is set (S175). The information from S100 to S175 is transmitted to the user terminal 15, and the viewpoint setting is performed in the automatic continuation means 17 provided in the user terminal 15 (S50).
0). At this time, the user sets the moving speed between the path and the viewpoint by using an input device (not shown) such as a mouse provided on the user terminal 15 in advance.

When setting the viewpoint, it is preferable to move the viewpoint according to the amount of movement of the mouse, as described in the process of capturing the basic image in S100. The specific method can be calculated by using Equations 4 and 5 (described later), similarly to the process in S150. Further, when setting the stop viewpoint, it is preferable to measure the time during which the mouse is stopped at the viewpoint using a timer (not shown) provided in the user terminal 15 in advance. It is.

That is, by executing S500, a path for automatic continuous display can be set. By setting the moving speed for each pass, the moving speed for each pass can be changed.

After setting the path and moving speed of the viewpoint in S500, the set viewpoint is transmitted to the automatic continuous creation means 18 and the processes from S505 to S570 are executed. By setting the moving speed, automatic continuous display can be performed.

<Process of Capturing Basic Image> The process of capturing a basic image will be described below. FIG. 8 shows an example of a flowchart illustrating an example of the flow of a process for capturing a basic image. First, the image creator extracts a two-dimensional image of an object photographed by a scanner or the like or previously stored as digital data on a computer,
It is read (S110). After reading the plurality of images, the image creator activates the image capturing means 2 (that is, activates the viewpoint definition screen for performing the viewpoint definition shown in FIG. 17A) to an initial state. Then, on the viewpoint definition screen, a two-dimensional plane serving as a viewpoint is placed on a three-dimensional virtual space (this three-dimensional virtual space is assumed to be a sphere having the origin of the three-dimensional virtual space as a coordinate system).
(S120). FIG. 17B shows a screen image of the viewpoint definition screen in this state. After creating the viewpoint in S120, the image read in S110 is fetched, and the basic image is made to correspond to the viewpoint created in S120. FIG. 17C shows a screen image of the viewpoint definition screen in this state. The position of the viewpoint such as up, down, left, and right of the basic image read into the viewpoint is adjusted (S130). FIG. 17D shows a screen image of the viewpoint definition screen in this state.

When the viewpoint set in S110 to S130 is not executed in all viewpoints (S14)
0) Move the viewpoint to input the next viewpoint. In this case, the viewpoint position and the line of sight (direction of the image plane) of each image may be converted into a numerical value and keyed. However, it is complicated for the image creator to numerically input the key. It is. Therefore, as shown in FIG. 18, in a sphere centered on the coordinate system origin in the assumed virtual three-dimensional space, the x and y coordinates on the viewpoint correspond to the polar coordinates (φ, θ) of the spherical surface. Then, polar coordinates (φ, θ) are calculated according to the amount of movement of an input device (not shown) such as a mouse, and the viewpoint is moved to that position (S150). FIG. 19A shows a screen image in which the viewpoint has moved in the three-dimensional virtual space.

The polar coordinates (φ, θ) are calculated as follows. Assuming that changes in the x and y coordinates in a unit time are Δx and Δy, Δφ and Δθ can be calculated by Equation 4.

[Number 4] ^{Δφ = Cx × Δx (∀ Cx∈Z} ) Δθ = Cy × Δy (∀ Cy∈Z)

The polar coordinates (φ, θ) can be calculated by using Expression 4 and Expression 5 shown below.

[Number 5] φ = (φ + Δφ) mod 2π θ = Θ max (θ + Δθ ≧ Θ max) θ + Δθ (Θ max> θ + Δθ> Θ min) Θ min (Θ min ≧ θ + Δθ) Θ max, Θ min is, 0.5π> Θ _max > Θ _min
> -0.5π

After moving the viewpoint in S150, the process returns to S11 again.
By repeating S0 to S140, two or more viewpoints are created. Screen images in which the second and subsequent viewpoints have been created are shown in FIGS. 19B to 19D. Although eight viewpoints are provided in the screen image used in the present embodiment, any number of viewpoints may be used as long as the number is two or more.

<Automatic Matching Process of Corresponding Points> When defining corresponding points from a set of basic images, it is necessary to manually specify all the basic images to be set using a mouse or the like. Was. However, manual work is very troublesome because it takes a lot of time and labor, and is extremely burdensome when the number of images to be grouped is large or the image size is large. So, among the basic images that make up a set,
Once points are defined in one basic image, the process of automatically extracting corresponding points from the remaining basic images in the set will be described below. First, two basic images (I ₀ ,
The case where I ₁ ) is a set will be described.

First, in each basic image, points along the outline of the object (hereinafter, edge end points) are automatically extracted by using the Susan operator. The method of automatically extracting the edge end point of the object using the Susan operator is described in S.M. Steve, M. Brady (SMStev
e and M. Brady), "SUSAN A New Approach to Low Level Ima"
ge Processing, International Journal of Computer Vi
sion, Vol.23, No.1, p.45-78, 1997). FIG. 44 shows a diagram in which edge end points are automatically extracted.

An arbitrary point forming a pair among the extracted edge end points is determined as a point for weak calibration (hereinafter, designated Delaunay triangle designated point), and the user designates it. FIG. 45 shows a case where a Delaunay triangle designation point is designated. Note that in FIG.
You have specified a point.

One point on the basic image I ₀ obtained by extracting the edge end points is defined as p _0, and the corresponding Delaunay triangle designated points on the basic images I ₀ and I ₁ (the corresponding points of the designated Delaunay triangle designated points). Let the number of pairs be n) be q _0i and q
_1i and (i = 0, 1,..., N−1). By using the sequential method from Delaunay designated point to generate a Delaunay triangle in the base image I _0. For the sequential method, see D. Lischinski, Graphic Gems, Volume 4, Sequential Delaunay Triangulation.
(Incremental Delaunay triangluation in Graphics G
ems IV, AP Professional, 1994). FIG. 46 shows an example of the basic image in which the Delaunay triangle is generated.

The vertices of the Delaunay triangle including the point p ₀ are defined as q
₀₀ , q ₀₁ and q ₀₂ . If the point p ₀ is outside the Delaunay triangle, q ₀₀ ,
Defined as q ₀₁ and q ₀₂ . At this time, u and v are defined as unknown scalar parameters as follows. a = q ₀₁ -q ₀₀ b = q ₀₂ -q ₀₀ p ₀ = ua + vb + q ₀₀

FIG. 47 is obtained from these equations, and u and v are calculated by solving them. However, if it is not regular is replaced with the next closest Delaunay triangle specified point q _02. These initial estimated position p ^ ₁ of corresponding points that put the base image I ₁ is given by the following equation. p ＾ ₁ = u (q ₁₁ −q ₁₀ ) + v (q ₁₂ −q ₁₀ )
+ _{Q 10}

[0235] does not satisfy the epipoles constraints even if the interpolation from the triangle of the Delaunay triangle specified point in linear, point p ^ ₁ of the basic image I ₁ satisfies the epipoles constraints between the point p ₀ of the basic image I ₀ There is a need to make modifications. When the same next coordinate representation of the point _p 0, _{p 1} and ^p _~ ^{0, p} _{~ ^1,}
As 1 homogeneous component since the corresponding point is a point of the finite spaces present in a basic ^{_{image, p ~ 1 = (p T}} 1,
1) = (p _1x , p _1y , 1) ^T.

At this time, the epipole constraint is a 3 × 3 basic row.
P using column F^{~ T} ₁Fp^~ ₀= 0. Elementary matrix
Each component of F is also a point p^~ ₀That the component of the point is also known
From the epipole constraint^~ ₁Is as shown in FIG.
It is clear that it is constrained on the epipole constraint line.
You. That is, L = Fp^~ ₀Then the epipole constraint is Lp
^~ ₀= 0.

[0237] When the projection of the initial estimated position p _{^ 1} in this epipoles constraints on a straight line, epipoles constraints straight line to a straight line perpendicular ^{_{p 1 = t (L x,}} L y) T + p ^ 1 is, L ^· _{p ~} 0 = 0
Therefore, the parameter t = − (L _x p _1x + L _{y
^{p ^ 1y + L z) /}} (L 2 x + L 2 y) in the calculation can be performed. P ₁ from these meet the epipoles constraints is obtained.

[0238] Since p ₁ satisfying epipoles constraints obtained earlier is not necessarily coincide with the corresponding points on the basic image by examining the luminance difference between epipoles constraints straight line, correcting the corresponding point (linear search ).

[0239] point search window defined around the point p ₀ for each pixel of the epipole constraints on a straight line from p ₁ within a predetermined distance (Figure 48 See: While in the present embodiment describes a case of a square , Or any other position) at which the difference in luminance is minimized. When two basic images are used, the brightness may be different even at the corresponding points due to the difference in viewpoint, but it is obvious that the relative error minimum point is relatively stably obtained. As for the difference in luminance, not only the average is used for the entire search window, but also the influence of the frequency is considered by calculating the luminance difference for the portion.

The brightness difference in the search window is determined by setting the width of the search window to 2
h, Basic image I₀, I₁Coordinates (x ₀, Y₀),
(X₁, Y₁Luminance difference C)₀(X₀, Y₀), C
₁(X₁, Y₁) And recursively calculated as shown in FIG.
It is possible by putting out. K is the size of the search window
Represents the weight of each size, and is the same for all sizes when k = 1
Consider by weight. For k <1, focus on the influence of small search window
However, when k> 1, the influence of a large search window is emphasized. general
To reduce the effect of noise and emphasize low frequency components
It is preferred to use k> 1. FIG. 50
FIG. 4 shows a diagram when matching of corresponding points is performed.

In the above description, the case of two basic images has been described. Next, the case of automatically matching points corresponding to three or more images will be described. In the case of three or more images, corresponding points are first obtained for two images as described above, and then corresponding points are obtained analytically for the third and subsequent images. The process will be described below.

Basic image I₀, I₁, I₂The corresponding point of
Each represented by homogeneous coordinates is p^~ ₀, P^~ ₁, P^~ ₂When
And basic image I₀And I₁, Basic image I₁And I₂, Basic painting
Statue I ₂And I₀The basic matrix of F₀₁, F₁₂, F₂₀Toss
You. p^~ ₀, P^~ ₁Is known, so f₁₂= F₁₂
p^~ ₁, F₂₀= F^T ₂₀p^~ ₁Then, in FIG.
Equation is obtained, and by obtaining this linear equation,
P in the third image₂It is possible to ask for
You. This is the basic image I₀, I₁Two epipaws from
To determine the intersection point of
This eliminates the need for performing a linear search. However, if it is not regular
In this case, it is obvious that the two straight lines are parallel.
It will be determined using

<Process for Parallelizing Two Images> An example of the process for parallelizing images in S210 will be described below. In this embodiment, two basic images I ₀ and I ₁
The parallelization will be described. Naturally, the parallelization needs to be created for all two target basic images.

To parallelize two basic images, first,
The two basic images are combined with the epicard calculated in S190.
Le e_ijArbitrary not parallel to (i = 0,1, j = 0,1)
Axis d of_ijAround θ_ijJust by rotating
I do. In this embodiment, for simplicity, d_ijEpi
Paul e_ijAnd a line d at infinity perpendicular to_ij= (E
_izye_ijx0)^TAnd Two basic images are flat
The line means that this conversion turns the epipole to infinity
Means being transferred. That is, the rotation matrix R (d _ij,
θ_ij) Shown in FIG. 23A, R (d_ij,
θ_ij) E_ijIs 0.

Next, the epipolar line is rotated about the z-axis so as to be horizontal. That is, as shown in FIG. 23B, this rotation matrix is R (φ _ij ), e ′ _ij = R (d _ij , θ
_ij ) e _ij , the y component of R (φ _ij ) e ′ _ij is 0.

P ＾ _ij = R (φ _ij ) R (d _ij , θ
_ij ) The plane-defined coordinates are aligned on the same scanning line except for the indefiniteness of a constant multiple due to the conversion of p _i . Assuming that a 3 × 3 shear deformation matrix T _i is applied to the second basic image for removing the indeterminacy of a constant multiple, the same scan is performed by the homography H _i0 , H _{i + 1,1} calculated by _Expressions 6 and 7. Parallelization in which surface definition coordinates are arranged on a line can be performed. This relationship is illustrated in FIG.
It looks like 4.

H _i0 = R (d _i0 , −θ _i0 ) R (−φ _i0 )

Equation 7] _{H i + 1,1 = (T i} R (φ i + 1, i) R (d i + 1,1, θ i + 1, 1)) -1 = R (di _{+ 1,1} , -θi _{+ 1,1} ) R (φi _{+ 1 , i} ) _Ti- ¹

Basic image I₀Coordinate point (of the surface definition coordinates
One point) p₀Is the point p ＾₀₀=
H^-1 ₀₀p₀And the basic image I₁Coordinate point p₁Is parallelized
The point p ＾ mapped by₁₁= H^-1 ₁₁p₁Same as
It has one y coordinate. Basic image I ₀, I₁S₁To the ratio of
Point_s0 (See FIG. 22)
Since it has the y-coordinate, it can be interpolated as shown in Expression 8. As well
The basic image I₀, I₂S₁The point p ＾ that is internally divided into the ratio
_s2Is shown as in Equation 9.

[Equation 8] _{p ^ s0 = (1-s} 1) p ^ 00 + s 1 p ^ 11

[Equation 9] _{p ^ s2 = (1-s} 1) p ^ 01 + s 1 p ^ 20

Basic image I₀, I₁The basic drawing
Statue I₀The four corner points of b_0i(I = 0,1,2,3)
And the point in the parallelized image is b ＾_00iAnd In addition,
Main image I₁The four corner points of b_1i(I = 0, 1, 2, 3)
And the point in the parallelized image is b ＾_11iAnd Base
Interpolation of the four corner points on the main image is b_s0i = (1-
s ₁) B_0i+ S₁b_1iAnd on the parallelized image
Interpolation of the four corner points is b ＾_{s0 i} = (1-s₁) B ＾
_00i+ S₁b_11iAnd At this time, p_s0i= H
_s0p ＾_s0i3 × 3 matrix that is the least squares solution of
H_s0To the basic image I₀, I₁Interpolation homography of
You. Similarly, the basic image I₀, I₂H_{s 2}To
It is possible to calculate.

After calculating the interpolation homography, the intermediate viewpoint I
The coordinates of the point at _s0 are calculated by using _{equation (} 10).

[ _Mathematical formula-see original _document ] p _s0 = H _s0 p ＾ _s0

<Process for Parallelizing Three Images> An example of a process for parallelizing three images will be described below. Basic image _{I 0, I 1,} respectively rotation matrix to collimate the _{I 2 R (d 0, θ} 0), R (d 1, θ 1) R
(D ₂ , θ ₂ ). _{However, R (d i, θ i} ) indicates the rotation about the angle theta _i of the rotation shaft _{d i.} Rotation matrix R (d
_i , θ _i ) are shown in FIG.

The rotation matrices of the basic images I ₀ , I ₁ and I ₂ around the z-axis are R (φ ₀ ), R (φ ₁ ) and R (φ ₂ ), respectively. The rotation matrix R (φ _i ) about the z-axis is shown in FIG. A 3 × 3 shear matrix for I ₀ of the basic image I ₁ is T ₁ , and a 3 × 3 shear matrix for I ₀ of the basic image I ₂ is T ₂ .

In the case of converting three basic images at a time, the rotation conversion for moving the epipole to infinity may be performed as in the case of the two basic images, but the rotation is obtained for each set of basic images. However, since all the basic images are converted at the same time, the degree of freedom of the rotation axis is lost and uniquely determined.

[0253] the rotation matrix _{R (d 0, θ 0)} , R (d 1, θ
₁ ) If the epipole transferred by R (d ₂ , θ ₂ ) is e ′ _ij (i = 0, 1, 2, j = 0, 1), (1) e ′ ₀₀ = R (d ₀ , θ ₀₎ ) E ₀₀ (2) e ′ ₂₁ = R (d ₀ , θ ₀ ) e ₂₁ (3) e ′ ₁₀ = R (d ₁ , θ ₁ ) e ₁₀ (4) e ′ ₀₁ = R (d ₁ , θ) ₁ ) e ₀₁ (5) e ′ ₂₀ = R (d ₂ , θ ₂ ) e ₂₀ (6) e ′ ₁₁ = R (d ₂ , θ ₂ ) e ₁₁

Here, that the images become parallel means that other
It means that the pole becomes infinity and the homogeneous component becomes 0
So e '_ijz= 0 holds. Basic image I₀No
The rotation axis d₀Of the same components
Then, (7) (-e_00xd_0y+ E_00yd_0x) Sin
θ₀+ Cosθ₀= 0 (8) (-e_01xd_0y+ E_01yd_0x) Sin
θ₀+ Cosθ₀= 0 holds. Therefore d_0y= D_0x× (e_01y-E
_00y) / (E_01x-E _00x) Is obtained.

Here, d_0xThe coefficient of α₀Then rotate
Since the axis is a unit vector, d² _0x+ D² _0y=
(1 + α² ₀) D² _0x= 1 and the rotation axis d₀The figure
42 (a). Substituting (7) into this equation
The rotation angle θ₀Is tanθ₀= 1 / (e_00xd
_0y-E_00yd_0x) Can be calculated. Therefore, the figure
By substituting 42 (b) into FIG. 41 (a), the rotation matrix R
(D₀, Θ₀) Can be calculated. R
(D₁, Θ₁), R (d₂, Θ₂)
It is possible to leave. This process is called a parallelization rotation conversion (3 images
(Parallel image transformation).

Next, a horizontal rotation conversion process for arranging sets of corresponding points on the same horizontal line is performed. In the case of two images, the conversion may be performed so that the y-coordinate of the epipole becomes 0. However, in the case of three images, since the epipole is defined for each set of images, the above method cannot be used. Therefore, performs collimation for the first to the base image I _0, I ₁ set, the base image I
_{For 2} , the relative transformation of the basic images I ₀ and I ₁ with respect to the coordinate system is determined.

First, basic image I₀, I₁About two
As in the basic image, the y-coordinate of the epipole is
By converting to 0, the corresponding points are aligned at the same y coordinate.
Perform the conversion as follows. That is, e ''₀₀= R (φ₀) E '
₀₀ , E ''₀₁= R (φ ₁) E '₀₁ Is converted by
The epi-coordinate y coordinate is e ''_00y= 0, e ''_0
1y= 0. Here the basic image I₀Solving for
e ''_00y= E '_{00 x}sinφ₀+ E '_00yco
sφ₀= 0 and the rotation angle φ₀= -E '
_00y / E '_00xIs required. Therefore, FIG.
By substituting (c) for FIG. 41 (b), the rotation matrix R (φ₀)But
Desired. Basic image I₁The same applies to

[0258] Then the base image I _0, it becomes possible to determine the relative translation of the basic image I ₂ with respect to the coordinate system of the I _1, the base image I _0, the basic image in converted into a parallel of I ₁ I ₂
(Because of the relative conversion) to a transformation of epipole e ₂₁ is the base image I _0, it becomes the direction of the epipole of I ₂ for. _{That, e '' 20 = -ke '} ' 21 is established.

Here, e ″ ₂₁ = R (φ ₀ ) R
(D ₀ , θ ₀ ) e ₂₁ is known, and similarly, e ″ ₂₀ =
Since it is R (φ ₂ ) e ′ ₂₀ , FIG. 42 (d) may be solved. Solving this results in FIG. 42 (e), where R (φ ₂ )
Can be obtained.

[0260] Next will be described a process of scaling to adjust the focal length and position of the base image I _2. That is, to determine the shear matrix T _2. The projected corresponding points p ″ _0y , p ″ _{2y of the} basic images I ₀ , I ₂ are p ″ _0y =
The p _{'' 2y} can be shown as in FIG. 43 (a).
Here, if R ₀ and R ₁ are defined as shown in FIG. 43 (b), from FIG. 43 (a), (T ₂ R ₂ p ₂ ) ^T KR ₀ p ₀ =
p ^T ₂ ( ^RT ₂ T ^T ₂ KR ₀ ) p ₀ = 0.

On the other hand, the basic images F ₀ and I ₂ have the basic matrix F ₂
Since the relationship of p ^T ₂ F ₂ p ₀ = 0 holds, FIG.
3 (c) holds. _{However, t ij} denotes the component of _{T 2.} Thus it is determined shear matrix _{T 2} as shown in FIG. 43 (d). Therefore homography matrix of the basic image _{I 2 H} 2 =
(T ₂ R (φ ₂ ) R (d ₂ , θ ₂ )) ^{− 1} . Similarly, the homography matrices H ₀ and H ₁ are calculated.

The basic images I ₀ , I ₁ , I ₂ are represented by (1-s−
The process of the interpolation homography H _st (that is, the interpolation homography matrix to be finally obtained) at the point where the ratio of t): s: t is internally divided is as follows. And calculate.

The four corner points of the basic images I ₀ , I ₁ , I ₂ are respectively set to b _0i , b _1i , b _2i (i = 0, 1, 2, 3)
And the points in the parallelized coordinates are b _{を 0i} , b ＾ _1i , b
^ When _{2 i,} the ^{_{b ^ 0i = H -1 0 b}} 0i b ^ 1i = H -1 1 b 1i b ^ 2i = H -1 2 b 0i.

However, these coordinates are represented by three-dimensional homogeneous coordinates.
Is done. Basic image I to be interpolated here _stPoint b at the four corners
_stiAnd b ＾ in its parallelized coordinates_stiAnd b
_sti= (1-st) b_0i+ Sb_1i+ Tb_2i,
b ＾_sti= (1-st) b ＾ _0i+ Sb ＾_1i+ T
b ＾_2iTo interpolate. At this time, b_sti= H_stb ＾
_{s ti}Interpolate a 3 × 3 matrix that gives the least squares solution of
MOGRAPHY H_stAnd

<Process of Texture Mapping> The texture mapping in S230 will be described below.
FIG. 9 is a flowchart illustrating an example of the flow of the texture mapping process. When performing texture mapping, first of all, which surface is to be used as a texture image is selected by the surface selecting means 11 (S300). The surface selection process will be described later. If the texture images of the faces have been selected for all the faces (S350), the processing for the contours of the faces is executed by the contour processing means 12 (S360). The process of the contour processing will be described later. This contour processing is a process for avoiding the corners in the contour portion when the curved surface is approximated by a rough polygon when defining the surface, thereby avoiding this and bringing the image closer to a more natural image. After the end of the contour processing in S360, the color correcting means 13 executes color correction processing for the surface. This color correction process is a process for eliminating a color difference occurring between adjacent surfaces and bringing the image closer to a natural image when a texture image is selected as in step S300, and an image may be selected from different images between adjacent surfaces. . The color correction process will be described later. The contour processing in S360 and the color correction processing in S410 are preferably provided in the texture mapping process. However, all or only one of them may or may not be provided. Good.

<Process of Selecting a Surface> The process of selecting a surface in S300 will be described below. FIG. 10 is a flowchart showing an example of the flow of the face selection process.
Shown in

At the time of performing texture mapping, if an image used as a texture is changed during movement of the viewpoint, the display becomes unnatural. Therefore, the plane selecting means 11 selects which plane is to be a texture image.

A texture image reference number is determined once for each surface by preprocessing. Which of the two basic images is to be used is determined based on the following two criteria. FIG. 25 shows a conceptual diagram of a texture selection criterion. (Reference 1) Surface orientation (Reference 2) Surface area

It is determined whether or not the image is a face-up image for the target surface (S310). If the corresponding surfaces are both facing up (that is, the surface 1 shown in FIG. 25), the area of the surface in each image is calculated (S320), and the image with the larger area is textured on the surface. Image (S33)
0). When the area is used as a reference, the image with the higher resolution of the two images is used, which is effective for improving the image quality.
If one face-up image is found in S310, the face-up image is used as the texture image of the surface (S34).
0). That is, as shown on the surface 2 in FIG. 25, when one is facing up and the other is facing down, the image facing up is set as the texture image of that surface.

Further, the plane selecting means 11 not only selects a plane to be used as a texture image from the two images, but also combines the two images according to the viewpoint and creates a new image. The created image may be selected as the texture image. This is because, in the above-described plane selection method, in order to select one image from two images, the texture image is frequently switched depending on the way of moving the viewpoint.

Here, combining two images according to the viewpoint means that the brightness of the image A is A, the brightness of the image B is B, and the viewpoint between the image A and the image B is from the image B. Assuming that the ratio of the distance up to the distance is d (0 ≦ d ≦ 1), the combination ratio is calculated by the following equation. S = A × d + B (1-d)

Therefore, for example, if the ratio of the distance from the viewpoint is A
Is 10% and B is 90%, the luminance S of the combined image is 0.9A + 0.1B. Further, when the ratio of the distance from the viewpoint is 50% from B and 50% from A, the luminance S of the image combined is 0.5A + 0.5B
Becomes In this case, the viewpoint is located between the image A and the image B.

<Other Process of Surface Selection> Further, in the above-described surface selection method, for example, in FIG. 25, since the surface is oriented, even if it is an image of a viewpoint on the way, The front and back can be distinguished, and the visibility of the surface itself can be determined,
When both face up as in the relationship between the dial and the wristband as shown in FIG. 52, occlusion hidden by other surfaces occurs, so that visibility cannot be determined. Therefore, a process of drawing sequentially from a surface located far from the viewpoint using an epipole in each image will be described.

Since the surface is two-dimensional and has no depth,
Hidden surface removal is performed depending on the order in which the surfaces are selected. Therefore, the order in which the planes are selected can be determined based on the criterion corresponding to the distance from the viewpoint in the image in which the plane is visible. As shown in FIG. 53, the viewpoints of the basic images I ₀ and I ₁ are C ₀ and C ₁ and are represented by epipoles e ₀ and e ₁ .
When occlusion occurs, approximate hidden surface elimination can be performed by drawing sequentially from the surface farthest from the epipole.

<Process of Contour Processing> The process of contour processing in the contour processing means 12 in S360 will be described below. FIG. 11 is a flowchart showing an example of the flow of the contour processing process.

When a curved surface is approximated by a rough polygon when defining a surface, the corner at the outline portion becomes conspicuous and an unnatural image is obtained. To avoid this problem, it is necessary to finely divide the polygon. However, in this case, the number of vertices of the surface to be interpolated in the parallelized image increases, and the data amount increases. At the same time, the input operation becomes difficult. In the three-dimensional CG, there are a method of adaptively subdividing only the outline portion and a method of adding a textured surface only to the outline portion with coarse division.

In the present invention, this is realized by the following method capable of creating a more natural image than the conventional method. That is, a mask for erasing other than the target object is created in the original image, a new quadrilateral surface is added outside the outline of the defined surface, and the texture with the mask is mapped. The mask can be created by manually painting the background from the original image using image editing software as shown in FIG. 26, but it is also possible to automatically extract a single color background such as a blue background. is there.

A quadrilateral surface outside the surface is called an offset surface. When faces are connected via a ridge line that is a connection between face vertices, adjacent faces are called adjacent faces. A ridge line having no adjacent surface is referred to as an outer boundary ridge line, and an ridge line having an adjacent surface with one facing up and the other facing down is referred to as an inner boundary ridge, and the outer boundary ridge and the inner boundary ridge are collectively called a boundary ridge. When the point at which the vertex coordinates of the surface are averaged is defined as the center of gravity of the surface, an outward unit vector passing through the center of gravity and perpendicular to the ridge is defined as a ridge offset vector. The average unit vector of the boundary ridge line offset vector including the vertex is called a vertex offset vector, and the point at which the vertex is moved in the vertex offset vector direction is called an offset vertex. FIG. 27 conceptually illustrates the above relationship.

In the case shown in FIG. 27, the display process of the offset plane is as follows. First, a ridge offset vector is calculated for each image (S370). A vertex offset vector is calculated by averaging the calculated edge offset vectors (S380). With respect to the created interpolated image, an offset plane constituted by both vertices of the boundary ridge line and vertices moved in the direction of the offset vector therefrom is displayed (S390). Thereafter, a normal surface is displayed (S40).
0).

In this case, only the interpolated vertex satisfies the epipole constraint shown in Equation 3, but points inside the offset plane other than the vertex do not satisfy the projective geometric constraint of the viewpoint. However, since the offset plane is usually defined near the edge line, the distortion is small.

<Process of Color Correction Processing> Hereinafter, step S410 will be described.
The color correction process in the color correction means 13 will be described.
FIG. 12 is a flowchart showing an example of the flow of the color correction process.

When a texture image is selected by the plane selection means 11 in S300, different images may be selected between adjacent planes, and the color difference at the boundary portion may be conspicuous and unnatural.

Thus, in the present invention, the color difference is corrected by mixing two texture images. When mixing, the mixing ratio is attenuated according to the distance from the ridgeline. This process is a process that is calculated only once for the texture image, and is executed as follows.

First, for each pixel inside the plane, the text on the adjacent plane
Distance from edge line with different texture image reference number to pixel
Is obtained (S420). Next, the edge line weight at pixel point p
W_iAnd the shortest distance from ridge line i is d_iAnd the face
The distance from the center of gravity to the ridgeline is r _iIs shown in FIG.
Weight w of the ridge_iBy calculating (S43)
0), the weight w of the ridge line is calculated from Equation 11 (S44)
0).

[Equation 11] w = min (Σ _i w _i , 1)

Texture images 0 and 1 corresponding to pixel point p
Assuming that the colors are C ₀ (p) and C ₁ (p), the corrected color C (p) is calculated by Equation 12 (S450).

C (p) = (1−w) C ₀ (p) + wC ₁ (p)

<Shadow Creation Process> A process of adding a pseudo shadow to the projected and restored object in the shadow creation means 4 in S260 will be described below. However, it is assumed that the shadow is projected on one plane.

Let N be a two-dimensional affine transformation matrix with six degrees of freedom. The point on the image of the vertex of the object projected on the floor surface (the plane on which the shadow is created is the floor surface, but may be other than m) is m
_{i = (u i, v i} , 1) and ^T. Since four degrees of freedom are determined by specifying two points to determine the relative position of the shadow, the direction and size of the shadow are specified by the remaining two degrees of freedom. Specifying the direction and size of the shadow can be realized by shear deformation. After the vertices of the projected and restored object are converted into relative coordinates based on two points, the original coordinates are restored by applying shear deformation. Let T be the transformation matrix that translates so that m ₁ is the origin, R be the rotation matrix about the origin such that m ₁ m ₂ is the x-axis, and S be the shear deformation matrix (where the shear deformation equation is A in the x direction and b in the y direction). That is, the matrices T, R, and S are shown as shown in FIGS.

Further, cos θ and sin θ are shown in FIG.
(D) and (e) are shown. By moving the viewpoint, two points m ₁ and m ₂ move to m ′ ₁ and m ′ ₂ , respectively.
Assuming that the translation matrix at the new coordinates is T 'and the rotation matrix is R', the affine transformation matrix N representing the coordinate transformation of the shadow
Is represented by the equation shown in FIG. FIG. 32 is a conceptual diagram showing a process of calculating a shadow. A shadow is created by tracing the converted vertices using surface information at the time of projective restoration and displaying a translucent surface. As a matter of course, the display of the projected and restored object is performed after the shadow display processing.

<Process of Creating Reflection> Hereinafter, the process of S270 will be described.
The process of creating an image to be reflected by the reflecting means will be described. The reflection image is also created using the affine transformation matrix of the object subjected to projection restoration similarly to the shadow. The two-dimensional affine transformation is 6
Since it has a degree of freedom, the transformation matrix can be determined by designating two points of the object on the reflecting floor and one point that appears to be reflected.

As in the case of the shadow, the object 2 on the floor surface
Point to m₁, M₂And m₁Translate so that the origin is
T, m₁m₂Is centered on the origin where
Let R be the rotation matrix that has been set (where the matrices T and R are
(A) and (b)). Parallel movement after moving the viewpoint
The matrix and rotation matrix are T 'and R'. At this time, the reflection
The affine transformation matrix M is shown in Expression 13. Also, S '
Reflecting point m₃= (U ₃, V₃, 1)^TUsing FIG. 33
It is represented by

M = T ′ ⁻¹ R ′ ⁻¹ S′RT

<Position and Size Correction Process> The position and size correction process in the correction processing means 8 in S290 will be described below. FIG. 13 shows an example of a process flow of the position and size correction processing.

The basic image captured in S100 is
In general, the distance from the camera to the target and the position of the target in the basic image are different for each viewpoint. And if these are different and the creation of the interpolated image is performed continuously,
Solved the problem that the displacement of the object other than the viewpoint movement caused the problem that the object was apparently enlarged / reduced and the object moved up and down without rotating around a certain point To do that.

In order to prevent this, the present invention solves this problem by adding a process of correcting the distance (= enlargement / reduction magnification) and position of the object in the basic image from the camera. .

First, in step S100, information indicating the range of the object (the upper, left, lower, and right position coordinates in the basic image, which is hereinafter referred to as “target frame information”) is taken in as a two-dimensional image. It is set when performing.

Next, when an interpolated image is created in S200, the enlargement / reduction magnification and the amount of parallel movement are calculated based on the target frame information and corrected (S470). Since the target frame information differs for each viewpoint, the target frame information is calculated based on Equation 14 by linearly interpolating using the target frame information of the start viewpoint and the total number of frames from the start viewpoint to the end viewpoint (S480). FIG. 39A shows an image when the position and size correction processing is not performed, and FIG. 39B shows an image when the correction processing is performed.

[Equation 14] ((total number of frames−number of frames) × information indicating range of start viewpoint + number of frames × information indicating range of end viewpoint) ÷ total number of frames

Naturally, shadows and reflections, composite display,
Correction processing or the like is not performed, and an interpolated image may simply be created.

In implementing the present invention, a storage medium storing a software program for realizing the functions of the present embodiment is supplied to the system, and the computer of the system reads and executes the program stored in the storage medium. It is natural to be realized by.

In this case, the program itself read from the storage medium realizes the functions of the above-described embodiment, and the storage medium storing the program naturally constitutes the present invention.

As a storage medium for supplying the program code, for example, a floppy (registered trademark) disk,
A hard disk, an optical disk, a magneto-optical disk, a magnetic tape, a nonvolatile memory card, or the like can be used.

When the computer executes the readout program, not only the functions of the above-described embodiments are realized, but also the operating system running on the computer based on the instructions of the program. It goes without saying that a part or all of the processing is performed and the functions of the above-described embodiments are realized by the processing.

[0301] Further, after the program read from the storage medium is written into a nonvolatile or volatile storage means provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the program is read. Based on the instruction, the arithmetic processing device or the like provided in the function expansion board or the function expansion unit may perform part or all of the actual processing, and the processing may realize the functions of the above-described embodiments. It is.

[0302]

According to the present invention, unlike the conventional display method using moving image data or three-dimensional data, the amount of data is small, and the problems which have existed so far have been solved, and a more realistic display has been made possible.

Further, by automatically creating an interpolated image continuously from the basic image and performing continuous display, a moving image can be formed with a smaller data amount than conventional moving image data. Also, at this time, the moving speed between the path and the viewpoint can be arbitrarily set, so that a more interactive moving image can be expressed.

[Brief description of the drawings]

FIG. 1 is a system configuration diagram showing an example of a system configuration of the present invention.

FIG. 2 is a diagram illustrating an example of a system configuration of an interpolated image creating unit according to the present invention.

FIG. 3 is a system configuration diagram showing an example of a system configuration when a means for performing automatic continuous display is provided.

FIG. 4 is a system configuration diagram showing another example of a system configuration provided with a means for performing automatic continuous display.

FIG. 5 is a first page of a flowchart showing an example of the process of the present invention.

FIG. 6 is a second page of a flowchart showing an example of the process of the present invention.

FIG. 7 is a flowchart illustrating an example of an interpolation image creation process.

FIG. 8 is a flowchart illustrating an example of a process of capturing a basic image.

FIG. 9 is a flowchart illustrating an example of a texture mapping process.

FIG. 10 is a flowchart illustrating an example of a surface selection process.

FIG. 11 is a flowchart illustrating an example of a contour processing process.

FIG. 12 is a flowchart illustrating an example of a color correction process.

FIG. 13 is a flowchart illustrating an example of a process of a position and size correction process.

FIG. 14 is a flowchart illustrating an example of a process for performing automatic continuous processing.

FIG. 15 is a first page of a flowchart showing an example of a process of automatic continuous display.

FIG. 16 is a second page of the flowchart showing an example of the process of automatic continuous display.

FIG. 17 is a diagram illustrating an example of a viewpoint definition screen.

FIG. 18 is a conceptual diagram of viewpoint movement.

FIG. 19 is a diagram showing another example of the viewpoint definition screen.

FIG. 20 is a diagram illustrating an example of a plane definition screen.

FIG. 21 is a conceptual diagram showing a positional relationship of each coordinate system among three viewpoints.

FIG. 22 is a conceptual diagram of three viewpoint interpolation.

FIG. 23 is a diagram showing a rotation matrix.

FIG. 24 is a conceptual diagram of image parallelization.

FIG. 25 is a conceptual diagram of plane selection.

FIG. 26 is an example of an image with a mask.

FIG. 27 is a conceptual diagram of an offset plane.

FIG. 28 is a diagram illustrating weights of ridge lines.

FIG. 29 is an example of an image when performing a composite display.

FIG. 30 is a diagram showing a transformation matrix, a rotation matrix, and a shear deformation matrix.

FIG. 31 is a diagram showing an affine transformation matrix.

FIG. 32 is a conceptual diagram showing a process of calculating a shadow.

FIG. 33 is a diagram showing points of reflection.

FIG. 34 is a diagram showing an example of a basic image.

FIG. 35 is a diagram showing an example of an interpolated image.

FIG. 36 is a conceptual diagram showing a positional relationship of each coordinate system between two viewpoints.

FIG. 37 shows an example of a basic image in which target frame information is set.

FIG. 38 is a conceptual diagram of automatic continuous display.

FIG. 39 is an image diagram of a correction process.

FIG. 40 is a conceptual diagram showing a positional relationship of each coordinate system among three viewpoints.

FIG. 41 is a diagram showing a rotation matrix.

FIG. 42 is a diagram showing intermediate expressions in the parallelization rotation conversion and the horizontal rotation conversion.

FIG. 43 is a diagram showing an intermediate expression in scale conversion.

FIG. 44 is a diagram in which edge end points are automatically extracted.

FIG. 45 is a diagram when a Delaunay triangle designation point is designated.

FIG. 46 is a diagram of a basic image in which Delaunay triangles have been generated.

FIG. 47 is a diagram showing simultaneous linear equations for solving a scalar parameter.

FIG. 48 is a conceptual diagram of an epipole constraint line and a search window.

FIG. 49 is a diagram showing an algorithm for obtaining a luminance difference.

FIG. 50 is a diagram in a case where matching of corresponding points is automatically performed.

FIG. 51 is a diagram showing a linear equation for three images.

FIG. 52 is a diagram illustrating an example of a case where surfaces overlap.

FIG. 53 is a conceptual diagram showing a drawing order of surfaces.

[Explanation of symbols]

 1: Image display system 2: Image capture means 3: Interpolated image creation means 4: Shadow creation means 5: Reflection creation means 6: Created image storage means 7: Composite display means 8: Correction processing means 9: Initial information calculation means 10: Parallelized image creation means 11: Surface selection means 12: Contour processing means 13: Color correction means 14: Server 15: User terminal 16: Network 17: Automatic continuation means 18: Automatic continuation creation means 19: Texture mapping means

 ──────────────────────────────────────────────────続 き Continued on front page F term (reference) 5B050 AA10 BA09 BA13 CA05 CA07 DA07 EA12 EA13 EA19 EA28 FA02 FA06 5B057 AA20 CA08 CA12 CA16 CB08 CB13 CB16 CC04 CD02 CD05 CD14 CE08 CH01 CH11 5B069 AA01 BA03 BB03 DD09 DD03 DD03 DD03 BA05 BA08 FA08 FA16 GA22 5C061 AA21 AB04 AB17

Claims (122)

[Claims] 1. An image display system for automatically creating an interpolated image of an object from a viewpoint desired by a user, comprising: a server having image capturing means for capturing two-dimensional images of the object from a plurality of viewpoints; In an image display system having a user terminal capable of transmitting and receiving data via a network, the user terminal receives the two-dimensional image from the server and receives an interpolation image of an object from a viewpoint desired by the user. An image display system comprising an interpolated image creating means for automatically creating an image. 2. An image display system for automatically creating an interpolated image of an object from a viewpoint desired by a user, wherein the interpolated image automatically creates an interpolated image of the object from a viewpoint desired by the user. In an image display system having a server capable of transmitting and receiving data via a network with a user terminal having a creating unit, the server captures a two-dimensional image of the object from a plurality of viewpoints and converts the two-dimensional image into An image display system, comprising: an image capturing unit that transmits an image to a user terminal. 3. The image capturing means according to claim 1, wherein the image capturing means defines a common vertex and a plane connecting the vertex with a line for the captured two-dimensional image set. 2
The image display system according to 1. 4. The image capturing means extracts an edge end point of the object when defining a common vertex and a plane connecting the vertex with a line in the captured two-dimensional image set. Then, a designated Delaunay triangle is designated in the set of the captured two-dimensional images, and one of the two-dimensional image
A Delaunay triangle is generated in the two-dimensional image, a modification satisfying the epipole constraint is performed from the designated Delaunay triangle, and the corrected linear search is performed. The image display system according to any one of claims 1 to 3, wherein a surface connected by a line is automatically defined. 5. The image capturing means assumes a virtual three-dimensional space coordinate system on the server, polar coordinates of a sphere centered on the origin of the virtual three-dimensional space coordinate system, and an amount of movement of an input device in the server. 5. The image display system according to claim 1, wherein a viewpoint of the two-dimensional image is set by associating the two. 6. An interpolated image creating means for receiving the two-dimensional image from the server and performing a predetermined operation on the received two-dimensional image, based on the result of the operation. 3. The image display system according to claim 1, further comprising: a parallelized image creating unit configured to automatically create a parallelized image by using a texture mapping unit that performs texture mapping on the parallelized image. . 7. The image display according to claim 6, wherein the initial information calculation means includes a calculation of a basic matrix and a calculation of an epipole for the set of two-dimensional images as the predetermined calculation. system. 8. The texture mapping means interpolates vertices of a surface defined in the server into the automatically created parallelized image, performs linear interpolation, and generates a surface inside the vertices of the linearly interpolated surface. 7. The image display system according to claim 6, wherein texture mapping is performed on internal pixels. 9. The apparatus according to claim 6, wherein said texture mapping means further comprises a face selecting means for selecting a texture image for all or some of the defined faces. The image display system according to 1. 10. The surface selecting means, when only one of the surfaces is facing up with respect to the viewpoint designated by the user, selects the surface as a texture image, and the two or more surfaces are facing up. In the case of, calculate the area of the surface facing upward and select the surface having the larger area as a texture image,
And / or the luminance of the image A is A, the luminance of the image B is B, and the ratio of the distance from the image B to the viewpoint at the viewpoint between the image A and the image B is d (0 ≦ d The image display system according to claim 9, wherein, when ≦ 1), the luminance S of the image calculated by S = A × d + B (1−d) is selected as a texture image. 11. The apparatus according to claim 9, wherein said plane selecting means draws in order from a plane which is farther from the epipole.
Or the image display system according to claim 10. 12. The image display system according to claim 6, wherein said texture mapping means further includes a contour processing means for performing contour processing of said surface. 13. An edge processing unit according to claim 1, wherein said contour processing means sets a point at which a vertex coordinate of said surface is averaged to a center of gravity of said surface, and passes through said center of gravity of said surface and is an outward unit vector perpendicular to an edge which is a connection between surface vertices. By calculating the offset vector for each surface, and averaging the calculated edge line offset vector, calculate the vertex offset vector which is the average unit vector of the boundary edge line offset vector including the vertex, for the created interpolation image 13. The contour processing is performed by displaying an offset plane formed by both end vertices of the boundary ridge line and vertices moved in the offset vector direction from the end vertices, and then displaying a normal plane.
The image display system according to 1. 14. The image display system according to claim 6, wherein said texture mapping means further comprises a color correction means for performing a color correction process on said surface. 15. The color correcting means, when the colors of the texture images 0 and 1 corresponding to the pixel point p are C ₀ (p) and C ₁ (p) and the weight of the edge is w, 15. The image display system according to claim 14, wherein the color to be corrected is calculated by calculating p) by Equation (1). ## EQU1 ## C (p) = (1−w) C ₀ (p) + wC ₁ (p) 16. The image capturing means according to claim 3, wherein:
When one set of two-dimensional images is used, when automatically creating the final interpolated image from the viewpoint desired by the user, the interpolated image creating means may include the set of two-dimensional images in the captured two-dimensional image. A primary interpolation image is automatically created using a combination of the two images,
Another primary interpolation image is automatically created using two combinations other than the two combinations in the set, and the user is created using the two primary interpolation images created automatically. 16. The image display system according to claim 1, wherein a final interpolated image from a desired viewpoint is automatically created. 17. The image capturing means according to claim 3, wherein:
When one set of two-dimensional images is used, when automatically creating an interpolated image from the viewpoint desired by the user, the interpolated image creating means uses a rotation matrix using the three captured two-dimensional images. Is calculated, a parallelization rotation conversion is performed using the rotation matrix, a horizontal rotation conversion is performed based on the parallelization rotation conversion, a scale conversion is performed based on the horizontal rotation conversion, and the parallelization rotation conversion is performed. 16. An interpolated image is automatically created by interpolating a homography matrix based on a horizontal rotation conversion and a scale conversion.
The image display system according to any one of the above. 18. The image display system according to claim 1, wherein said user terminal further comprises a shadow creating means for creating a shadow on said created interpolated image. . 19. The image display system according to claim 18, wherein said shadow creating means creates a shadow for the object using an affine transformation matrix. 20. The image display system according to claim 1, wherein said user terminal further comprises reflection creating means for creating a reflection for said created interpolation image. . 21. The image display system according to claim 20, wherein said reflection creating means creates a reflection for an object using an affine transformation matrix. 22. The user terminal restores at least one or more objects from the same viewpoint by restoring the three-dimensional information from at least one or more interpolated images, shadows, reflections, and / or two-dimensional images. 22. The image display apparatus according to claim 1, further comprising a combining display unit for combining the image with the image.
The image display system according to any one of the above. 23. The image display system according to claim 22, wherein projection restoration or Euclidean restoration is used as said restoration. 24. The server according to claim 1, wherein said server further comprises automatic continuation means for setting a path of a viewpoint and a moving speed when performing automatic continuous display of said interpolated image. 23. The image display system according to any one of 23. 25. The automatic continuation means, assuming a virtual three-dimensional space coordinate system on the server, the polar coordinates of a sphere centered at the origin of the virtual three-dimensional space coordinate system, and the movement of an input device in the server. The image display system according to claim 24, wherein a path is set by associating the amount with the amount. 26. The apparatus according to claim 1, wherein said user terminal further comprises an automatic continuation means for setting a path of a viewpoint and a moving speed when performing automatic continuous display of said interpolated image.
The image display system according to any one of claims 1 to 23. 27. The automatic continuation means, assuming a virtual three-dimensional space coordinate system on the user terminal, polar coordinates of a sphere centered on the origin of the virtual three-dimensional space coordinate system, and an input device at the user terminal. 27. The image display system according to claim 26, wherein a route is set by associating the movement amount with the moving amount. 28. The method according to claim 24, wherein said automatic continuation means sets a moving speed on said set route by associating a moving amount of said input device with time. The image display system according to any one of the above. 29. An image display system according to claim 24, wherein said automatic continuation means sets information indicating a range of a target object in said two-dimensional image. . 30. The user terminal further includes automatic continuous creation means for automatically creating and displaying an interpolated image continuously based on a route and a moving speed set on the server or on the user terminal. The image display system according to any one of claims 1 to 29, wherein: 31. A correction processing means, wherein the user terminal calculates an enlargement / reduction magnification and a translation amount based on information indicating a range of the object, and corrects a position and a size of the object with respect to the object. 2. The method according to claim 1, further comprising:
31. The image display system according to any one of claims 30 to 30. 32. The image display system according to claim 31, wherein said correction processing means calculates a range of an interpolated image of said object by Equation 2 and corrects the interpolated image within said range. . [Equation 2] ((total number of frames−number of frames) × information indicating range of start viewpoint + number of frames × information indicating range of end viewpoint) ÷ total number of frames 33. An image display method for automatically creating an interpolated image of an object from a viewpoint desired by a user in a user terminal capable of transmitting and receiving data via a server and a network, wherein the user terminal From the server
An image display method, wherein an interpolated image of an object from a viewpoint desired by the user is automatically created by receiving a two-dimensional image. 34. An image display method in which a server capable of transmitting and receiving data to and from a user terminal via a network has an image display method for automatically creating an interpolated image of an object from a viewpoint desired by the user at the user terminal. And the server is
An image display method, comprising: capturing a two-dimensional image of the object from a plurality of viewpoints, and transmitting the two-dimensional image to the user terminal. 35. The server as claimed in claim 2, wherein the user terminal includes
Before transmitting a two-dimensional image, a plurality of the two-dimensional images are captured, and a common vertex and a surface connecting the vertex with a line are defined for the set of the captured two-dimensional images. 35. The image display method according to claim 33 or 34. 36. The server, when defining a common vertex and a plane connecting the vertex with a line for the set of the captured two-dimensional images, extracts an edge end point of the object, A Delaunay triangle designated point is designated in the set of the captured two-dimensional images, a Delaunay triangle is generated in one two-dimensional image of the set of the two-dimensional images, and an epipole constraint is defined from the designated Delaunay triangle. The method according to claim 1, wherein a common vertex and a plane connecting the vertex with a line are automatically defined for the set of two-dimensional images by performing a correction that satisfies and performing the corrected linear search. The image display method according to any one of claims 33 to 35. 37. The server, when capturing the two-dimensional image, assumes a virtual three-dimensional space coordinate system on the server, and calculates the polar coordinates of a sphere centered on the virtual three-dimensional space coordinate system origin and the server. 37. The image display method according to claim 33, wherein the viewpoint setting of the two-dimensional image is performed by associating the movement amount of the input device with the viewpoint. 38. The user terminal, when automatically creating an interpolated image of an object from a viewpoint desired by the user, receives the two-dimensional image from the server, and receives the two-dimensional image from the server. 35. The image display according to claim 33, wherein a predetermined calculation is performed, a parallelized image is automatically created based on the calculation result, and texture mapping is performed on the parallelized image. Method. 39. The image display method according to claim 38, wherein the predetermined operation includes an operation of a basic matrix and an operation of an epipole for the set of two-dimensional images. 40. When performing the texture mapping,
Interpolating the vertices of the surface defined in the server into the automatically created parallelized image, performing linear interpolation, and texture-mapping the pixels inside the surface inside the vertices of the linearly interpolated surface. 39. The image display method according to claim 38. 41. When performing the texture mapping,
4. The method according to claim 3, further comprising the step of selecting a texture image for all or some of the defined surfaces.
The image display method according to any one of claims 8 to 40. 42. In the process of selecting the texture image, if only one surface of the surface is facing up to a viewpoint designated by the user, the surface is selected as a texture image. When the above-mentioned surface is facing up, the area of the facing surface is calculated and the surface having the larger area is selected as a texture image, and / or the brightness of the image A is A, the brightness of the image B is B, In the viewpoint between the image A and the image B, when the ratio of the distance from the image B to the viewpoint is d (0 ≦ d ≦ 1), S = A × d + B (1-d) 42. The image display method according to claim 41, wherein a luminance S of the image to be processed is selected as a texture image. 43. The image display method according to claim 41, wherein, in the process of selecting the texture image, drawing is performed in order from a surface farther from the epipole. 44. When performing the texture mapping,
4. The method according to claim 3, further comprising performing contour processing on the surface.
The image display method according to any one of claims 8 to 43. 45. An outward unit vector perpendicular to a ridge line passing through the center of gravity of the surface and taking a point at which the coordinates of the vertices of the surface are averaged in the contour processing as a center of gravity of the surface. The edge offset vector is calculated for each surface, and the calculated edge line offset vector is averaged to calculate a vertex offset vector which is an average unit vector of the boundary edge line offset vector including the vertex. 45. Contrast processing is performed by displaying an offset surface composed of vertices at both ends of a boundary ridge line and vertices moved in the direction of an offset vector from the boundary surface, and then displaying a normal surface.
The image display method described in 1. 46. When performing the texture mapping,
The image display method according to any one of claims 38 to 45, further comprising performing a color correction process on the surface. 47. When performing the color correction processing, the colors of the texture images 0 and 1 corresponding to the pixel points p are represented by C ₀ (p) and C ₁
(P) and the weight of the ridgeline is w, the color C to be corrected
47. The image display method according to claim 46, wherein the color to be corrected is calculated by calculating (p) by Expression 1, and the color is corrected. 48. In the case where three sets of the two-dimensional images are set as one set in the server, when the user automatically creates a final interpolated image from a viewpoint desired by the user, the user terminal performs A primary interpolated image is automatically created using a combination of two images in a set of the captured two-dimensional images, and a two-image combination other than the combination of the two images in the set is used. 5. The method according to claim 1, wherein another primary interpolation image is automatically created, and a final interpolation image from a viewpoint desired by the user is automatically created using the two automatically created primary interpolation images. The image display method according to any one of claims 33 to 47. 49. In the case where three sets of the two-dimensional images are set as one set in the server, when automatically creating an interpolated image from a viewpoint desired by the user, the user terminal sets the A rotation matrix is calculated using the three two-dimensional images, a parallelization rotation conversion is performed using the rotation matrix, a horizontal rotation conversion is performed based on the parallelization rotation conversion, and the horizontal rotation conversion is performed. 48. The method according to claim 33, wherein a scale conversion is performed on the basis of the parallelization rotation conversion, the horizontal rotation conversion, and the homography matrix based on the scale conversion, and an interpolation image is automatically created. The image display method according to any of the above. 50. The user terminal according to claim 33, wherein, after creating the interpolated image, the user terminal performs a process of creating a shadow on the created interpolated image. Image display method described. 51. The image display method according to claim 50, wherein said shadow creating process creates a shadow for an object using an affine transformation matrix. 52. The user terminal according to claim 33, further comprising, after creating said interpolated image, a process of creating a reflection on said created interpolated image.
The image display method according to any one of the above. 53. The image display method according to claim 52, wherein said processing for creating a reflection creates a reflection for an object using an affine transformation matrix. 54. The user terminal restores at least one or more objects from the same viewpoint by restoring the three-dimensional information from at least one or more interpolated images, shadows, reflections, and / or two-dimensional images. The image display method according to any one of claims 33 to 52, further comprising the step of combining the image with the image. 55. The image display method according to claim 54, wherein projection restoration or Euclidean restoration is used as said restoration. 56. The server according to claim 33, wherein said server further performs an automatic continuous process for setting a path of a viewpoint and a moving speed when performing automatic continuous display of said interpolated image. Image display method described in Crab. 57. When performing the automatic continuous processing, a virtual 3
Assuming a three-dimensional space coordinate system on the server, and setting a route by associating polar coordinates of a sphere centered on the origin of the virtual three-dimensional space coordinate system with a movement amount of an input device in the server. 57. The image display method according to claim 56, wherein: 58. The user terminal according to claim 33, wherein said user terminal further performs an automatic continuous process for setting a viewpoint path and a moving speed when performing automatic continuous display of said interpolated image. The image display method according to any of the above. 59. When performing the automatic continuous processing, a virtual 3
A path is set by assuming a three-dimensional space coordinate system on the user terminal and associating the polar coordinates of a sphere centered on the origin of the virtual three-dimensional space coordinate system with the movement amount of the input device in the user terminal. 58. The method of claim 58, wherein
The image display method described in 1. 60. When the automatic continuous processing is performed, the moving speed of the set route is set by associating the moving amount of the input device with time. The image display method according to claim 59. 61. When the automatic continuous processing is performed,
61. The image display method according to claim 56, wherein information indicating a range of the target in the two-dimensional image is set. 62. The user terminal further performs an automatic continuous reproduction process for automatically creating and displaying an interpolated image continuously based on a route and a moving speed set on the server or on the user terminal. The image display method according to any one of claims 33 to 61, wherein: 63. A method according to claim 63, wherein the user terminal calculates an enlargement / reduction magnification and a translation amount based on the information indicating the range of the object, and performs a position and size correction process on the object. The image display method according to any one of claims 33 to 62. 64. The method according to claim 63, wherein, when performing the correction processing, a range of the interpolated image of the object is calculated by Expression 2 and the interpolated image is corrected within the range. Image display method. 65. An image display system for automatically creating an interpolated image on a server for automatically creating and viewing an interpolated image of an object from a viewpoint desired by a user, wherein the server comprises: An image display system comprising: an image capturing unit that captures a two-dimensional image of an object from a plurality of viewpoints; and an interpolated image creating unit that automatically creates an interpolated image of the object from the desired viewpoint. . 66. The image capturing means, comprising:
The image display system according to claim 65, wherein a common vertex and a plane connecting the vertex with a line are defined for the set of the dimensional images. 67. The image capturing means according to claim 42, wherein:
When defining a common vertex and a plane connecting the vertices with a line for a set of two-dimensional images, an edge end point of the object is extracted, and a Delaunay triangle is designated in the set of the captured two-dimensional image. A point is designated, and one of the two-dimensional image sets 2
A Delaunay triangle is generated in the two-dimensional image, a modification satisfying the epipole constraint is performed from the designated Delaunay triangle, and the corrected linear search is performed. 67. The image display system according to claim 65 or 66, wherein a surface which is connected by a line is automatically defined. 68. The image capturing means, assuming a virtual three-dimensional space coordinate system on the server, polar coordinates of a sphere centered on the origin of the virtual three-dimensional space coordinate system, and an input device of the server. By associating the amount of movement,
65. A viewpoint setting of a two-dimensional image is performed.
The image display system according to any one of claims 1 to 67. 69. An interpolation information creating means for performing a predetermined operation on the two-dimensional image, and a parallelized image creation means for automatically creating a parallelized image based on the operation result. The image display system according to claim 65, further comprising: a texture mapping unit configured to perform texture mapping on the parallelized image. 70. The image display according to claim 69, wherein said initial information calculation means includes a calculation of a basic matrix and a calculation of an epipole for said set of two-dimensional images as said predetermined calculation. system. 71. The texture mapping means interpolates the vertices of the defined surface into the automatically created parallelized image, performs linear interpolation, and assigns pixels inside the surface to the vertices of the linearly interpolated surface. The image display system according to claim 70, wherein mapping is performed. 72. The apparatus according to claim 69, wherein said texture mapping means further comprises a face selecting means for selecting a texture image for all or some of the defined faces. Image display system as described. 73. When only one surface of the surface is facing the viewpoint specified by the user, the surface selecting means selects that surface as a texture image, and the two or more surfaces face up. In the case of, calculate the area of the surface facing upward and select the surface having the larger area as a texture image,
And / or the luminance of the image A is A, the luminance of the image B is B, and the ratio of the distance from the image B to the viewpoint at the viewpoint between the image A and the image B is d (0 ≦ d 73. The image display system according to claim 72, wherein, when ≤ 1), the luminance S of the image calculated by S = A x d + B (1-d) is selected as a texture image. 74. The apparatus according to claim 7, wherein the plane selecting means draws the plane in order from a plane farther from the epipole.
An image display system according to claim 2 or 73. 75. The image display system according to claim 69, wherein said texture mapping means further comprises a contour processing means for performing contour processing of said surface. 76. The contour processing means, wherein a point at which a vertex coordinate of the surface is averaged is set as a center of gravity of the surface, and a ridge line which is an outward unit vector passing through the center of gravity of the surface and perpendicular to a ridge line connecting the surface vertices. By calculating the offset vector for each surface, by averaging the calculated edge line offset vector, calculate the vertex offset vector which is the average unit vector of the boundary edge line offset vector including the vertex, for the created interpolation image 76. The contour processing is performed by displaying an offset plane formed by both end vertices of the boundary ridge line and vertices moved in the offset vector direction from the end face, and then displaying a normal plane.
The image display system according to 1. 77. The image display system according to claim 69, wherein said texture mapping means further comprises a color correction means for performing a color correction process on said surface. 78. If the colors of the texture images 0 and 1 corresponding to the pixel point p are C ₀ (p) and C ₁ (p) and the weight of the ridge line is w, the color correction means corrects the color C ( 78. The image display system according to claim 77, wherein a color to be corrected is calculated by calculating p) by Equation 1, and the color is corrected. 79. In the image taking means, three sheets of the 2
When one set of two-dimensional images is used, when automatically creating the final interpolated image from the viewpoint desired by the user, the interpolated image creating means may include the set of two-dimensional images in the captured two-dimensional image. A primary interpolation image is automatically created using a combination of the two images,
Another primary interpolation image is automatically created using two combinations other than the two combinations in the set, and the user is created using the two primary interpolation images created automatically. The image display system according to any one of claims 65 to 78, wherein automatically generates a final interpolated image from a desired viewpoint. 80. In the image taking means, three sheets of the 2
When one set of two-dimensional images is used, when automatically creating an interpolated image from the viewpoint desired by the user, the interpolated image creating means uses a rotation matrix using the three captured two-dimensional images. Is calculated, a parallelization rotation conversion is performed using the rotation matrix, a horizontal rotation conversion is performed based on the parallelization rotation conversion, a scale conversion is performed based on the horizontal rotation conversion, and the parallelization rotation conversion is performed. The interpolated image is automatically created by interpolating a homography matrix based on an image, a horizontal rotation conversion, and a scale conversion.
9. The image display system according to any one of 8. 81. The image display system according to claim 65, wherein said server further comprises shadow creating means for creating a shadow on said created interpolated image. 82. The image display system according to claim 81, wherein said shadow creating means creates a shadow for the object using an affine transformation matrix. 83. The image display system according to claim 65, wherein said server further comprises reflection creating means for creating a reflection on said created interpolated image. 84. The image display system according to claim 83, wherein said reflection creating means creates a reflection for an object using an affine transformation matrix. 85. The server restores at least one or more objects from the same viewpoint by restoring the three-dimensional information from at least one or more interpolated images, shadows, reflections and / or two-dimensional images. The image display system according to any one of claims 65 to 84, further comprising a combination display unit for combining the image with the image. 86. The image display system according to claim 85, wherein projection restoration or Euclidean restoration is used as said restoration. 87. The apparatus according to claim 65, wherein said server further comprises automatic continuation means for setting a path of a viewpoint and a moving speed when performing automatic continuous display of said interpolated image. 86. The image display system according to any one of 86. 88. The automatic continuation means, assuming a virtual three-dimensional space coordinate system on the server, and polar coordinates of a sphere centered at the origin of the virtual three-dimensional space coordinate system and movement of an input device in the server. The image display system according to claim 87, wherein a path is set by associating the amount with the amount. 89. The apparatus according to claim 87, wherein said automatic continuation means sets a moving speed on said set route by associating a moving amount of said input device with time. Image display system as described. 90. The image display system according to claim 87, wherein said automatic continuation means sets information indicating a range of a target in said two-dimensional image. . 91. The server according to claim 65, wherein said server further comprises automatic continuous creation means for automatically creating and displaying an interpolated image continuously based on said set route and moving speed. 90. The image display system according to claim 90. 92. A correction processing means for calculating an enlargement / reduction magnification and a translation amount based on information indicating the range of the object and performing a correction process of a position and a size with respect to the object. The image display system according to any one of claims 65 to 91, further comprising: 93. The image display system according to claim 92, wherein said correction processing means calculates a range of an interpolated image of said object by Equation 2 and corrects the interpolated image within said range. . 94. An image display method for automatically creating an interpolated image on a server for automatically creating and viewing an interpolated image of an object from a viewpoint desired by a user, wherein the server comprises: An image display method, comprising taking in a two-dimensional image of an object from a plurality of viewpoints and automatically creating an interpolated image of the object from the desired viewpoint. 95. The image display method according to claim 94, wherein said server defines a common vertex and a plane connecting said vertex with a line for said set of fetched two-dimensional images. . 96. The server, when defining a common vertex and a plane connecting the vertex with a line for the set of the captured two-dimensional images, extracts an edge end point of the object, A Delaunay triangle designation point is specified in the set of the captured two-dimensional images, a Delaunay triangle is generated in one of the two-dimensional images in the set of the two-dimensional images, and an epipole constraint is defined from the Delaunay triangle specification point. The method according to claim 1, wherein a common vertex and a plane connecting the vertex with a line are automatically defined for the set of two-dimensional images by performing a correction that satisfies and performing the corrected linear search. The image display method according to claim 94 or 95. 97. The server, when capturing the two-dimensional image, assumes a virtual three-dimensional space coordinate system on the server, and calculates polar coordinates of a sphere centered on the origin of the virtual three-dimensional space coordinate system and the server. The image display method according to any one of claims 94 to 96, wherein the viewpoint setting of the two-dimensional image is performed by associating the movement amount of the input device with the moving amount of the input device. 98. When automatically creating an interpolated image of an object from a viewpoint desired by the user, the server
The image according to claim 94, wherein a predetermined calculation is performed on the two-dimensional image, a parallelized image is automatically created based on the calculation result, and texture mapping is performed on the parallelized image. Display method. 99. The image display method according to claim 98, wherein said predetermined operation includes an operation of a basic matrix and an operation of an epipole for said set of two-dimensional images. 100. When the texture mapping is performed, vertices of the defined surface are interpolated into the automatically generated parallelized image, linear interpolation is performed, and pixels inside the surface are included in the vertices of the linearly interpolated surface. The image display method according to claim 99, wherein texture mapping is performed. 101. The method according to claim 98, wherein when performing the texture mapping, a process of selecting a texture image for all or some of the defined surfaces is further performed. Image display method described. 102. In the process of selecting the texture image, if only one surface of the surface is facing the viewpoint specified by the user, the surface is selected as a texture image. When the above-mentioned surface is facing up, the area of the facing surface is calculated and the surface having the larger area is selected as a texture image, and / or the brightness of the image A is A, the brightness of the image B is B, In the viewpoint between the image A and the image B, when the ratio of the distance from the image B to the viewpoint is d (0 ≦ d ≦ 1), S = A × d + B (1-d) 102. The image display method according to claim 101, wherein a luminance S of the image to be processed is selected as a texture image. 103. The image display method according to claim 101, wherein, in the process of selecting the texture image, drawing is performed in order from a surface far from the epipole. 104. The image display method according to any one of claims 98 to 103, wherein, when performing said texture mapping, contour processing of said surface is further performed. 105. In the contour processing, a point at which a vertex coordinate of the surface is averaged is defined as a center of gravity of the surface, and is an outward unit vector which passes through the center of gravity of the surface and is perpendicular to a ridge line which is a connection between surface vertices. The edge offset vector is calculated for each surface, and the calculated edge line offset vector is averaged to calculate a vertex offset vector which is an average unit vector of the boundary edge line offset vector including the vertex. 11. The contour processing is performed by displaying an offset plane composed of both end vertices of the boundary ridge line and vertices moved in the offset vector direction from the end vertices, and then displaying a normal plane.
4. The image display method according to 4. The image display method according to any one of claims 98 to 105, wherein a color correction process for said surface is further performed when said texture mapping is performed. 107. When performing the color correction processing, a pixel point p
The colors of the texture images 0 and 1 corresponding to C ₀ (p), C
₁ (p) and the weight of the ridgeline is w, the color C to be corrected
107. The image display method according to claim 106, wherein the color to be corrected is calculated by calculating (p) by Expression 1, and the color is corrected. 108. In the case where the server sets three two-dimensional images as one set, when automatically creating a final interpolated image from a viewpoint desired by the user, the acquired two-dimensional image is used. 1 using the combination of two images in the set of images
A next interpolated image is automatically created, another primary interpolated image is automatically created by using two combinations other than the two combinations in the set, and the two automatically created ones are used. 95. The final interpolated image from the viewpoint desired by the user is automatically created using the next interpolated image.
07. The image display method according to any one of the above aspects. In the case where the server sets three two-dimensional images as one set, when automatically creating an interpolated image from a viewpoint desired by the user, the acquired three-dimensional image is used.
A rotation matrix is calculated using the two two-dimensional images, a parallelization rotation conversion is performed using the rotation matrix, a horizontal rotation conversion is performed based on the parallelization rotation conversion, and a horizontal rotation conversion is performed based on the horizontal rotation conversion. Perform a scale conversion, the parallelization rotation conversion,
108. The image display method according to claim 94, wherein an interpolated image is automatically created by interpolating a homography matrix based on horizontal rotation conversion and scale conversion. 110. The server according to claim 94, wherein, after creating said interpolated image, said server performs a process of creating a shadow on said created interpolated image. Image display method. 111. The image display method according to claim 110, wherein said shadow creating process creates a shadow for an object using an affine transformation matrix. 112. The server according to claim 94, wherein, after creating said interpolated image, said server further performs a process of creating a reflection on said created interpolated image.
10. The image display method according to any one of items 9. 113. The image display method according to claim 112, wherein said reflection creating process creates a reflection for an object using an affine transformation matrix. 114. The server restores at least one or more objects from the same viewpoint by restoring the three-dimensional information from at least one or more interpolated images, shadows, reflections and / or two-dimensional images. The image display method according to any one of claims 94 to 113, further comprising a step of combining the image with the image. 115. A method according to claim 114, wherein projection restoration or Euclidean restoration is used as said restoration.
The image display method described in 1. 116. The server according to claim 94, wherein said server further performs an automatic continuous process for setting a path of a viewpoint and a moving speed when performing automatic continuous display of said interpolated image. Image display method described in Crab. 117. When performing the automatic continuous process, a virtual three-dimensional space coordinate system is assumed on the server,
117. The image display method according to claim 116, wherein a route is set by associating polar coordinates of a sphere centered on the origin of a dimensional space coordinate system with a movement amount of an input device in the server. 118. When the automatic continuous processing is performed, the moving speed of the set route is set by associating the moving amount of the input device with time. The image display method according to claim 117. 119. The information processing apparatus according to claim 116, wherein when the automatic continuous processing is performed, information indicating a range of an object in the two-dimensional image is set. Image display method. 120. The server according to claim 94, wherein said server further performs an automatic continuous reproduction process for automatically creating and displaying an interpolated image continuously based on said set route and moving speed. 119. The image display method according to any one of [119]. 121. The server calculates the enlargement / reduction magnification and the amount of parallel movement based on the information indicating the range of the object, and performs a position and size correction process on the object. The image display method according to any one of claims 94 to 120. 122. The apparatus according to claim 121, wherein, when performing said correction processing, a range of an interpolated image of said object is calculated by Expression 2 and the interpolated image is corrected within said range. Image display method.