JP2006350852A - Image generation system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image generation system for generating and displaying a perspective projection image from a position designated in a virtual three-dimensional space on the basis of at least two panoramic images. <P>SOLUTION: The image generation system for generating a perspective projection image from a position designated by a user in a virtual three-dimensional space on the basis of at least two panoramic images comprises: a panoramic image receiving means for reading a panoramic image; a position input receiving means for receiving information of a position designated by the user in the virtual three-dimensional space; an image generation operation processing means for generating a perspective projection image at a received position from the at least two read panoramic images; and a display processing means for outputing the generated perspective projection image. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

The present invention relates to an image generation system that generates and displays a perspective projection image from a specified position in a virtual three-dimensional space based on at least two panoramic images.

  A model such as VRML (Virtual Reality Modeling Language) is used as a method of displaying an image (perspective projection image) when viewing the 3D space from a position specified by the user in a virtual 3D space on a computer terminal. There are a base method and an image based method using a real image.

  In the model-based method, in order to generate a perspective projection image from an arbitrary position in the virtual three-dimensional space, a three-dimensional shape model of each object in the virtual three-dimensional space is usually created by a polygon or the like. This is done by defining a texture suitable for the model or defining light data so that the brightness and color are appropriate. When the user designates an arbitrary position in the virtual three-dimensional space where each object is created as described above, a perspective projection image from that position is generated and displayed.

  On the other hand, in the image-based method, based on a plurality of still images, an entire surrounding image from one fixed point in a virtual three-dimensional space is generated and displayed, or images at a number of consecutive points are displayed as if There are things such as moving images, which are displayed by reproducing images from arbitrary points using video compression / playback technology and related technologies.

  The following six patent documents are shown as examples representing these conventional techniques.

JP 2003-187261 A JP 2003-24624 A JP 2002-52240 A JP 2001-229402 A JP 2000-67274 A JP 10-49704 A

  However, in the case of the model-based method, an arbitrary perspective projection image in the virtual three-dimensional space can be generated, but it is very difficult to create each object arranged in the virtual three-dimensional space. In particular, the creation of a three-dimensional shape model is usually performed by a skilled engineer using dedicated software, and it is difficult for a general user to perform the operation. A lot of know-how is required for the generation and arrangement of textures and lights as well. Therefore, it takes a lot of time and a lot of money to create the data (scene data) that is the basis for using this method.

  In addition, in order to make the scene data display quality equivalent to that of a live-action still image, it is necessary to create a high-definition three-dimensional shape model. Since such a model has a large amount of data, more time and cost are required. In addition, when data is transmitted / received via the Internet or the like, the communication speed and display speed are reduced.

  On the other hand, in the case of an image-based method using a real image, high-definition image data can be obtained without creating a three-dimensional shape model. However, with the prior art in this method, for example, QuickTimeVR can only display the entire surrounding image from a fixed point in the virtual three-dimensional space, and in principle, the position (viewpoint) can be moved. I can't. There is a method that can be moved, but in that method, the user can specify only the marker displayed in the image and jump to another position. Cannot move. In addition, when displaying an image from an arbitrary position, there are various restrictions such as the need for camera calibration or special equipment. A perspective projection image from an arbitrary position cannot be generated and displayed.

  Therefore, the present inventor can generate a perspective projection image from a position designated by the user in the virtual three-dimensional space by using at least two panoramic images in the virtual three-dimensional space, and perform the processing. The present invention has invented an image generation system capable of continuously displaying a perspective projection image by performing continuously.

  A first invention is an image generation system that generates a perspective projection image from a position designated by a user in a virtual three-dimensional space based on at least two panoramic images, and the image generation system includes the panorama image. A panoramic image receiving means for reading an image, a position input receiving means for receiving information on a position designated by the user in the virtual three-dimensional space, and at the received position from the read at least two panoramic images. An image generation system includes an image generation calculation processing unit that generates a perspective projection image and a display processing unit that outputs the generated perspective projection image.

  With the configuration according to the present invention, it is possible to output a perspective projection image from an arbitrary position in the virtual three-dimensional space based on at least two or more panoramic images.

  In the first invention, in order to generate a perspective projection image, an image generation system using the following method may be used. That is, the image generation calculation processing unit includes an intermediate panorama image generation unit and a perspective projection image generation unit, and the intermediate panorama image generation unit is configured to specify a shooting target common to the panorama image. Based on the point information, the triangulation method calculates where the corresponding point in the panoramic image is located in the intermediate panoramic image, and the texture of the face constituted by the corresponding point in the panoramic image Is generated by texture mapping onto the face that has the calculated position in the intermediate panorama image corresponding to the face as a vertex, and the perspective projection image generation means generates the perspective projection image. Divide the image area into a plurality of arbitrary faces, and the vertices that make up the face become the intermediate panorama image. The texture of the face constituted by vertices of the calculated position in the intermediate panorama image is mapped to the corresponding face divided in the perspective projection image. This is an image generation system that generates the perspective projection image.

  With such a configuration, it is possible to generate an intermediate panorama image from a position input by the user and output a perspective projection image based on the intermediate panorama image.

  In the first invention, as another method for generating a perspective projection image, an image generation system using the following method may be used. That is, the image generation calculation processing means includes corresponding point calculation means and perspective projection image generation means, and the corresponding point calculation means calculates the corresponding corresponding points of the photographing object common to the panoramic image. Based on the information, the triangulation method calculates where the corresponding point in the panoramic image is located in the intermediate panoramic image, converts the calculated position into a position on the perspective projection image, The perspective projection image generation means uses the position on the perspective projection image of the corresponding point of the calculated panoramic image as a vertex, and texture-maps the texture corresponding to the panoramic image on the face constituted by the vertex, thereby An image generation system for generating a perspective projection image.

  With such a configuration, the position of the corresponding point in the intermediate panorama image from the position input by the user can be calculated, and the corresponding point can be further converted into a perspective projection image. As a result, texture mapping is performed only once, so that a high-speed processing and high-accuracy perspective projection image can be generated.

  In the first invention, as another method for generating a perspective projection image, an image generation system using the following method may be used. In other words, the image generation calculation processing means includes a spatial position calculation means and a perspective projection image generation means, and the spatial position calculation means determines the corresponding corresponding points of the photographing object common to the panoramic image. Based on the information, the position of the corresponding point in the virtual three-dimensional space is determined on a half line that is a possible range of the virtual three-dimensional space position of the imaging object corresponding to the corresponding point, The projection image generating means converts the position of the corresponding point in the determined panoramic image in the virtual three-dimensional space to the coordinate of the intermediate panoramic image at the received position, and the converted coordinate is the fluoroscopic image. The position corresponding to the calculated panoramic image is converted into a position on the perspective image, and the texture corresponding to the panoramic image is applied to the face constituted by the vertex. By catcher mapping, it generates the perspective projection image, an image generation system.

  Even with such a configuration, it is possible to generate a high-speed and high-accuracy perspective projection image by performing texture mapping only once.

  In the first invention, as another method for generating a perspective projection image, an image generation system using the following method may be used. In other words, the image generation calculation processing means includes a spatial position calculation means and a perspective projection image generation means, and the spatial position calculation means determines the corresponding corresponding points of the photographing object common to the panoramic image. Based on the information, the position of the corresponding point in the virtual three-dimensional space is determined on a half line that is a possible range of the virtual three-dimensional space position of the imaging object corresponding to the corresponding point, The projection image generation means converts the position of the corresponding point in the determined panoramic image in the virtual three-dimensional space into the coordinate of the intermediate panoramic image at the received position, and the corresponding in the panoramic image Texture mapping of the face composed of points is texture-mapped to the face constructed with the converted position in the intermediate panorama image corresponding to this face as a vertex. To generate the intermediate panorama image, divide the region of the perspective projection image into an arbitrary plurality of faces, calculate the position of the vertex constituting the face in the intermediate panorama image, The perspective projection image is generated by texture-mapping the texture of the face formed by using the calculated position in the intermediate panorama image as a vertex to the corresponding face divided in the perspective projection image. An image generation system.

  Even with this configuration, an intermediate panorama image from a position input by the user can be generated and a perspective projection image can be output based on the intermediate panorama image, as in the first invention.

  In the first invention, as another method for generating a perspective projection image, an image generation system using the following method may be used. In other words, the image generation calculation processing means includes a spatial position calculation means and a perspective projection image generation means, and the spatial position calculation means determines the corresponding corresponding points of the photographing object common to the panoramic image. Based on the information, the position of the corresponding point in the virtual three-dimensional space is determined on a half line that is a possible range of the virtual three-dimensional space position of the imaging object corresponding to the corresponding point, The projection image generation means constitutes a three-dimensional polygon having a face formed by a face constituted by the determined position of the corresponding point in the virtual three-dimensional space, and the texture of the face of the panoramic image is represented by the three-dimensional The three-dimensional polygon is generated by texture mapping to the corresponding face of the polygon, and the three-dimensional polygon is converted into a perspective projection image from the received position. Generating a perspective projection image, an image generation system.

  Even with this configuration, an intermediate panorama image from a position input by the user can be generated and a perspective projection image can be output based on the intermediate panorama image, as in the first invention. In this case, since the data can be made into three-dimensional data, the data can be easily handled.

  In the first invention, as another method for generating a perspective projection image, an image generation system using the following method may be used. In other words, the image generation calculation processing means includes a spatial position calculation means and a perspective projection image generation means, and the spatial position calculation means determines the corresponding corresponding points of the photographing object common to the panoramic image. Based on the information, the position of the corresponding point in the virtual three-dimensional space is used as a predetermined interpolation parameter between the half lines that are the possible range of the virtual three-dimensional space position of the object to be imaged corresponding to the corresponding point. A position to be internally divided is determined as a position in the virtual three-dimensional space, and the perspective projection image generation unit is configured to determine the position in the virtual three-dimensional space of the corresponding point in the determined panoramic image as the received position. The coordinates of the intermediate panorama image in the image, the converted coordinates are converted into positions on the perspective projection image, the position of the corresponding point of the calculated panorama image on the perspective projection image as a vertex, Ri to constituted face, by texture mapping a corresponding texture of the panorama image, and generates the perspective projection image, an image generation system.

  Even with such a configuration, it is possible to generate a high-speed and high-accuracy perspective projection image by performing texture mapping only once. In calculating the position of the corresponding point in the virtual three-dimensional space, the half line may not have an intersection. In such a case, it is effective to use the processing as in the present invention.

  In the first invention, as another method for generating a perspective projection image, an image generation system using the following method may be used. In other words, the image generation calculation processing means includes a spatial position calculation means and a perspective projection image generation means, and the spatial position calculation means determines the corresponding corresponding points of the photographing object common to the panoramic image. Based on the information, the position of the corresponding point in the virtual three-dimensional space is used as a predetermined interpolation parameter between the half lines that are the possible range of the virtual three-dimensional space position of the object to be imaged corresponding to the corresponding point. A position to be internally divided is determined as a position in the virtual three-dimensional space, and the perspective projection image generation unit is configured to determine the position in the virtual three-dimensional space of the corresponding point in the determined panoramic image as the received position. Is converted to the coordinates of the intermediate panorama image, and the texture of the face constituted by the corresponding points in the panorama image is changed to the previous in the intermediate panorama image corresponding to the face. The intermediate panorama image is generated by texture-mapping the converted position on the face constituting the vertex, the region of the perspective projection image is divided into a plurality of arbitrary faces, and the vertex constituting the face is the intermediate panorama The position of the face calculated by calculating the position in the image and having the calculated position in the intermediate panorama image as a vertex is divided into the corresponding face divided in the perspective projection image. The image generation system generates the perspective projection image by texture mapping.

  Even with this configuration, an intermediate panorama image from a position input by the user can be generated and a perspective projection image can be output based on the intermediate panorama image, as in the first invention. In calculating the position of the corresponding point in the virtual three-dimensional space, the half line may not have an intersection. In such a case, it is effective to use the processing as in the present invention.

  In the first invention, as another method for generating a perspective projection image, an image generation system using the following method may be used. In other words, the image generation calculation processing means includes a spatial position calculation means and a perspective projection image generation means, and the spatial position calculation means determines the corresponding corresponding points of the photographing object common to the panoramic image. Based on the information, the position of the corresponding point in the virtual three-dimensional space is used as a predetermined interpolation parameter between the half lines that are the possible range of the virtual three-dimensional space position of the object to be imaged corresponding to the corresponding point. A position to be internally divided is determined as a position in the virtual three-dimensional space, and the perspective projection image generation means uses a face constituted by the position of the determined corresponding point in the virtual three-dimensional space as a plane. Forming a three-dimensional polygon, generating the three-dimensional polygon by texture mapping the texture of the face of the panoramic image to the corresponding face of the three-dimensional polygon, Generating said perspective projection image by converting the dimension polygons perspective projection image from the received position, an image generation system.

  Even with this configuration, an intermediate panorama image from a position input by the user can be generated and a perspective projection image can be output based on the intermediate panorama image, as in the first invention. In this case, since the data can be made into three-dimensional data, the data can be easily handled. Further, in calculating the position of the corresponding point in the virtual three-dimensional space, the half line may not have an intersection. In such a case, it is effective to use the processing as in the present invention.

  In the first invention, the image generation calculation processing means generates a panoramic image in which distortion in the panoramic image is corrected from the read at least two panoramic images, and based on the corrected at least two panoramic images. Thus, an image generation system that generates a perspective projection image at the received position may be used.

  Since a panoramic image is an image in a spherical coordinate system or a cylindrical coordinate system, unlike a perspective projection image, the image is distorted. Therefore, if a perspective projection image generation process is performed based on a panoramic image with such distortion, the texture mapping is performed with the distorted texture even during texture mapping, and the perspective projection image is also distorted. End up.

  Therefore, in order to solve such a problem, in addition to the above-described basic processing process, the panorama image read by the panorama image receiving unit is not directly processed by the image generation calculation processing unit. Instead, the processing for correcting the distortion of the panoramic image may be performed before that, and the above-described perspective projection image generation processing may be performed by the image generation calculation processing means 4 using the panoramic image that has been subjected to the correction processing. In this way, a beautiful perspective projection image with little distortion can be generated.

  In each of the above-described inventions, the image generation system may be an image generation system that performs texture mapping of a texture according to a ratio from a position received by the position input reception unit to a position of each panoramic image. I can do it.

  As described above, when texture mapping is performed at a position designated by the user in accordance with the ratio of points that internally divide between panoramic images, a natural perspective projection image can be obtained.

  In each of the above-described inventions, the position input receiving means sequentially receives the position information, and the image generation calculation processing means is based on the received position information, and the perspective projection image at each position. Can be sequentially generated, and the display processing unit can be an image generation system that outputs the generated perspective projection image.

  With this configuration, the perspective projection images are generated and output sequentially, so that they can be output continuously as if they were moving images, and the user can interactively change the position and viewing direction.

  In each of the above-described inventions, when the position input receiving means receives a position in the morphing section in the morphing section composed of the at least two panoramic images, the image generation calculation processing means It is also possible to provide an image generation system that generates a perspective projection image in a morphing section based on panoramic images located at both end points of the morphing section.

  When there are a plurality of morphing sections, it may be configured as in the present invention.

  In each of the above-described inventions, when the morphing section is switched, the image generation calculation processing means converts the interpolation parameter in the morphing section that is the current processing target to the interpolation parameter of the morphing section that is the next processing target. In addition, an image generation system that generates the perspective projection image based on the panoramic image can be provided.

  When there are a plurality of morphing sections, a smoother perspective projection image can be output when the morphing sections are switched by configuring as in the present invention.

  In each of the above-described inventions, the image generation system performs a process of synthesizing the perspective projection image generated by the image generation calculation processing unit with a photographed image, a CG image, or an image obtained by another image generation method. It is also possible to provide an image generation system in which the combined image is output by the display processing means.

  By providing such a process for combining, the usage applications are expanded.

  According to the present invention, a general user can easily generate and display a perspective projection image from a position designated by a user in a virtual three-dimensional space. This eliminates the need for creating a conventional three-dimensional shape model or creating scene data. Further, as in the image-based method, it is possible to perform continuous and free movement without using special equipment or the like, not discontinuous movement.

In addition, since a three-dimensional image is generated and displayed by generating an intermediate image from the original panoramic image of the two viewpoints, the amount of data can be extremely small even with the display quality of the photographed image.

  In the image generation system 1 of the present invention, the processing is realized on a computer terminal (including a server) having an arithmetic device 11 such as a CPU and a storage device 12 such as a RAM or a hard disk. The computer terminal may further include an input device 10 such as a keyboard and a mouse, a display device 14 for display, and a communication device 13 that transmits / receives data to / from other computer terminals via a network such as the Internet.

  A program that realizes the image generation system 1 is recorded on the hard disk of the computer terminal, and is read into a RAM or the like when a predetermined operation is performed, and the function (means) is executed by the arithmetic unit 11. Then, the process is performed. Further, when the image generation system 1 of the present invention is realized on a browser by a JAVA applet or the like, the above-described hardware configuration used by a user from a predetermined server by a program (applet) for realizing the image generation system 1 Is downloaded to a computer terminal and stored in the storage device 12. Then, the program is read from the storage device 12 and executed by the arithmetic device 11, whereby the image generation system 1 of the present invention can be realized on the browser.

  Information used in these processes is stored in the storage device 12 such as a RAM or a hard disk in various file storage formats such as a database and a data file, and is processed by being read out appropriately during the process. The processing result is also written to the storage device 12 such as a RAM or a hard disk and stored as appropriate. Accordingly, the calculated coordinates and texture information are stored in the storage device 12, and calculation processing such as calculation of new coordinates and texture mapping is executed by appropriately reading out the information. Each means in the present invention is only logically distinguished in function, and may be physically or practically the same area.

  The image generation system 1 includes a panoramic image reception unit 2, a position input reception unit 3, an image generation calculation processing unit 4, and a display processing unit 5.

  The panorama image receiving means 2 is a means for reading at least two arbitrary panorama images in the virtual three-dimensional space. A panoramic image is an image obtained by imaging the entire circumference of 360 degrees around a certain position, and includes a cylindrical coordinate system, a spherical coordinate system, and the like. A conceptual diagram of a panoramic image of a cylindrical coordinate system is shown on the left side of FIG. 9, and a conceptual diagram of a panoramic image of a spherical coordinate system is shown on the right side of FIG.

で表される。 As shown in FIG. 9, when the position is the origin, when the virtual three-dimensional space P (Px, Py, Pz) is projected on the panoramic image of the cylindrical coordinate system, the corresponding cylindrical coordinates (θ, h) are

It is represented by

で表される。 When the virtual three-dimensional space P (Px, Py, Pz) is projected on the panoramic image of the spherical coordinate system, the corresponding spherical coordinates (θ, φ) are

It is represented by

  In this specification, a case where a panoramic image in a spherical coordinate system is read will be described, but the same processing can be performed even when another coordinate system such as a cylindrical coordinate system is used.

  The position input receiving unit 3 is a unit that receives an input of a position where the user wants to view the perspective projection image. This position may be received directly from the input device 10 of the computer terminal that executes the image generation system 1, or information on the input position is sent from the computer terminal used by the user via the network to the image generation system 1. It may be received by the communication device 13 of the computer terminal to be executed and received by the position input receiving means 3. In the former case, the image generation system 1 of the present invention is made to function stand-alone, and in the latter case, the image generation system 1 of the present invention is made to function using a network.

  Based on the panoramic image at least two points received by the panoramic image receiving unit 2 and the information on the position where the perspective projection image received by the position input receiving unit 3 is to be viewed, the image generation calculation processing unit 4 Is a means for generating a perspective projection image.

  The display processing unit 5 is a unit that displays the perspective projection image generated by the image generation calculation processing unit 4 on the display device 14 of the computer terminal. When the image generation system 1 of the present invention is functioned using a network, the display processing unit 5 uses the perspective projection image generated by the image generation calculation processing unit 4 from the communication device 13 via the network. By transmitting to a computer terminal, the perspective projection image is displayed on the computer terminal used by the user. In this specification, display processing on the display device 14, printing processing by a printer, and processing transmitted to another computer terminal using a network are collectively referred to as output processing.

  First, an example of a basic embodiment of the image generation system 1 of the present invention will be described with reference to the system configuration diagrams of FIGS.

  The image generation calculation processing unit 4 of the present embodiment includes an intermediate panorama image generation unit 6 and a perspective projection image generation unit 7.

The intermediate panorama image generation means 6 receives the input by the position input reception means 3, and based on the information of the position where the user wants to see the perspective projection image and at least two or more panorama images in the virtual three-dimensional space, This is means for generating a panoramic image (intermediate panoramic image I _s ) at the position.

Perspective projection image generation means 7, based on the intermediate panoramic image I _s generated by the intermediate panorama image generating means 6 is a means for generating a perspective projection image I _v to be displayed on the display device 14.

  Next, an example of the processing process of the present invention will be described with reference to the flowchart of FIG. 6 and the system configuration diagrams of FIGS.

  First, at least two or more panoramic images in the virtual three-dimensional space are read into the panoramic image receiving means 2 (S100). The panoramic image read at this time may be a panoramic image in a cylindrical coordinate system, a panoramic image in a spherical coordinate system, or a panoramic image in other coordinate systems. In this specification, a case of a panoramic image in a spherical coordinate system will be described, but the present invention can be similarly applied to other coordinate systems.

  When there is no panoramic image, a panoramic image may be generated from an image captured by a digital camera or the like by using general stitching software or the like and read into the panoramic image receiving means 2.

で表される。 In this embodiment, a panoramic image of a spherical coordinate system will be described, so that the viewpoint is the origin as shown on the right side of FIG. As described above, the spherical coordinates (θ, φ) corresponding to P (Px, Py, Pz) in the virtual three-dimensional space are

It is represented by

A conceptual diagram of a virtual three-dimensional space provided with panoramic images at two points is shown in FIG. In FIG. 10, there are two points P ₀ and P ₁ having the same height from the ground, and panoramic images I ₀ and I ₁ at the two points are read into the panoramic image receiving means 2.

Then, the user inputs the position in the virtual three-dimensional space using the input device 10, and receives the position information in the virtual three-dimensional space by the position input receiving means 3 (S110). When this position as P _s, intermediate panorama image generating means 6 of the image generating processing means 4, in point in the panoramic image I _0, I ₁ is where the position of in the middle panoramic image I _s to the position P _s by calculating based on whether the to triangulation, performs arithmetic processing for generating the intermediate panoramic image I _s (S120).

To generate an intermediate panoramic image I _s, first specify the panoramic image I ₀ and the common shooting object to I ₁ as specified in the corresponding point (corresponding point). That is, the position of the object to be photographed in the virtual three-dimensional space is associated with the panoramic images I ₀ and I ₁ . This common corresponding point may be found automatically, or may be specified in advance by the input device 10 or the like, and stored in the storage device 12 in a file format. The coordinate information of the corresponding points is referred to as corresponding point information.

Here, based on the corresponding point information, the panoramic image generation means divides the panoramic images I ₀ and I ₁ into polygonal regions (referred to as “faces”) constituted by corresponding points. Therefore, the vertex of the face is a corresponding point. 11 and 12 show a specification of corresponding points that put in the panoramic image I ₀ and I _1, the state of dividing a panorama image I ₀ and I ₁ to the face. As a result, a corresponding point constituting a face on the panoramic image I ₀ is associated with any corresponding point constituting a face on the panoramic image I ₁ .

A face on the panorama image I ₀ is R ₀ , and a corresponding face on the panorama image I ₁ is R ₁ . Further, regarding the corresponding points constituting R ₀ and R ₁ , a face constituted by points that are associated on the intermediate panorama image _{Is is} defined as R _s . Therefore, the intermediate panorama image generation means 6 calculates R _s as follows using the triangulation method based on the corresponding point information of R ₀ and R ₁ .

Here, one of the corresponding points of the face constituting R ₀ is X ₀ , the corresponding point of the face constituting R ₁ , the corresponding point corresponding to X ₀ is X ₁ , and the corresponding point of the face constituting R _s the corresponding point and X _s that there are corresponding to X ₀ at.

In this specification, the case where the panoramic images I ₀ and I ₁ are represented by a normalized spherical coordinate system having a radius of 1 is described. Therefore, the position corresponding to the point X is set to X ₀ (θ ₀ , φ ₀ , respectively). ), X ₁ (θ ₁ , φ ₁ ).

とする。また図１０に示すように、位置Ｐ_０、Ｐ_１を含む面をｘｚ平面とし、Ｐ_０を原点、ｚ軸上にＰ_１が位置するように空間の座標系を定める。 here,

And As shown in FIG. 10, the plane including the positions P ₀ and P ₁ is defined as the xz plane, the coordinate system of the space is determined so that P ₀ is the origin and P ₁ is located on the z axis.

となる。従って、この演算を中間パノラマ画像生成手段６が、Ｒ_０、Ｒ_１を構成する各対応点に関して行うと、中間パノラマ画像Ｉ_ｓに於けるＲ_ｓの対応点の座標が決定される。これをパノラマ画像Ｉ_０、Ｉ_１の全てのフェイスに関して行う。そうすると、パノラマ画像Ｉ_０、Ｉ_１の各フェイスを構成する各対応点の座標を、中間パノラマ画像Ｉ_ｓの座標に変換することとなる。 And normalized spherical coordinates in X _s in the middle panoramic image I _s at the specified point P _s by the user and (θ _s, φ _s). Then, since P _s is a point that internally divides P ₀ and P ₁ into s: 1-s,

It becomes. Therefore, when the intermediate panorama image generating means 6 performs this calculation for each corresponding point constituting R ₀ and R ₁ , the coordinates of the corresponding point of R _{s in} the intermediate panorama image I _s are determined. This is performed for all the faces of the panoramic images I ₀ and I ₁ . Then, the converting the coordinates of corresponding points constituting each face of the panoramic image I _0, I _1, the coordinates of the intermediate panoramic image I _s.

The intermediate panorama image generating means 6, for each face _{R s} on the intermediate panorama image _{I s,} at the panorama image _I 0, _{I 1,} to texture mapping a face _R 0, _{R 1} corresponding to _{R s} as a texture it makes it possible to generate an intermediate panoramic image I _s.

When the intermediate panorama image generating means 6 generates the position P _s in in the middle panoramic image I _s, perspective projection image generation means 7, based on the intermediate panoramic image I _s, the position P _s in the perspective projection to the image I A process of generating _v is performed (S130).

ここで、ｆｘ、ｆｙはラジアン単位で表した透視投影画像Ｉ_ｖのｘ、ｙ方向画角、Ｒｏｔｘ（−α）、Ｒｏｔｘ（−β）はそれぞれ座標系をｘ軸、ｙ軸を中心に−α、−β回転させたときの点の座標変換を表す行列である。 First perspective projection image generation means 7 divides the area of the perspective projection image I _v to any of a plurality of face R _v. That is, a plurality of vertices (x _v , y _v ) constituting the face R _v are provided in the perspective projection image I _v . The face _Rv may be any polygon, and may or may not have the same shape as R ₀ and R ₁ . The vertex coordinates constituting the face _{R v} of the perspective projection image _{I v (x v, y v} ) is, what the coordinates (θ _s, φ _s) on the intermediate panorama image _{I s} is calculated before it becomes. That associating the vertex coordinates constituting the face R _v of the perspective projection image I _v, and on the intermediate panorama image I _s coordinates. This can be calculated by the perspective projection image generation means 7 calculating Equation 6.

Here, fx, fy are the perspective projection image _{I v,} expressed in radians x, y-direction angle, Rotx (-α), Rotx ( -β) x -axis, respectively, a coordinate system, about the y-axis - It is a matrix representing the coordinate conversion of points when rotated by α and −β.

The perspective projection image generation means 7 calculates Equation 6, so that each vertex (x _v , y _v ) of the face R _{v in} the perspective projection image I _v is located in the intermediate panorama image I _s. Can be calculated. Therefore, it is possible to convert the face R _v of the perspective projection image I _v on the intermediate panorama image I _s. The face of this face _{R v} is converted to the intermediate panorama image _{I s} and _{R sv.}

The perspective projection image generation means 7 texture-maps the texture of the face R _{sv in} the intermediate panorama image I _s to the face R _v of the perspective projection image I _v . By performing each face in the perspective projection image I _v this intermediate panorama image I _s from perspective projection image I _v can be generated. FIG. 26 shows an example of a perspective projection image I _v from the position P _s based on the panoramic images I ₀ and I ₁ .

Thus generated perspective projection image I _v is display processing means 5 is displayed on the display device 14 (S140). Further, when the image generation system 1 of the present invention is made to function using a network, the perspective projection image I _v generated by the image generation calculation processing unit 4 is displayed by the display processing unit 5 from the communication device 13 via the network. By transmitting to the computer terminal used by the user, the perspective projection image _Iv is displayed on the computer terminal used by the user.

By performing the above-described processing, it is possible to generate and display a perspective projection image I _v at an arbitrary point from the panoramic images I ₀ and I ₁ . However, since the panoramic image is an image of a spherical coordinate system or a cylindrical coordinate system, unlike the perspective projection image, the image is distorted. For example, the ceiling portion at the upper left of the panoramic image shown in FIG. 12 is rounded and distorted (an enlarged image of this portion is shown in FIG. 13). Therefore, if the image generation calculation processing means 4 generates the perspective projection image based on the panorama image having the distortion as it is, the texture mapping is performed with the distorted texture even in the texture mapping. Even the perspective projection image _Iv is distorted.

Therefore, in order to solve such a problem, in addition to the above-described basic processing process, the panorama image read by the panorama image receiving unit 2 is directly subjected to the generation processing of the perspective projection image by the image generation calculation processing unit 4. Instead, a distortion correction processing means (not shown) is provided in the image generation calculation processing means 4 before that, a process for correcting distortion is performed on the panoramic image received by the panorama image receiving means 2, and the correction processing is performed. The generated panoramic image may be used as the panoramic images I ₀ and I ₁ , and the above-described perspective projection image generation processing may be performed by the image generation calculation processing means 4.

  FIG. 27 shows a conceptual diagram of correction processing for eliminating distortion in the distortion correction processing means. FIG. 27A shows a conceptual diagram of correction processing for eliminating distortion, and FIG. 27B shows a part of panoramic image pixels of spherical coordinates.

  The distortion correction processing unit divides the panoramic image before distortion correction processing into a plurality of faces (portions indicated by triangles in FIG. 27B) as described above, and relates to each pixel F of each face. The following processing is performed.

尚、点Ｆの座標を（θ_０、φ_０）とする。 First, the position F ′ on the face A on the spherical panorama image corresponding to the position of the pixel F is calculated, the direction vector of the half line h expressed by Equation 7 is calculated, and the panorama corresponding to that direction is calculated by Equation 2. A position on the image is calculated, and the color (color information; a value representing the color of the pixel such as an RGB value) at that position is used as the color of the pixel F. Note that the position F ′ is a position to which the position F is mapped when texture mapping is performed.

Note that the coordinates of the point F are (θ ₀ , φ ₀ ).

  When the position F ′ calculated in this way is projected onto the panoramic image as G, the pixel F is projected onto the position G in the color of the position F ′, so that the distortion is corrected. It becomes.

  By performing such distortion correction processing for each pixel F of each face by the distortion correction processing means, it is possible to perform panorama image correction processing. FIG. 14 shows an image after such a correction process for FIG.

Thus, based on the panorama image after the distortion correction processing, a more beautiful perspective projection image I _v is obtained by using the basic image for generating the perspective projection image I _v as the above-described panorama images I ₀ and I _1. Can be generated.

  Since the basis of the processing is a panoramic image, when dividing into faces, as shown in FIG. 15, there may be a face straddling the panoramic image. In this case, the texture mapping can be easily processed by repeating the texture as shown in FIG. 15 and continuously treating it as one texture.

  That is, when a panoramic image using a projection method having a common left and right end, such as a panoramic image in a spherical coordinate system or a panoramic image in a cylindrical coordinate system, is used as a panoramic image, the face corresponding to the same corresponding point is uniquely shown in FIG. Is not determined.

  Therefore, as shown in FIG. 17, a flag is assigned to each face (flag R or flag L in FIG. 17), and which texture is used for the face is determined by the flag. In FIG. 17, the flag R means that when the image is repeated, it is on the right side of the right end of the original image (face in the broken line portion in FIG. 15).

  Alternatively, instead of providing a flag for each face, as shown in FIG. 18, the image is divided into a predetermined number of parts in the x direction, for example, four parts, and the positions of all corresponding points on the face are as shown in FIG. If it is within the range of either region RL or RR, the face is determined as shown in FIG. 18C, and even if the position of the corresponding point is outside the range of regions RL and RR as shown in FIG. 18B. If there is, the face is determined as shown in FIG. That is, the face can be automatically determined from the vertex position information.

Further, when texture mapping is performed for each face, for example, when faces R ₀ and R ₁ are used as textures and texture mapping is performed on R _s , R ₀ and R ₁ are combined in accordance with a predetermined interpolation parameter s. the such texture may be pasted to R _s.

Interpolation parameter s, for example at 10, when the _{P 0} to _{P 1} and 1, _{P s} is s, the _{P 1} from _{P s} becomes 1-s from _{P 0.} Then, P _s is a point that internally divides P ₀ and P ₁ into s: 1-s, so that the face R _{0 of} P ₀ is a ratio of s and the face R _{1 of} P ₁ is a ratio of 1-s. synthesized as one of the texture, paste the texture of post-synthesis to R _s.

Thus, by performing the process of Example 1, and outputs the perspective projection image I _v to files from any point in the virtual three-dimensional space can be displayed on the display device 14. At this time, upon generation of a perspective projection image I _v performs the operation only at the corresponding point, since merely the texture mapped to the corresponding points, can very fast processing, even real-time display Yes.

Next, as the modification of the first embodiment, calculates a corresponding point that put on the intermediate panorama image I _s based on the corresponding point information of the panoramic image I ₀ and I _1, the texture mapping to the intermediate panorama image I _s without (i.e., without generating an intermediate panoramic image I _s) describing a case of generating a perspective projection image I _v.

  As shown in FIG. 4, the image generation calculation processing means 4 in the present embodiment has a corresponding point calculation means 8 and a perspective projection image generation means 7.

Corresponding point calculation means 8 receives the input by position input reception means 3, and based on the information of the position where the user wants to see the perspective projection image and at least two or more panoramic images in the virtual three-dimensional space. This is means for calculating the corresponding point X _s of the panoramic image (intermediate panoramic image I _s ) at the position and calculating the corresponding point X _v of the perspective projection image I _v from the position based on the calculated point X _s .

The perspective projection image generation means 7 is a means for generating a perspective projection image I _v based on the corresponding point X _v calculated by the corresponding point calculation means 8.

  Next, an example of the processing process in the present embodiment will be described with reference to the flowchart of FIG. 7 and the system configuration diagrams of FIGS. Since S200 and S210 are the same as S100 and S110 of the first embodiment, description thereof is omitted.

As in the first embodiment, a face constituted by corresponding points of the panoramic image I ₀ is R _0, and a face corresponding to the face R ₀ on the panoramic image I ₁ is R ₁ . Accordingly, the corresponding point X ₀ constituting the face R ₀ is associated with the corresponding point X ₁ constituting the face R ₁ . Further, the faces corresponding to the faces R ₀ and R ₁ in the intermediate panorama image I _s are R _s , and the corresponding faces in the perspective projection image I _v are R _v . Therefore, the corresponding points X ₀ and X ₁ constituting the faces R ₀ and R ₁ are associated with the vertex X _s constituting the face R _s and the vertex X _v constituting the face R _v . Thus, the corresponding point calculation unit 8 may calculate the coordinates of the vertex X _v constituting the R _v.

Information at position P _s on the input virtual three-dimensional space is accepted by the position input receiving means 3 by the user (S210), the corresponding point calculation unit 8 of the image generation processing unit 4 is in the middle to P _s performs arithmetic processing for calculating the corresponding point _{X s} panoramic image _{I s} (S220).

In such a case, the corresponding point calculation means 8 first calculates X _s corresponding to X ₀ and X ₁ by using the above-described equations 4 and 5 as in the first embodiment. Then, the corresponding point calculation means 8 performs this calculation for all the faces of the panoramic images I ₀ and I ₁ . Then, the coordinates of the corresponding points constituting each face of the panoramic image I _0, I _1, can be converted into the coordinate X _s of the vertex on the intermediate panorama image I _s.

After calculating X _s based on X ₀ and X ₁ by the corresponding point calculating means 8, the vertex X _v on the perspective projection image I _v corresponds to X _s . Next, the corresponding point calculating means 8 This _Xv is calculated. Here, the normalized spherical coordinate of X _s is (θ _s , φ _s ), and the perspective projection image of X _v is (x _v , y _v ).

に投影されるものである。この概念図を図１９に示す。 Here, the normalized spherical coordinates are the same as those in the first embodiment, and the perspective projection image has the center of the field of view in the (α, β) direction of the normalized spherical coordinates and the x and y directions of the perspective projection image I _v expressed in radians. When the angle of view is fx, fy

Is projected onto the screen. This conceptual diagram is shown in FIG.

となる。ここでＲｏｔｘ（β）、Ｒｏｔｙ（α）はそれぞれｘ軸、ｙ軸周りに座標系をβ回転、α回転させたときの点の座標変換を表す行列とする。 First, the corresponding point calculation means 8 performs coordinate conversion so that the line-of-sight direction (α, β) is the center (0, 0) of the spherical panoramic image. Let X _n (θ _n , φ _n ) be the corresponding point of X _{s in} these coordinates. Then

It becomes. Here, Rotx (β) and Roty (α) are matrices representing coordinate transformation of points when the coordinate system is rotated β and α around the x and y axes, respectively.

Then, the corresponding point calculation means 8 calculates Equation 10 for the vertex X _v (x _v , y _v ) in the perspective projection image I _v by the vertex X _s (θ _s , φ _s ) of the intermediate panorama image I _s. Is required.

This is carried out with respect to all of the face of the perspective projection image I _v. Then, the coordinates of each vertex X _s constituting each face R _s of the intermediate panorama image I _s can be converted to the coordinates of each vertex X _v constituting each face R _v of the perspective projection image I _v .

On the basis of the calculated X _v and for each face of the perspective projection image I _v, face R _0, R ₁ perspective projection image generation means 7 by texture mapping to the face R _v as texture, perspective projection generating an image _{I v} (S230).

That is, the corresponding points X ₀ and X ₁ constituting the faces R ₀ and R ₁ in the panoramic images I ₀ and I ₁ are converted into the coordinates of the vertices X _v of the face R _v constituting the perspective projection image I _v. Thus, the faces R ₀ and R _{1 of} the panoramic images I ₀ and I ₁ can be converted into the face R _v on the perspective projection image I _v . Therefore, the texture mapping of the faces R ₀ and R ₁ is performed on the face R _v on the perspective projection image I _v . Then, by performing this for each face, a perspective projection image I _v can be generated from the panoramic images I ₀ and I ₁ .

Thus generated perspective projection image I _v is display processing means 5 is displayed on the display device 14 (S240). Further, when the image generation system 1 of the present invention is functioned using a network, the display processing means 5 transmits the perspective projection image I _v generated by the image generation calculation processing means 4 from the communication device 13 via the network. By transmitting to the computer terminal used by the user, the perspective projection image _Iv is displayed on the computer terminal used by the user. FIG. 26 shows an example of a perspective projection image I _v from the position P _s based on the panoramic images I ₀ and I ₁ .

  Note that the distortion correction processing according to the first embodiment and the pasting of the texture according to the interpolation parameter s can be similarly added to the processing.

  Since the texture mapping is performed only once in the processing of the second embodiment, when the position designated by the user frequently changes, the processing can be performed at a higher speed than the first embodiment. In addition, since texture mapping only needs to be performed once, the fact that there is little deterioration in image quality is more effective than in the first embodiment.

Next, as a modification of the basic processing of the first embodiment, R ₀ is calculated by calculating the position in the virtual three-dimensional space corresponding to each vertex of the faces R ₀ and R _{1 in} the panoramic images I ₀ and I _1. A case where the spatial position of the photographing object R ₁ is approximately calculated to generate a perspective projection image I _v will be described.

The image generation calculation processing means 4 of this embodiment has a spatial position calculation means 9 and a perspective projection image generation means 7. The spatial position calculation means 9 of this embodiment is a means for approximately calculating the spatial position in the virtual three-dimensional space of the photographing object of the faces R ₀ and R _{1 in} the panoramic images I ₀ and I _1. .

The perspective projection image generation means 7, based on the spatial position of the imaging object calculated by the spatial position calculating means 9 is means for generating a perspective projection image I _v by either of the following methods. As a first method for generating the perspective projection image I _v from the spatial position of the object to be imaged, a method of generating a face on the two-dimensional perspective projection image I _v and attaching a texture to the face, a second method Is a method in which a face is created on a spherical surface at a position specified by the user, a texture is pasted there, and texture mapping is performed in the same manner as in the first embodiment. As a third method, a three-dimensional polygon is generated, This is a method of pasting a texture on a face.

  An example of the processing process in the next embodiment will be described with reference to the flowchart of FIG. 8 and the system configuration diagrams of FIGS. Since S300 and S310 are the same as S100 and S110 of the first embodiment, description thereof is omitted.

When receiving the information at the position P _s on the input virtual three-dimensional space by the user in the position input receiving means 3 (S310), the spatial position calculating means 9 of the image generation processing unit 4, the panoramic image I _0, I _The spatial position in the virtual three-dimensional space of the imaging object of the faces R ₀ and R _{1 in 1} is approximately calculated (S320).

As in the second embodiment, the corresponding point constituting the face R ₀ is X ₀ , the corresponding point constituting the face R ₁ is X ₁ , the vertex constituting the face R _s is X _s , and the vertex constituting the face R _v is represented Let _Xv . Then, let X be the spatial position in the virtual three-dimensional space corresponding to X ₀ , X ₁ , X _s , and X _v of the object to be imaged.

As shown in FIG. 20, the half lines representing the existence range of P determined by the positions of the corresponding points X ₀ and X ₁ at the positions P ₀ and P ₁ are defined as l ₀ and l ₁ . Then, l ₀ and l ₁ ideally intersect, and the intersection is the position of the point X in the virtual three-dimensional space. Therefore, the position of the point X in the virtual three-dimensional space can be calculated as the intersection of l ₀ and l ₁ .

Specifically, as shown in FIG. 20, the plane including the positions P ₀ and P ₁ is the xz plane, P ₀ is the origin, and P ₁ (0, 0, −1) is positioned on the z axis. Define the coordinate system. Assuming that the normalized spherical coordinate systems of X ₀ and X ₁ are (θ ₀ , φ ₀ ) and (θ ₁ , φ ₁ ), respectively, the direction vectors of the corresponding half lines l ₀ and l ₁ are given by Equation 11, respectively. .

The spatial position calculation means 9 can calculate the position of the point X (x, y, z) by calculating Expression 12 using Expression 11.

As described above, R _{0 is} calculated by calculating the positions in the virtual three-dimensional space corresponding to the corresponding points X ₀ and X ₁ of the faces R ₀ and R ₁ from the intersections of the half lines l ₀ and l ₁ as described above. , The spatial position of the object to be imaged in R ₁ in the virtual three-dimensional space can be approximately calculated. This is performed for all the faces of the panoramic images I ₀ and I ₁ . Then, the positions in the virtual three-dimensional space of the corresponding points X ₀ and X ₁ constituting all the faces R ₀ and R ₁ in the panoramic images I ₀ and I ₁ can be calculated.

After the spatial position calculation means 9 calculates the position of the point X in the virtual three-dimensional space, the perspective projection image generation means 7 performs the fluoroscopy according to any one of the three methods described above based on the position in the space. generating a projection image _{I v (S330~S350).}

尚、ここでユーザが指定した位置Ｐ_ｓをこの仮想３次元空間上で（Ｐ_ｓｘ，Ｐ_ｓｙ，Ｐ_ｓｚ）とする。そうすると、中間パノラマ画像Ｉ_ｓ上の頂点Ｘ_ｓから透視投影画像Ｉ_ｖ上の頂点Ｘ_ｖの位置への変換を実施例２と同様の方法で行うことが出来る。 First perspective projection image generation unit 7, the case of generating a perspective projection image I _v by the first method described above (S330). In this case, from the point X (X _x , X _y , X _z ) in the virtual three-dimensional space, the corresponding position X _s (θ _s) on the spherical coordinates of the intermediate panoramic image I _s at the position P _s specified by the user. , Φ _s ) is obtained when the perspective projection image generation means 7 calculates Equation (13).

Here, the position P _s designated by the user is assumed to be (P _sx , P _sy , P _sz ) in this virtual three-dimensional space. Then, you are possible to carry out an intermediate panoramic image I _s on the vertex transformation Example 2 In the same manner as from X _s to the position of the vertex X _v on the perspective projection image I _v.

On the basis of the calculated X _v and, according to the same method as Example 2, for each face of the perspective projection image I _v, face R _0, R ₁ perspective projection image by texture mapping to the face R _v as texture generation means 7 generates a perspective projection image I _v.

That is, the corresponding points X ₀ and X ₁ constituting the faces R ₀ and R ₁ in the panoramic images I ₀ and I ₁ are converted into the coordinates of the vertices X _v of the face R _v constituting the perspective projection image I _v. Thus, the faces R ₀ and R _{1 of} the panoramic images I ₀ and I ₁ can be converted into the face R _v on the perspective projection image I _v . Therefore, the texture mapping of the faces R ₀ and R ₁ is performed on the face R _v on the perspective projection image I _v . Then, by performing this for each face, a perspective projection image I _v can be generated from the panoramic images I ₀ and I ₁ .

Then the perspective projection image generation means 7, the case of generating a perspective projection image I _v by the second method described above (S340).

In this method, since the point X is converted into the corresponding vertex P _sp (X _sp , Y _sp , Z _sp ) on the spherical coordinate of the intermediate panoramic image I _s at the position P _s specified by the user, a perspective projection image generation is performed. The means 7 calculates the formula 14.

By performing this operation on the corresponding points X ₀ and X ₁ constituting the faces in the panoramic images I ₀ and I ₁ , the vertexes X _s constituting the faces in the intermediate panorama image I _s can be converted. Then in the same manner as in Example 1, and the panoramic image _I 0, in the _{I 1} Face _R 0, the texture of the _{R 1,} by texture mapping on the face _{R s} of the intermediate panoramic image _{I s,} intermediate panoramic image _{I s} Can be generated.

Thus in the same manner as in Example 1, the vertices of the at face R _v in perspective projection image I _v calculates whether in any position at the intermediate panoramic image I _s, face R _v of the perspective projection image I _v converting the on intermediate panoramic image I _s. The perspective projection image generation unit 7, by texture mapping texture in the face in the middle panoramic image I _s to face R _v of the perspective projection image I _v, the perspective projection image I _v from the intermediate panoramic image I _s Generate.

Although the processing method similar to that of the first embodiment has been described here, the same processing method as that of the second embodiment can also be used. That is, the correspondence between the corresponding points X ₀ and X ₁ constituting the faces in the panoramic images I ₀ and I ₁ and the vertices X _v of the face R _{v in} the perspective projection image I _v is calculated. in, and calculates the coordinates of the vertex _{X v} constituting the face _{R v.} Then, based on thus calculated _{X v,} for each face of the perspective projection image _{I v,} face _R 0, _{R 1} by texture mapping to the face _{R v} as texture, projected from the panoramic image _I 0, _{I 1} A projection image _Iv is generated.

  It should be noted that since the points are in the virtual three-dimensional space, four or more points are generally not on the same plane, but the four or more points in the virtual three-dimensional space are appropriately divided by the perspective projection image generation means 7 for rental. It can be done by processing. For this rental process, APIs such as OpenGL and DirectX can be used, for example. This makes it possible to automatically divide and perform the rental process.

Further perspective projection image generation unit 7, the case of generating a perspective projection image I _v by the third method described above (S350).

In this method, in the process of S320, the positions on the virtual three-dimensional space of the corresponding points X ₀ and X ₁ constituting all the faces R ₀ and R ₁ in the panoramic images I ₀ and I ₁ are determined. Since it is calculated, a three-dimensional polygon having the face as a surface can be formed. Then, the textures of the faces R ₀ and R ₁ of the panoramic images I ₀ and I ₁ are texture-mapped to the surface corresponding to the face of the three-dimensional polygon.

Then the perspective projection image generation means 7, since the three-dimensional polygon in the virtual three-dimensional space can be generated, generating a perspective projection image I _v from the position P _s specified by the user. A known three-dimensional polygon processing method can be used to generate the perspective projection image I _v from the three-dimensional polygon.

As described above, the display processing unit 5 is displayed on the display device 14 a perspective projection image I _v generated by the perspective projection image generation means 7 (S360). Further, when the image generation system 1 of the present invention is functioned using a network, the display processing means 5 transmits the perspective projection image I _v generated by the image generation calculation processing means 4 from the communication device 13 via the network. By transmitting to the computer terminal used by the user, the perspective projection image _Iv is displayed on the computer terminal used by the user. FIG. 26 shows an example of a perspective projection image I _v from the position P _s based on the panoramic images I ₀ and I ₁ .

  Note that the distortion correction processing according to the first embodiment and the pasting of the texture according to the interpolation parameter s can be similarly added to the processing.

When calculating the point X of the spatial position of the object to be photographed, which is the premise of the present embodiment, as shown in FIG. 20, the intersections of the half lines l ₀ and l ₁ from the positions P ₀ and P ₁ are calculated. It is processed by. However, actually, the half lines l ₀ and l ₁ do not necessarily intersect at the point X as shown in FIG. 21 due to distortion of the panoramic images I ₀ and I ₁ , errors during stitching, and the like.

In this case, a point close to the point closest to the half-line _{l 1} on half line _{l 0} on the _Q 0, _{l 1} the most half-line _{l 0} on the _{Q 1,} interpolating a line segment _Q 0 _{Q 1} given The spatial position calculation means 9 performs a calculation process that determines that the point X is internally divided into s: 1-s by the parameter s.

Q _{0, Q 1} position, respectively _{Q 0 (q 0x, q 0y} , q 0z), Q 1 (q 1x, q 1y, q 1z) When, by these using the number 11 of the above, the number 15 It is expressed as Accordingly, the spatial position calculation means 9 calculates the positions of Q ₀ and Q ₁ by calculating the formula 15.

As described above, even when the half lines l ₀ and l ₁ do not necessarily intersect at the point X, when the point X is determined, X _s = X ₀ and X _s = when s = 0 and s = 1, respectively. X ₁ becomes, it is possible to push the distortion of perspective projection image I _v. Further, the length of the line segment Q ₀ Q ₁ is the smallest among these conditions, and is a preferable approximation to the actual position of the object to be photographed. Q ₀ and Q ₁ need only be calculated once at the beginning, and need not be calculated every time interpolation processing is performed.

It should be noted that X may not be calculated correctly when the distortion is very large, or when there is a subject to be photographed in the vicinity of the straight line connecting the positions P ₀ and P ₁ . Or in some cases you want to deliberately distort the perspective projection image I _v in order to make a special representation. In such a case, a function for enabling input from the user may be provided for Q ₀ and Q ₁ or X. In this case, the perspective projection image generation means 7 receives information on Q ₀ , Q ₁ , and X input from the input device 10 by the user, and performs arithmetic processing based on the information. Further, the information of Q ₀ , Q ₁ , and X may be stored in advance in the storage device 12 in a predetermined file format and read out so as to be processed by the calculation device 11.

Based on the panoramic images I ₀ and I ₁ of at least two points in the first to third embodiments, the user sequentially inputs the position P _s sequentially, and the perspective projection image I _v from the position P _s is obtained. By continuously performing the processing to be generated, the perspective projection image _Iv is continuously generated as if moving in the virtual three-dimensional space.

In this case, as shown in FIG. 22, a plurality of panoramic images in a virtual three-dimensional space are prepared, and a plurality of corresponding point information between the panoramic images is prepared. In FIG. 22, panoramic images I ₀ , I ₁ , I ₂ , and I ₃ from positions P ₀ , P ₁ , P ₂ , and P ₃ in the virtual three-dimensional space are provided, and I ₀ and I ₁ , I ₁ and I ₂ , corresponding point information is set between I ₂ and I ₃ . This corresponding point information is assumed to be T ₀ , T ₁ , T ₂ .

Based on the corresponding point information of T ₀ , T ₁ , and T ₂ , the image generation calculation processing means 4 continuously generates the perspective projection image I _v by any of the methods of the first to third embodiments described above. To do. That is, the user sequentially from the input device 10, by entering in position in the virtual three-dimensional space, accepts the position by the position input receiving means 3 sequentially generates the image generation processing unit 4 a perspective projection image I _v Then, the display processing means 5 displays it on the display device 14.

As shown in FIG. 23, when generating a perspective projection image I _v in a virtual three-dimensional space from positions P ₀ to P ₃ , first, the first to third embodiments are applied to the morphing section M ₀ . The perspective projection image I _v0 is generated by any one of the methods, and when the position reaches P ₁ , the perspective projection image I _v1 is generated by the method according to any one of the first to third embodiments for the morphing section M ₁ . generated, the position to generate a perspective projection image I _v2 in any of the method of example 3 from example 1 with respect to morphing section M ₂ when it reaches the P _2. The designation of the position may be that the user inputs sequentially from the input device 10 and accepts it by the position input accepting means 3 or accepts information on continuous position movement in advance by the position input accepting means 3. It may be to keep.

The morphing section is a section in which the perspective projection image I _v is continuously generated by the method of any one of the first to third embodiments using the same panoramic images I ₀ and I ₁ , corresponding point information T _{0 and the} like. Show.

Here, when the morphing section is switched, the viewpoint position and the line-of-sight direction may change discontinuously. Therefore, a process is provided for converting the interpolation parameter in the current morphing section into the interpolation parameter and line-of-sight direction in the new morphing section. For example, as shown in FIG. 24, space coordinates for each of the morphing sections M ₀ and M ₁ are set as local coordinates (relative coordinates), and world coordinates (absolute coordinates) are provided in a virtual three-dimensional space. Conversion parameters between world coordinates and local coordinates, such as a position at the position P ₀ viewed in the world coordinate system and a unit vector from the position P ₀ to the position P ₁ , are given in advance as information. Then, the viewpoint position and the line-of-sight direction in the local coordinates before the morphing section shift are converted into world coordinates, and further converted into the viewpoint position and the line-of-sight direction in the local coordinates after the morphing section shift. By doing so, switching of the morphing section changes smoothly.

For example, as shown in FIG. 25, continuously generating a perspective projection image in the morphing section M _0, and displays, when switching from the morphing section M ₀ to morphing section M ₁ is a continuous condition and world coordinates of morphing section Parameter conversion is performed using information prepared in advance as a set of scene data for parameters for conversion with local coordinates, and perspective projection images are continuously generated and displayed in adjacent sections M ₀ and M ₁ . And when located next morphing section M _1, continuously generating a perspective projection image in the morphing section M _1, performs processing for displaying.

  When the world coordinate system is provided and the correspondence relationship with each local coordinate system is set, when there is a common corresponding point in multiple morphing sections, the other morphing section is based on the corresponding point data set in one morphing section. Corresponding points at can be acquired automatically. That is, from the corresponding point set in one morphing section, the position of the point in the space in the local coordinate system of the morphing section is calculated, and the calculation is performed to convert it to the world coordinate system, and the local coordinate system of the other morphing section The corresponding point can be automatically acquired by converting to a position of a point on the space and converting to a position on each panoramic image corresponding to the position of the point on the space. By providing this process, the amount of work for acquiring corresponding points can be significantly reduced.

Note that real image, CG image, or by reading the images obtained by other imaging methods, the perspective projection image I _v obtained by the image generation system 1 of the present invention as a background, the image read as the foreground Means for performing image composition processing may be provided. As another image generation method, for example, JP-A-2002-324249 can be used.

In addition, as a further application of the present invention, the left eye, and generates a perspective projection image I _v seen from the respective viewpoints of the right eye, it three-dimensional correspondence display device 14 (e.g., manufactured by Sharp Corporation LCD monitor LL-151D Etc.) can also be displayed in 3D. In this case, it is possible to generate and display the perspective projection images I _vl and I _vr from the panoramic image for both the right eye and the left eye. The display device 14 may be a goggle type display device 14 as well as a liquid crystal monitor, and is not limited thereto.

Unlike the conventional method, the panorama images I ₀ , I _{1 are used} to generate a perspective projection image I _vl for the left eye and a perspective projection image I _vr for the right eye and perform three-dimensional display. No need for your own image. In addition, since the viewpoint position and line-of-sight direction can be adjusted arbitrarily, the parallax can be freely adjusted, and the display device 14 and the user's age, display purpose, etc. can be combined to increase the parallax and display more stereoscopically. You can also make adjustments to reduce parallax and reduce eye strain.

  According to the present invention described above, a general user can easily generate and display a perspective projection image from an arbitrary position in a virtual three-dimensional space. This eliminates the need for creating a conventional three-dimensional shape model or creating scene data. Even when the position is moved, it is possible to perform continuous and free movement without using special equipment or the like as in the image-based method.

Further, since a perspective projection image is generated and displayed by generating an intermediate image from the two original panoramic images, the amount of data can be extremely small even with the display quality of the photographed image.

It is the figure which showed an example of the hardware constitutions of the computer terminal which functions an image generation system. 1 is a system configuration diagram illustrating functions of an image generation system. FIG. 2 is a system configuration diagram illustrating functions of a system of an image generation arithmetic processing unit according to the first exemplary embodiment. FIG. 10 is a system configuration diagram illustrating functions of a system of an image generation arithmetic processing unit according to a second embodiment. FIG. 10 is a system configuration diagram illustrating functions of a system of an image generation arithmetic processing unit according to a third embodiment. 3 is a flowchart illustrating an example of a processing process according to the first exemplary embodiment. 10 is a flowchart illustrating an example of a processing process according to the second embodiment. 10 is a flowchart illustrating an example of a processing process according to a third embodiment. It is a conceptual diagram of a panoramic image. It is a conceptual diagram of the virtual three-dimensional space which provided the panoramic image of 2 points | pieces. Designated corresponding points and that put in the panoramic image I _0, it is a diagram showing a state where divided into the face by it. Designated corresponding points and that put in the panoramic image I _1, showing a state where divided into the face by it. Is an example portion of the panoramic image I ₀ to read if not corrected. Is an example portion of the panoramic image I ₀ to load when performing correction. When dividing the intermediate panoramic image I _s to the face, it is a conceptual view when the face crosses the intermediate panoramic image I _s. It is the conceptual diagram which showed that the face by a corresponding point was not determined uniquely. It is a conceptual diagram at the time of giving a flag to a corresponding point. It is a conceptual diagram in the case where a face is automatically determined from vertex position information by dividing an image into four in the x direction. It is a conceptual diagram which shows conversion of a gaze direction. In Example 3, it is a conceptual diagram of the process which calculates the position of the point X in the virtual three-dimensional space of the point X. In Example 3, it is a conceptual diagram of the calculation process in the case where the position of the point X cannot be calculated as an intersection point due to the distortion of the panoramic images I ₀ and I ₁ or an error at the time of stitching. In Example 4, it is a figure which shows the concept of a panoramic image and corresponding point information. In Example 4, it is a figure which shows the concept of the relationship between a morphing area and a position. In Example 4, the morphing section M _0, the spatial coordinates of each M ₁ and the local coordinates, is a conceptual view of a case of providing world coordinates at the virtual three-dimensional space. In Example 4, the conversion parameters between the morphing interval and the world coordinates and local coordinates are prepared in advance as a set of scene data, and when the morphing interval is switched, the scene data is reproduced. It is a conceptual diagram in the case of generating a perspective projection image in a new morphing section after the scene data ends. Based on the panoramic image _{I 0} and _{I 1,} an example of a perspective projection image _{I v} from the position _{P s.} It is a conceptual diagram of the correction process of distortion.

Explanation of symbols

1: Image generation system 2: Panoramic image receiving means 3: Position input receiving means 4: Image generation calculation processing means 5: Display processing means 6: Intermediate panorama image generating means 7: Perspective projection image generating means 8: Corresponding point calculating means 9 : Space position calculation means 10: input device 11: arithmetic device 12: storage device 13: communication device 14: display device

Claims (15)

An image generation system for generating a perspective projection image from a position specified by a user in a virtual three-dimensional space based on at least two panoramic images,
The image generation system includes:
Panorama image receiving means for reading the panorama image;
Position input receiving means for receiving information of a position designated by the user in the virtual three-dimensional space;
Image generation calculation processing means for generating a perspective projection image at the received position from the read at least two panoramic images;
Display processing means for outputting the generated perspective projection image;
An image generation system comprising: The image generation calculation processing means includes an intermediate panorama image generation means and a perspective projection image generation means,
The intermediate panorama image generating means
Based on the information of the designated corresponding points of the object to be photographed common to the panoramic image, calculating where the corresponding points in the panoramic image are located in the intermediate panoramic image by triangulation method, Texture mapping of a face constituted by corresponding points in the panoramic image is texture-mapped to a face constituted with the calculated position in the intermediate panoramic image corresponding to the face as a vertex, whereby the intermediate panoramic image is Generate and
The perspective projection image generation means includes:
The region of the perspective projection image is divided into a plurality of arbitrary faces, the position of the vertex constituting the face is calculated in the intermediate panorama image, and the calculated position in the intermediate panorama image is calculated. Generating the perspective projection image by texture-mapping the texture of the face constituted by vertices to the corresponding face divided in the perspective projection image,
The image generation system according to claim 1. The image generation calculation processing means includes corresponding point calculation means and perspective projection image generation means,
The corresponding point calculation means includes
Based on the information of the designated corresponding points of the object to be photographed common to the panoramic image, calculating where the corresponding points in the panoramic image are located in the intermediate panoramic image by triangulation method, Converting the calculated position into a position on the perspective projection image;
The perspective projection image generation means includes:
The position of the calculated corresponding point of the panoramic image on the perspective projection image is a vertex, and the perspective projection image is generated by texture mapping the texture corresponding to the panoramic image on the face constituted by the vertex.
The image generation system according to claim 1. The image generation calculation processing means has a spatial position calculation means and a perspective projection image generation means,
The spatial position calculation means includes
Based on the information of the designated corresponding point of the photographing object common to the panoramic image, the position of the corresponding point in the virtual three-dimensional space is determined as the virtual three-dimensional space of the photographing object corresponding to the corresponding point. Decide on the half line that is the range where the position can exist,
The perspective projection image generation means includes:
The position of the corresponding point in the determined panoramic image in the virtual three-dimensional space is converted into the coordinate of the intermediate panoramic image at the received position, and the converted coordinate is converted to the position on the perspective projection image. Converting the perspective projection image by converting the texture corresponding to the panoramic image to a face constituted by the position of the corresponding point of the calculated panoramic image on the perspective projection image as a vertex and a face constituted by the vertex. Generate,
The image generation system according to claim 1. The image generation calculation processing means has a spatial position calculation means and a perspective projection image generation means,
The spatial position calculation means includes
Based on the information of the designated corresponding point of the photographing object common to the panoramic image, the position of the corresponding point in the virtual three-dimensional space is determined as the virtual three-dimensional space of the photographing object corresponding to the corresponding point. Decide on the half line that is the range where the position can exist,
The perspective projection image generation means includes:
A face constituted by the corresponding point in the panoramic image by converting the position of the corresponding point in the determined panoramic image in the virtual three-dimensional space into the coordinates of the intermediate panoramic image at the received position. The texture is mapped to a face that has the converted position in the intermediate panorama image corresponding to the face as a vertex to generate the intermediate panorama image, and an area of the perspective projection image is arbitrarily set. A face that is divided into a plurality of faces, calculates the position of the vertex constituting the face in the intermediate panorama image, and includes the calculated position in the intermediate panorama image as a vertex. Texture mapping is performed on the corresponding face divided in the perspective projection image. And it allows to generate the perspective projection image,
The image generation system according to claim 1. The image generation calculation processing means has a spatial position calculation means and a perspective projection image generation means,
The spatial position calculation means includes
Based on the information of the designated corresponding point of the photographing object common to the panoramic image, the position of the corresponding point in the virtual three-dimensional space is determined as the virtual three-dimensional space of the photographing object corresponding to the corresponding point. Decide on the half line that is the range where the position can exist,
The perspective projection image generation means includes:
A three-dimensional polygon having a face constituted by the determined position of the corresponding point in the virtual three-dimensional space is formed, and the texture of the face of the panoramic image is used as the corresponding face of the three-dimensional polygon. Generating the three-dimensional polygon by texture mapping, and generating the perspective projection image by converting the three-dimensional polygon to a perspective projection image from the received position;
The image generation system according to claim 1. The image generation calculation processing means has a spatial position calculation means and a perspective projection image generation means,
The spatial position calculation means includes
Based on the information of the designated corresponding point of the photographing object common to the panoramic image, the position of the corresponding point in the virtual three-dimensional space is determined as the virtual three-dimensional space of the photographing object corresponding to the corresponding point. A position that internally divides half straight lines that are positions where the position can exist into predetermined interpolation parameters is determined as a position in the virtual three-dimensional space;
The perspective projection image generation means includes:
The position of the corresponding point in the determined panoramic image in the virtual three-dimensional space is converted into the coordinate of the intermediate panoramic image at the received position, and the converted coordinate is converted to the position on the perspective projection image. Converting the perspective projection image by converting the texture corresponding to the panoramic image to a face constituted by the position of the corresponding point of the calculated panoramic image on the perspective projection image as a vertex and a face constituted by the vertex. Generate,
The image generation system according to claim 1. The image generation calculation processing means has a spatial position calculation means and a perspective projection image generation means,
The spatial position calculation means includes
Based on the information of the designated corresponding point of the photographing object common to the panoramic image, the position of the corresponding point in the virtual three-dimensional space is determined as the virtual three-dimensional space of the photographing object corresponding to the corresponding point. A position that internally divides half straight lines that are positions where the position can exist into predetermined interpolation parameters is determined as a position in the virtual three-dimensional space;
The perspective projection image generation means includes:
A face constituted by the corresponding point in the panoramic image by converting the position of the corresponding point in the determined panoramic image in the virtual three-dimensional space into the coordinates of the intermediate panoramic image at the received position. The texture is mapped to a face that has the converted position in the intermediate panorama image corresponding to the face as a vertex to generate the intermediate panorama image, and an area of the perspective projection image is arbitrarily set. A face that is divided into a plurality of faces, calculates the position of the vertex constituting the face in the intermediate panorama image, and includes the calculated position in the intermediate panorama image as a vertex. Texture mapping is performed on the corresponding face divided in the perspective projection image. And it allows to generate the perspective projection image,
The image generation system according to claim 1. The image generation calculation processing means has a spatial position calculation means and a perspective projection image generation means,
The spatial position calculation means includes
Based on the information of the designated corresponding point of the photographing object common to the panoramic image, the position of the corresponding point in the virtual three-dimensional space is determined as the virtual three-dimensional space of the photographing object corresponding to the corresponding point. A position that internally divides half straight lines that are positions where the position can exist into predetermined interpolation parameters is determined as a position in the virtual three-dimensional space;
The perspective projection image generation means includes:
A three-dimensional polygon having a face constituted by the determined position of the corresponding point in the virtual three-dimensional space is formed, and the texture of the face of the panoramic image is used as the corresponding face of the three-dimensional polygon. Generating the three-dimensional polygon by texture mapping, and generating the perspective projection image by converting the three-dimensional polygon to a perspective projection image from the received position;
The image generation system according to claim 1. The image generation calculation processing means includes:
Generating a panoramic image in which distortion in the panoramic image is corrected from the read at least two panoramic images, and generating a perspective projection image at the received position based on the corrected at least two panoramic images;
The image generation system according to claim 1. The image generation system includes:
Depending on the ratio from the position received by the position input receiving means to the position of each panoramic image, texture mapping of the texture is performed.
The image generation system according to claim 1, wherein the image generation system is an image generation system. The position input receiving means sequentially receives the position information,
The image generation calculation processing means includes:
Based on the received position information, sequentially generate perspective projection images at each position,
The display processing means includes
Outputting the generated perspective projection image;
The image generation system according to claim 1. In the morphing section constituted by the at least two panoramic images, when the position input receiving means receives a position in the morphing section,
The image generation calculation processing means includes:
Generating a perspective projection image in the morphing section based on panoramic images located at both end points of the morphing section;
The image generation system according to claim 1. The image generation calculation processing means includes:
When the morphing section is switched, the perspective projection image is generated based on the panoramic image after converting the interpolation parameter in the morphing section that is the current processing target into the interpolation parameter of the morphing section that is the next processing target. To
The image generation system according to claim 13. The image generation system includes:
Performing a process of combining the perspective projection image generated by the image generation calculation processing unit with a real image, a CG image, or an image obtained by another image generation method, and outputting the combined image by the display processing unit;
The image generation system according to claim 1, wherein the image generation system is an image generation system.

Images