JP2001229397A - Method and system for constructing three-dimensional virtual space and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To construct a continuous three-dimensional virtual space for providing a virtual environment having no sense of incompatibility in forward and back ward movements, translation and rotation by using a comparatively simple image synthesizing process. SOLUTION: A plurality of all around panoramic images 100 around each vertex of regular triangles which are aligned in a mesh form in a real space are stored. When the movement of a point of observation and rotation of a line of sight are indicated by using a mouse 103 and a keyboard 102, the image 100 that is selected by a panoramic image selector 105 is inputted to a display image production device 107 on the basis of the result of analysis of an input analysis device 104. The device 107 synthesizes the images which are seen from the point of observation during movements or rotations, and this synthetic image is shown on an image display device 101 by an image display controller 110.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a technique for constructing a three-dimensional virtual space.

[0002]

2. Description of the Related Art Computer graphics (CG)
There is a method of constructing a three-dimensional virtual space by using In this method, a three-dimensional model is prepared in advance, and a scene viewed in an arbitrary direction from an arbitrary position is drawn on a screen by making full use of techniques such as texture mapping and rendering. However, in order to realize a natural three-dimensional virtual space that can freely walk around, there are many difficult technical issues such as acquisition of three-dimensional model data.

As another method of realizing a three-dimensional virtual space, a panoramic image representing a scene around the point is synthesized from a plurality of two-dimensional photographed images photographed at a point, and an arbitrary image is obtained from the panoramic image. There is a method in which an image representing a scene seen from an observation point is cut out and displayed on a screen.
However, since the image representing the scene at the time of the forward movement of the observation point is created by a simple enlargement process of the extracted image,
As one moves forward, the image quality of the scene deteriorates. In addition, since the scene is represented by a single panoramic image, the degree of freedom and range of motion are limited.

Another method is disclosed in, for example,
As disclosed in JP-A-91814, there is a method that allows a user to move between several panorama images prepared in advance by a technique called “hot link”. However, even though it is possible to move between panoramic images, it is not possible to see changes in the environment when moving from one panoramic image to another panoramic image without a break, and a continuous three-dimensional virtual space is constructed. Can not.

[0005]

In view of the above problems,
An object of the present invention is to realize a method and a system for constructing a three-dimensional virtual space that can provide a continuous virtual environment without discomfort during movement of an observation point or a line of sight by relatively simple image synthesis processing.

[0006]

According to the present invention, a plurality of all-around panoramic images centered on discretely arranged points in a real space are prepared as digital data, and at least a panoramic image having a center close to an observation point is prepared. A continuous three-dimensional virtual space is constructed by creating and displaying an image representing a scene viewed from an observation point using two panoramic images of the entire circumference. More specifically, the center of each perimeter panoramic image is located at each vertex of an equilateral triangle arranged in a mesh on the real space.

In the present invention, in order to provide a continuous virtual environment without discomfort during forward movement of the observation point, a first full-circle panoramic image having a center close to the observation point,
An image representing a scene seen from the observation point when the observation point moves forward is synthesized using the second full-circle panorama image having the closest center in the line of sight from the center of the full-circle panorama image. In addition, in order to maintain the resolution of the central part of the scene, pixel information of the central part of the image representing the scene is provided from the second full-panoramic image, and pixel information of the peripheral part is provided from the first full-panoramic image. And a synthesis method.

In the present invention, in order to provide a continuous virtual environment without discomfort during the backward movement of the observation point, a first full-circle panoramic image having a center close to the observation point,
And a second full-circle panoramic image having the closest center in the direction opposite to the line of sight from the center of the full-circle panoramic image. In addition, in order to maintain the resolution of the central portion of the scene, pixel information of the central portion of the image representing the scene is provided from the first full panoramic image, and pixel information of the peripheral portion is provided from the second full panoramic image. And a synthesis method.

In the present invention, in order to provide a continuous virtual environment without discomfort during translational movement of the observation point, a first full-circle panoramic image having a center close to the observation point, Using the second full-circle panoramic image having the closest center in the translation direction of the observation point from the center of the image,
An image representing a scene seen from the observation point during the translational movement of the observation point is synthesized. In order to express a scene with parallax corresponding to translation, in the composite image, pixel information of a portion where the first and second full-circle panoramic images overlap is determined by using the center of the first full-circle panoramic image and the second The first and second full-circle panoramic images are combined by weighted addition using a weight corresponding to the distance between the center of the full-circle panoramic image and the observation point.

In the present invention, in order to provide a continuous virtual environment without a sense of incongruity during the rotational movement of the line of sight, a new omnidirectional panoramic image is synthesized from three omnidirectional panoramic images having centers close to the observation point. From the panoramic image of the entire circumference, an image representing a scene seen during the rotational movement of the line of sight is created.

[0011]

FIG. 1 shows a block configuration of a three-dimensional virtual space construction system according to an embodiment of the present invention.
This three-dimensional virtual space construction system includes an image display device 10
1. Keyboard 102, mouse (pointing device)
103, input analysis device 104, panorama image selection device 1
05, a panoramic image storage device 106, a display image creation device 107, and an image display control device 110. Such a system is generally realized by software using resources of a general-purpose or dedicated computer, but can also be realized by dedicated hardware.

The panoramic image storage device 106 digitally stores a plurality of all-around panoramic images 100 centered on a plurality of discrete points (points 1,..., Point n) having a predetermined relative positional relationship in the real space. It is stored in advance as data. The centers (photographing points) of the plurality of all-around panoramic images are arranged at respective vertices (shown by black circles) of regular triangles arranged in a mesh in the real space, for example, as shown in FIG. Each circle in FIG. 2 represents a full-circle panoramic image.

Note that an image such as an all-around panoramic image is represented as digital data, but may be simply referred to as an image in this specification.

The keyboard 102 and the mouse 103 are used to move the observation point or the line of sight in the three-dimensional virtual space (forward, backward, translation or rotation), and the amount of movement (forward distance,
It is used by the user to instruct a retreat distance, a translation distance, a rotation angle, and the like. Input analysis device 10
Reference numeral 4 analyzes information input from the keyboard 102 or the mouse 103, and determines the current position and line-of-sight direction of the observation point in the three-dimensional virtual space, the type of specified movement (forward, backward, translation or rotation), and the amount of movement. The determination is performed, and the determined information is provided to the panoramic image selection device 105 and the display image creation device 107. The panoramic image selection device 105 is a
Display image creation device 1 based on the information given by
07, a required panoramic image is selected from the panorama image storage device 106 and the display image creation device 10 is selected.
7 is output. The display image generating device 107 includes:
From the panoramic image input from the panoramic image storage device 106, a display image representing a scene viewed from the observation point is created and passed to the image display control device 110. The image display control device 110 displays the display image on the image display device 10.
1 is displayed on the screen.

The operation of the three-dimensional virtual space construction system will be described below. When the three-dimensional virtual space construction system is started, for example, one full-circle panoramic image centered on a predetermined point or one full-circle panoramic image selected in advance by the user is displayed in the panoramic image selecting device 1.
The panoramic image is selected from the panorama image and output to the display image creation device 107 from the panorama image storage device 106. The display image creation device 107 cuts out a range of a viewing angle (corresponding to the size of the screen of the image display device 101) in a predetermined line of sight direction at the center from the panoramic image of the entire circumference, and displays the cut out image for the first time. The image is passed to the image display control device 110 as an image. For example, the panoramic image 100A around the point A in FIG.
7, the default line-of-sight direction is set to the direction from point A to point B, an image A in a range of the viewing angle φ at point A (observation point) is cut out, and this is displayed as the first display image. Transfer to control device 110.

Thereafter, it is assumed that the user instructs, for example, forward movement by using the mouse 103 or the keyboard 102 while viewing the screen of the image display device 101 on which the first display image is displayed. The input analysis device 104 analyzes the input information and provides the panorama image selection device 105 and the display image creation device 107 with information on the current position of the observation point (for example, point A), forward motion, and forward distance. The advance distance is determined according to, for example, the length of time during which a button of the mouse 103 or a specific key of the keyboard 102 is pressed. Here, the forward movement is movement of the observation point in the direction of the line of sight. In the case of the forward movement, a display image is created in which the object at the center of the scene looks gradually larger (approaches) as the observation point advances. In order to create such a display image, different panoramic images are required as the observation point moves. A necessary panoramic image is selected by the panorama image selection unit 105 and output from the panorama image storage unit 106 to the display image creation unit 107.

For example, when the observation point advances from the point A to the point P in FIG. 3, the panoramic image selection device 105 uses the panoramic image panorama centering on the nearest point B in the forward direction (sight direction) from the point A. The image 100B is selected, and the selected image 100B is transferred from the panoramic image storage device 106 to the display image creation device 10
7 is output. An all-around panoramic image 100A centered on the point A has already been input and stored in the display image creation unit 107. The display image creation device 107 sequentially creates a display image representing a scene seen from the observation point during the forward movement.

The method of creating a display image representing a scene seen at point P is as follows. As shown in FIG. 3, the display image creation device 107 cuts out the image A in the range of the viewing angle φ at the point A from the panoramic image 100A. The image A is enlarged at a magnification corresponding to the distance from the point A to the point P, that is, the forward distance of the observation point, and a portion (image a) corresponding to the range of the viewing angle φ at the point P of the enlarged image is cut out. In other words, the peripheral portion of the enlarged image is trimmed to match the size of the display image. Note that a portion corresponding to the image a may be directly cut out from the full-circumference panoramic image 100A and may be enlarged. In addition, an all-around panoramic image 100B
Thus, the image B corresponding to the range of the viewing angle φ at the point B is cut out. The image B is reduced at a magnification corresponding to the forward distance. That is, the image B is reduced to a size corresponding to the range of the viewing angle φ at the position advanced by the forward distance from the point B. Then, the image a is placed around the reduced image b of the image B.
Is used to synthesize a display image representing the scene seen at the point P. In other words, the pixel information of the peripheral portion of the display image is provided from the panoramic image A, and the pixel information of the central portion of the display image is provided from the panoramic image B. By such a combining method, a display image with a high resolution at the center can be created.

In the same manner, display images at each point from the point A to the point B are sequentially created and displayed on the image display device 101. When the observation point further advances and reaches the point B, the image B cut out from the all-around panoramic image 100B is displayed as a display image. When the observation point further advances, a display image is created and displayed in a similar manner from the full-circle panoramic image 100C (not shown) and the full-circle panoramic image B centered on the closest point C in the line of sight of the point B. Is done.

Note that, conceptually, the display image at the point B is created by the same combining method as the display image at the point passing the point B. However, since the image cut out from the full-circle panoramic image 100C (not shown) is reduced to zero size and then synthesized, as a result, a display image at the point B is created using only the full-circle panoramic image 100B. It becomes a form. This may be similarly understood in the case of the backward movement described below.

Since the display images created in this way are sequentially displayed on the screen of the image display device 101, the user looking at the screen of the image display device 101 can view the natural three-dimensional space as if it were continuous. You can experience the feeling of moving forward. In addition, since the central portion of the scene maintains a good resolution, the advance distance is not limited in terms of the image quality of the scene.

Next, the mouse 103 or the keyboard 102
Assume that a backward movement is instructed. At this time, the input analysis device 104 provides the current position of the observation point, information indicating the retreating motion and the retreat distance to the panoramic image selecting device 105 and the display image creating device 107. The retreat distance is
For example, it is determined according to the length of time during which a button of the mouse 103 or a specific key of the keyboard 102 is pressed. Here, the backward movement means that the observation point moves in the direction opposite to the line of sight. In the case of the backward movement, a display image is created in which the object at the center of the scene looks gradually smaller (moves away) as the observation point moves. In order to create such a display image, different panoramic images are required as the observation point moves.

For example, in the case of retreating from point B to point A in FIG. 4, the panoramic image selecting device 105 centers on the closest point A in the retreating direction (the direction opposite to the line of sight) from point B. The entire circumference panoramic image 100A is input from the panoramic image storage device 106 to the display image creating device 107. All-around panoramic image 10 centered on point B
0B has already been stored in the display image creation unit 107.

The method of creating the display image of the scene seen at the point P in the middle of the retreat is processingally equivalent to the forward motion described with reference to FIG. However, as can be understood from FIG. 4, pixel information of the central portion of the display image is provided from the panoramic image 100B corresponding to the current position of the observation point, and pixel information of the peripheral portion of the display image moves in the backward direction. In terms of being provided from a certain panoramic image 100A, the process is conceptually the reverse of the case of forward movement. Further, display images at each point from the point B to the point P are sequentially created, but the order is reverse to that in the case of the forward movement, and the display images at points near the point B are created first. Thus, the user can simulate the feeling of retreating in a seamless natural three-dimensional space. Also, good resolution is maintained at the center of the scene.

A current general digital camera has an optical zoom mechanism and an electronic zoom mechanism. In the electronic zoom mechanism, since the zoom-in is realized by cutting out a part of the image and performing an enlargement process, the image quality is inferior to the image zoomed in by the optical zoom mechanism. In contrast,
According to the method of synthesizing the displayed images at the time of the forward movement and the backward movement described above, the resolution at the center of the image does not deteriorate, so that a zoom function close to that of the optical zoom mechanism can be realized.

FIG. 5 shows such a zoom state. In the figure, forward movement of the observation point from right to left corresponds to zoom-in, and backward movement from left to right corresponds to zoom-out. Black circles indicate points 1 to n, which are the centers of the all-around panoramic images 100 stored in the panorama image storage device 106 in advance, and images 1 to n are displayed when the observation point is located at points 1 to n. This is an image that is directly cut out from one entire panoramic image centered on the point. The displayed images at the points between the points 1 to n are two behind each other.
This is a “composite image” in FIG. 5, and the resolution at the center is images 1 to
It is equivalent to n.

Next, it is assumed that the user instructs a translational movement. Translational movement means that the observation point moves in a direction orthogonal to the line of sight. For example, as shown in FIG. 6, consider a case where translation is performed from point A to point B. The image A is an image obtained by cutting out the range of the viewing angle φ at the point A from the all-around panoramic image 100A, and the image B is an image obtained by cutting out the range of the viewing angle φ at the point B of the all-around panoramic image 100B. A display image (synthetic image) representing a scene seen at point P in the middle of translation is created as follows using images A and B.

In FIG. 6, the space of the viewing angle φ at the point P is divided into A ′, D, and B ′. The space D is a space where a part of the image A having a parallax and a part of the image B are overlapped and projected. The space A ′ is a space where only a part of the image A is projected, and the space B is a space where only a part of the image B is projected. Paying attention to this, using the weight corresponding to the distance of the point P from the points A and B (in other words, the weight corresponding to the moving distance from the point A), the portion corresponding to the space D of the images A and B The pixel information of the D portion of the display image is synthesized by the weighted addition, and the pixel information of the corresponding portion of the images A and B is used as the pixel information of the portion corresponding to the spaces A ′ and B ′ of the display image. That is, the pixel information of the overlap portion D of the display image is given from the full-width panoramic image 100A and the full-width panorama image 100B, and the pixel information of the A ′ portion of the display image is given from the full-width panorama image 100A. The pixel information of the B ′ portion of
B. According to such a combining method, a more natural scene with parallax corresponding to the position of the observation point can be displayed.

By the same method, display images representing scenes seen at points A to B are sequentially synthesized. Since the display images synthesized in this way are sequentially displayed on the image display device 101, the user experiences a feeling as if translating from the point A to the point B in a seamless natural three-dimensional space. can do. When translating beyond point B, the panoramic image 100
A display image representing a scene is synthesized and displayed in a similar manner using B and the entire panoramic image on the right side of B.

Next, it is assumed that the user instructs a rotational movement. The rotational motion is a motion of horizontally moving the line of sight, that is, a horizontal rotation of the line of sight. The rotation angle is specified by, for example, the pressing time of a button of the mouse 103 or a specific key of the keyboard 102. In the case of the rotational motion, the panoramic image selecting device 105 selects three full-circle panoramic images centered on a point near the observation point, and causes the display image creating device 107 to input the panoramic image. The display image creation device 107 creates a display image representing a scene at the time of the rotational movement using the three panoramic images. This will be described with reference to FIG.

When a rotational movement is made at a point P in the hatched area shown in FIG. 7, panoramic images 100A, 100B, 100 around the points A, B, and C closest to the point P are used.
Using C, a new panoramic image 100P around the point P is synthesized. This panoramic image 100P
The PA, PB, and PC portions of the panoramic image 100
The A ', B', and C 'portions (thick line portions) of A, 100B, and 100C are pasted, but the connection end portions of the PA, PB, and PC portions have the common (over) portions of A', B ', and C'. A pasting process is performed using the (wrapped) image data. An all-around panoramic image 10 synthesized in this way
From 0P, the range of the viewing angle at each point during rotation is cut out as a display image and displayed on the image display device 101.
In this way, the user can simulate the sensation of performing a rotating motion in a continuous natural three-dimensional space.

However, in the case of performing a rotational movement around the center of a certain panoramic image as an observation point, an image obtained by cutting out the range of the viewing angle at each point in the panoramic image is used as a display image. For example, when performing a rotary motion at the point A, a display image is cut out from the entire circumference panoramic image 100A.

The three-dimensional virtual space construction processing according to the present invention as described above can be realized by software using the resources of a general-purpose or special-purpose computer as described above.
For example, the CPU 200 and the main memory 20 as shown in FIG.
1, a hard disk 203, an input device 207 such as a keyboard and a mouse, a display device 204 such as a CRT display and a liquid crystal display, a display controller 205 for controlling the display device 204, a frame buffer memory 206 for storing display data, various storage media ( Magnetic disk, optical disk, magneto-optical disk, memory card, etc.) 210
A medium drive 209 for reading and writing data, a communication device (modem, network adapter, etc.) 208 for communication with external devices and networks, etc., are implemented by software using a computer having a general configuration connected via a bus 211. You can also.

In this case, from among a plurality of all-around panoramic image data prepared in advance and centered on discretely arranged points in the real space, at least two all-around panoramic images having a center close to the observation point are provided. A step of selecting image data, a step of using the omnidirectional panoramic image data selected in this step to create image data representing a scene seen from the observation point, and a step of transmitting the image data created in this step to the display device 204. A program for causing a computer to execute a three-dimensional virtual space construction process including at least a displaying step is read into the main memory 201 by the medium drive 209 from the storage medium 210 in which the program is recorded, or communicated from an external device. Read into the main memory 201 by the device 208,
Alternatively, the program is read from the hard disk 203 into the main memory 201 and executed by the CPU 200. This program may be implemented as a semiconductor ROM in which the program is fixedly stored. Storage media such as various storage media 210 and semiconductor ROMs on which such programs are recorded are also included in the present invention. Further, the full-circle panoramic image data is input from, for example, the communication device 208 or the medium drive 209 and temporarily stored in the hard disk 203, and one or a plurality of necessary full-circle panoramic image data is stored in the main memory 201 from the hard disk 203. Read and processed. Note that two-dimensional image data photographed by a digital camera or the like may be input to a computer, and the entire circumference panoramic image data may be created from the two-dimensional image data in the computer and stored in the hard disk 203.

[0035]

As described in detail above, according to the first to seventh aspects of the present invention, a continuous three-dimensional image can be obtained by relatively simple image synthesizing processing only by preparing a plurality of panoramic images. It is possible to construct a virtual space and freely move an observation point or a line of sight in the space. According to the invention of claims 3 to 6, it is possible to provide a continuous virtual environment without discomfort during forward movement, backward movement, translation movement, and rotation movement. According to
It is possible to avoid deterioration of the image quality of the scene when moving forward or when moving backward. According to the invention described in claim 5, a natural scene having parallax according to translation can be expressed. According to the invention of claim 8, it is possible to easily construct such a three-dimensional virtual space using a general computer.
And so on.

[Brief description of the drawings]

FIG. 1 is a block diagram of a three-dimensional virtual space construction system according to the present invention.

FIG. 2 is a diagram showing an arrangement of a center of an all-around panoramic image in a real space.

FIG. 3 is a diagram illustrating a method of synthesizing a display image representing a scene during a forward motion.

FIG. 4 is a diagram for explaining a method of synthesizing a display image representing a scene at the time of a backward movement.

FIG. 5 is an explanatory diagram of a zoom function realized by a forward movement or a backward movement in a three-dimensional virtual space.

FIG. 6 is a diagram illustrating a method for synthesizing a display image representing a scene during a translational motion.

FIG. 7 is a diagram illustrating a method of synthesizing a display image representing a scene during a rotational motion.

FIG. 8 is a block diagram illustrating an example of a computer that can be used for constructing a three-dimensional virtual space according to the present invention.

[Explanation of symbols]

 REFERENCE SIGNS LIST 100 panoramic image 101 image display device 102 keyboard 103 mouse 104 input analysis device 105 panorama image selection device 106 panorama image storage device 107 display image creation device 110 image display control device

Claims (8)

[Claims] 1. A plurality of omnidirectional panoramic images centered on discretely arranged points in a real space are prepared as digital data, and at least two omnidirectional panoramic images having a center close to an observation point are used. A method for constructing a continuous three-dimensional virtual space by creating and displaying an image representing a scene seen from an observation point. 2. The three-dimensional virtual space construction method according to claim 1, wherein the center of each panoramic image is arranged at each vertex of an equilateral triangle arranged in a mesh on the real space. 3. A first all-around panoramic image having a center close to the observation point and a second all-around panoramic image having a center closest to the line of sight from the center of the first all-around panoramic image. An image representing a scene seen from the observation point during forward movement of the observation point is synthesized using the pixel information at the center of the synthesized image from the second full-width panoramic image and the pixel information at the periphery of the image at the first full panorama. The three-dimensional virtual space construction method according to claim 1, wherein the three-dimensional virtual space construction method is provided from a circumferential panoramic image. 4. A first full-circle panoramic image having a center close to an observation point, and a second full-circle panorama having a closest center from the center of the first full-circle panoramic image in a direction opposite to a line-of-sight direction. An image representing a scene seen from the observation point during the backward movement of the observation point is synthesized using the image and the pixel information at the center of the synthesized image is obtained from the first panoramic image of the entire circumference, and the pixel information at the periphery is obtained from the first panoramic image. 3. The method according to claim 1, wherein each of the plurality of panoramic images is provided from each of the two panoramic images.
Dimension virtual space construction method. 5. A first full panoramic image having a center close to the observation point, and a second full panorama having a closest center from the center of the first full panoramic image in the translation direction of the observation point. Using the image, an image representing a scene seen from the observation point during the translational movement of the observation point is synthesized, and pixels of a portion of the synthesized image where the first full-circle panoramic image and the second panoramic image overlap each other The information includes a first full-circle panoramic image and a second full-circle panorama using weights according to the distance between the center of the first full-circle panoramic image and the center of the second full-circle panoramic image and the observation point. 3. The method according to claim 1, wherein the image is determined by weighted addition of images.
Dimension virtual space construction method. 6. A new panoramic image is synthesized from three panoramic images having a center close to the observation point, and an image representing a scene seen from the observation point when the line of sight is rotated is created from the new panoramic image. The method according to claim 1, wherein the method is performed. 7. A panoramic image storage unit for storing a plurality of panoramic images as digital data, an input unit for inputting information relating to observation point and line-of-sight movements, and information input by the input unit. Input panorama image selecting means for selecting an entire panorama image according to the analysis result by the input analysis means, and outputting the selected panoramic image from the panorama image storage means, Display image creation means for creating a display image representing a scene seen from an observation point from a peripheral panorama image in accordance with the analysis result by the input analysis means, and an image for displaying the display image created by the display image creation means on an image display device And a display control means for implementing the three-dimensional virtual space construction method according to claim 3, 4, 5, or 6. A three-dimensional virtual space construction system, characterized in that: 8. At least two panoramic panoramic images having a center close to an observation point from among a plurality of panoramic panoramic image data prepared in advance, each centering on a discretely arranged point in a real space. A step of selecting data, a step of creating image data representing a scene seen from the observation point using the panoramic image data selected in this step, and a step of displaying the image data created in this step. A computer-readable storage medium on which a program for causing a computer to execute a three-dimensional virtual space construction process including the program is recorded.