JPH113437A - Image synthesizer and image synthesizing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image synthesizer and an image synthesizing method which synthesize an image that has high reality and rich diversity. SOLUTION: An image that is photographed by a video camera 26 is written as a texture in a texture storing part 308, and the texture that is stored in the part 308 is mapped to an object (such as a player character and a racing car) in three-dimensional virtual space. An image that is photographed by the camera 26 in real time is mapped in real time, and an image that is photographed by a still picture camera before game play starts is mapped. The head image of a player is photographed and mapped. Plural images which are acquired by photographing the player in plural directions are mapped to an object that represents the head of the player. Photographed images of 1st and 2nd players are mapped to 1st and 2nd objects respectively, the 2nd and 1st objects are shown on 1st and 2nd screens respectively which are seen by the 1st and 2nd players.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an image synthesizing apparatus and an image synthesizing method.

[0002]

BACKGROUND ART In recent years, in the fields of a drive simulator, a flight simulator, a game device, a system kitchen simulated experience system, and the like,
Various image synthesizing apparatuses capable of realizing so-called virtual reality have been proposed. And in this kind of image synthesis device, how to bring the virtual world closer to the real world is
It is a major technical challenge. As a technique for realizing such a technical problem with small-scale hardware, a technique called texture mapping can be considered.

However, in the conventional texture mapping, a texture storage unit for storing a texture to be mapped to an object in a virtual three-dimensional space is constituted by a texture ROM in which data cannot be electrically rewritten. For this reason, only the texture preset by the manufacturer is always mapped to the object in the virtual three-dimensional space, and the obtained image is monotonous. That is, the first texture is always mapped to the first object in the virtual three-dimensional space, and the first object to which the first texture is always mapped is displayed on the screen viewed by the player. . For this reason, it was not possible to diversify the images to be synthesized.

The present invention has been made to solve the above technical problems, and an object of the present invention is to provide an image synthesizing apparatus and an image synthesizing method capable of synthesizing images with high reality and variety. Is to provide.

[0005]

In order to solve the above-mentioned problems, an image synthesizing apparatus according to the present invention comprises a texture storage unit for storing a texture to be mapped on an object in a virtual three-dimensional space, and an image picked up by an image pickup unit. Means for writing the obtained image as the texture in the texture storage means, and mapping the texture stored in the texture storage means to an object in a virtual three-dimensional space, and forming an image viewed from a given viewpoint in the virtual three-dimensional space. Image combining means for combining.

[0006] In the conventional texture mapping, the texture storage means is constituted by a texture ROM, and objects (player characters,
Only a predetermined texture could be mapped to a racing car. On the other hand, according to the present invention, the image captured by the image capturing unit can be freely written as a texture in the texture storage unit, and the captured image can be mapped to an object in the virtual three-dimensional space. . As a result, the texture to be mapped can be diversified, and an image with high reality and richness can be synthesized.

Further, the present invention is characterized in that an image captured in real time by the image capturing means is mapped in real time to an object in a virtual three-dimensional space. By doing so, the texture mapped to the object in the virtual three-dimensional space changes in real time,
The virtual reality can be further improved.

Further, the present invention is characterized in that an image picked up by an image pickup means before the start of game play is mapped to an object in a virtual three-dimensional space. This makes it possible to change the texture to be mapped to the object in the virtual three-dimensional space every time the game is played.

Further, the present invention is characterized in that an image obtained by imaging the player by the imaging means is mapped to an object in a virtual three-dimensional space. This makes it possible to use the image captured by the image capturing means as the player's identification image.

[0010] The present invention is characterized in that a plurality of images obtained by imaging the player from a plurality of directions by a plurality of imaging means are mapped to an object in a virtual three-dimensional space. By doing so, images taken from the front, right, rear, and left of the player, for example, are read, for example, forward, right, rear, and left of the object (for example, the head of the player character) in the virtual three-dimensional space. Can be mapped to

Further, the present invention is characterized in that an image of the player's head obtained by imaging the player's head by the imaging means is mapped to an object representing the player's head in a virtual three-dimensional space. . By doing so, it is possible to make the image of the object representing the player's head full of reality.

Further, according to the present invention, an image of the first player picked up by the first image pick-up means is converted into a first image in a virtual three-dimensional space.
And maps the image of the second player imaged by the second imaging means to the second object in the virtual three-dimensional space, and displays the second image on the first screen viewed by the first player. Is displayed, and the first object is displayed on a second screen viewed by a second player. In this way, the first player can see the second object on which the image of the second player is mapped, and the second player can see the image of the first player on which the image of the first player is mapped. It is possible to see the first object that has been set.
This makes it possible to provide an image optimal for a multi-player game.

The object in the virtual three-dimensional space can be constituted by, for example, a polygon.

[0014]

Next, preferred embodiments of the present invention will be described in detail with reference to the drawings.

1. First Embodiment FIG. 1 shows a block diagram of a first embodiment using the present invention. FIG. 2 shows an example of a virtual experience device in which this is applied to a driving game.

In FIG. 2A, a real racing car 30 for a game is installed on a floor 4 in a dome 1 whose inside is entirely painted in blue. Here, as with the inside of the dome 1, the floor 4 is all painted in blue. And this racing car 30
, An operator, for example, a player 50 in a game is on board. It is assumed that the term “operator” includes not only a player in a game but also all persons who use a virtual experience device such as a pilot operating a flight simulator and a player playing sports.

The racing car 30 includes, for example, a vehicle body 36, tires 32, a rear wing 34, a steering wheel 40,
It is configured to include a side mirror 41, an instrument panel 44, a shift gear, an accelerator, a brake (not shown), and the like.
For example, the front tire 32 is formed so as to be freely steered by operating the handle 40 of the player 50, and the rear wing 34 also moves up and down due to the operation of the player or a change in the speed of the racing car 30. It is formed as follows. Then, as described later, the player 50 can see the change of these movements by the video camera 10.

The instrument panel 44 includes, for example, a speedometer, a tachometer, a fuel gauge, and a warning gauge (not shown).
It is configured to change according to the driving state of the player 50. That is, the speedometer and the tachometer change according to the accelerator operation, the brake operation, and the shift gear operation of the player 50, and the fuel gauge is configured to indicate the end of the game when the fuel runs out. Further, when a trouble occurs in the engine of the racing car 30 or the like, the warning gauge blinks, and the player 50 can know this by the video camera 10.

An attitude control unit 24 is provided below the racing car 30 to control the racing car according to the game situation (change of road surface, change of road, etc.), steering operation of the player 50, accelerator operation, and brake operation. 30 posture changes and acceleration changes are controlled. Thus, the player 50 can experience a change in the posture and a change in the acceleration G,
You will be able to experience a virtual world closer to the real world.

The head mounted body 9 is mounted on the player 50 so as to cover the field of view of the player 50. The head-mounted body 9 includes a video camera 10 and a space sensor 1 for a player.
2. It includes an image display device 20 and a speaker 22.

FIGS. 4A and 4B show an example of the shape of the head mounted body 9. FIG. 4A shows a video camera 10, a space sensor 12 for a player, and an image display device 2.
0, the wearing body 9 configured by providing the speaker 22 on the helmet 14 is shown. According to this type of mounting,
When the player 50 wears the helmet 14, a world completely isolated from the outside can be created, so that a more realistic virtual reality can be enjoyed. On the other hand, the mounting body 9 shown in FIG. 4B is configured such that the video camera 10, the spatial sensor 12 for the player, the image display device 20, and the speaker 22 are integrally attached to the mounting band 16. Therefore, it is possible to realize a more comfortable wearing feeling. The mounting body used in the present embodiment is not limited to the shapes shown in FIGS. 4A and 4B, and various shapes can be used.

The image display device 20 is mounted in front of the player 50 so as to cover the field of view of the player 50, and converts an image signal transmitted from the image synthesizing device incorporated in the vehicle body 36 through the connection line 18 into an image. To display. As an image display method in this case, it is desirable to use a small display such as a color liquid crystal display or a small CRT in order to reduce the size of the head mounted body 9 and improve the wearing feeling. In addition, it is desirable that the image be corrected by an optical system so that the image is focused on the eye and the viewing angle is widened and the sense of reality is improved.

Here, the shape of the small display may be such that it covers the field of view of the player along the shape of the face of the player 50 to obtain a panoramic image effect, or two small displays may be used. Each of them may be formed in front of both eyes of the player 50. In the latter case, it is desirable to form a three-dimensional image by giving an image with parallax to a two-dimensional two-dimensional image given to both eyes. With such a configuration, the size of the object and the distance to the object can be grasped, so that a virtual world closer to the real world can be created.

The video camera 10 is used by the player 50 to view the real world. For example, as shown in FIG. 4, the video camera 10 is set at a position close to the viewpoint position (eye position) of the player 50, and its angle is set. It is also desirable to set the same as the field of view of the player 50. This is because, by setting in this way, the real-world video that is actually viewed from the player 50 can be viewed more comfortably. As a result, for example, the player 50 can check a racing car of another player who is chasing from behind by an operation in which the player 50 actually looks back.
It is possible to create a tense game world.

In this case, however, another video camera is attached to, for example, the rear of
The user may be configured to be able to switch and view this by the operation of. Further, for example, the racing car 50 may be provided with a side mirror 41, a rear-view mirror, and the like, and a structure such as texture mapping or the like may be used to project an image that should actually be seen behind.

As the imaging means used as the video camera 10, it is desirable to use, for example, a high-resolution and smaller camera, for example, a high-resolution CCD camera, and it is desirable to use one having a function such as automatic focusing. .

The spatial sensors 12 for the players are sensors for detecting three-dimensional information of the player 50. For example, one side is on the ceiling of the dome 1 and the other side is on the player 50 side.
Is attached to the head mounted body 9.

Here, the three-dimensional information of the player 50 refers to, for example, positional information of the player 50 and view direction information, and obtains a relative positional relationship including the directions of the space sensors 12, 12 in the three-dimensional space. Is detected by However, if the position of the player 50 does not move in the setting of the game or is within the allowable range even if it moves, the three-dimensional information does not necessarily need to include the position information of the player 50. Conversely, when the direction of the field of view of the player 50 is not changed in setting the game, the three-dimensional information does not necessarily include the player 50.
Need not be included. In these cases, subsequent image calculation processing may be performed based on the position or view information set on the game.

As a method of detecting three-dimensional information by the space sensors 12, 12, for example, the following method can be considered. That is, each of the space sensors 12, 12 is constituted by three coils orthogonal to each other. Then, a current is caused to flow through the coil of one of the space sensors 12, and at this time, the positional relationship including the directions of the space sensors 12, 12 is detected from the current value induced in the coil of the other space sensor 12. . As a result, the spatial information of the player 50 is detected.

The method of detecting three-dimensional information by the space sensors 12, 12 is not limited to the method using the above-described dynamic magnetic field, but may be, for example, a method using a static magnetic field or a method using an ultrasonic wave. Is also good.

Next, an image synthesizing method according to the present embodiment will be described below.

In this embodiment, as shown in FIG. 2B, a real space image 100 in a real three-dimensional space captured by the video camera 10 and a virtual view image 102 in a virtual three-dimensional space are synthesized. , A display image 104 is formed. Then, the display image 104 is output to the image display device 20 through the connection line 18 and is actually
Is a view image to be viewed.

In the present embodiment, the image synthesis is performed by blue mat synthesis. In other words, racing car 3
0 and its accessories, the player 50 itself, etc., that is, the inside of the dome 1 and the floor 4 are all colored blue. By doing so, the real space image 10
At 0, all parts other than the racing car 30, the steering wheel 40, and the player's hand 54 have a blue background. And
The display image 104 can be obtained by setting all the pixels of the blue color portion in the real space image 100 as empty dots and superimposing them on the virtual view image 102. In this case, for example, the dome 1 shows the background of the circuit mainly seen from the player 50, and the floor 4 shows the road surface condition of the road on which the racing car 30 runs.

FIG. 1 is a block diagram showing an example of the image synthesizing apparatus in this case.

In FIG. 1, the virtual visual field image calculation device 70
Is for calculating a virtual image that can be viewed by the player 50 in the virtual three-dimensional space, and includes a virtual three-dimensional space calculation unit 72 and a view image calculation unit 76. And
The virtual three-dimensional space calculation unit 72 calculates a virtual three-dimensional space that represents a game space set by being superimposed on the real three-dimensional space. The view image calculation unit 76 calculates the virtual three-dimensional space that has been calculated. Thus, a virtual field-of-view image in a direction that can be seen from the player 50 is calculated.

The program section 74 stores a game calculation program. For example, taking the driving game of the present embodiment as an example, all objects (racing cars, roads, other racing cars, backgrounds visible from roads, etc.) on the circuit constituting the driving game space are represented by a combination of polyhedrons, Three-dimensional information and associated data of each vertex of the polyhedron are stored as image information. In addition to the above data, the program section 74 also stores a program for controlling the progress of the game and the like, and stores all the circuit information if the user can enjoy a plurality of circuit runs.

Operation signals input by the player 50 through the steering wheel 40, accelerator, brake, and shift gear are as follows:
The data is input to the virtual three-dimensional space calculation unit 72 via the operation unit 38. Then, the operation signal and the program unit 7
4, the position and direction of the racing car 30 in the virtual three-dimensional space are calculated in real time. Then, based on the calculation result, image information of the racing car 30 in the virtual three-dimensional space and all objects on the circuit around the racing car 30 is output to the view image calculation unit 76.

The visual field image calculation unit 76 performs the following coordinate conversion based on the three-dimensional information detected by the space sensor 12 for the player. That is, as shown in FIG. 12, image information calculated by the virtual three-dimensional space calculation unit 72 (for example, an object 17 such as a signboard, a building, etc. forming a background screen)
0, 172) are converted from the world coordinate system (XW, YW, ZW) in the virtual three-dimensional space to the viewpoint coordinate system (Xv, YV, ZV) of the player 50. Then, the image information of the object 172 such as a signboard passing outside the field of view of the player 50, for example, out of the field of view, is clipped and removed, and the object 1 in the field of view is removed.
The image information 70 is perspectively transformed into a screen coordinate system (XS, YS), that is, a coordinate system that can be actually viewed from the player 50. In this case, the perspective-transformed image information is used as polygons 174, 17 corresponding to the polyhedron constituting the object.
5, 176 (the back side polygon is omitted). Next, the polygons 174, 17
5, 176. The image information of all the dots in 176 is calculated based on the accompanying information such as the color and luminance given to each polygon or each vertex of the polygon, and this is used as virtual view image information as a display image synthesizing device. 80.

In the display image synthesizing device 80, the image synthesis of the virtual visual field image 102 from the visual field image calculation unit 76 and the real space video 100 from the video camera 10 is performed. Various methods can be considered as a method of the image synthesis. In the present embodiment, as described above, the image is synthesized by the method of the blue mat synthesis. FIG. 13 shows details of the display image synthesizing device 80 in this case.

That is, in FIG. 13, an image signal representing the real space image 100 input from the image camera 10 is first passed through a filter 200 in the display image synthesizing device 80 and is divided into three primary color components of RGB. Then, each of these components is A / D converted into 8-bit digital data by the A / D conversion circuit 202, thereby obtaining 24-bit RGB digital data for each pixel. Then, whether or not the 24-bit RGB digital data of each pixel in the real space image 100 matches the 24-bit RGB digital data of blue color painted on the back side of the dome 1 and the floor 4 is determined by whether or not there is an empty dot. The judgment circuit 204 calculates and judges for each pixel.
Then, the result of this determination is written to the empty dot memory 206. The empty dot memory 206 has a configuration of a 1-bit memory corresponding to all pixels of the display image, and empty dot determination data indicating whether or not the pixel is an empty dot is written as 1-bit data for each pixel.

The display image synthesizing device 80 has a built-in field buffer 210 corresponding to each pixel of the display image. Then, the data control unit 208 refers to the empty dot determination data written in the empty dot memory 206, and writes a real space image at each pixel position of the field buffer 210. That is, if the pixel is determined to be a free dot by the free dot determination data, the pixel position in the field buffer 210 is
No real space video is written. Conversely, if it is determined from the empty dot determination data that the pixel is not an empty dot, the 24-bit RGB digital data of the real space video is written as it is.

Next, the data control unit 208 refers to the empty dot determination data written in the empty dot memory 206, and stores the virtual view image calculated by the view image calculation unit 76 at each pixel position of the field buffer 210. Information is overwritten. That is, when it is determined from the empty dot determination data that the pixel is an empty dot, the virtual view image information is written as it is. vice versa,
If it is determined from the empty dot determination data that the pixel is not an empty dot, nothing is written, and a real space image is displayed at this pixel position.

After the above writing, the image information data at each pixel position is read from the field buffer 210 by the data control unit 208. Then, the image information data is output as an image to the image display device 20 through the connection line 18, and the player 50 can see the display image 104 in which the virtual view image 102 is incorporated in the real space image 100 in real time.

It is more preferable that the writing and reading of the image information be performed simultaneously by, for example, configuring the field buffer 210 for two screens.

In the virtual three-dimensional space calculation unit 72, not only the calculation of the image information but also the sound synthesis unit 7
It also generates an audio signal output from the speaker 22 through the controller 8 and an attitude control signal for controlling the attitude of the racing car 30. That is, according to the game progress situation in the game space, which is calculated based on the game program from the program 74 and the operation signal from the operation unit 38,
Generate more effective voice signal and attitude control signal,
The game's presence is further enhanced.

Further, in this embodiment, the virtual view image calculation device 70 is divided into a virtual three-dimensional space calculation unit 72 and a view image calculation unit 76, but this is for convenience. This may be configured by a method of integrally performing the function of performing the calculation and the function of calculating the view image. That is, as long as a virtual view image in a direction visible from the player 50 can be obtained in the virtual three-dimensional space, the calculation order, the calculation method, and the like are not limited to the method illustrated in FIG.

The method of synthesizing images in the display image synthesizing device 80 is not limited to the method described above. For example, image synthesis using red instead of blue, image synthesis using a plurality of colors, Can be used.

The method of obtaining the image information of all the dots in the calculated polygons in the view image calculation unit 76 is not limited to the above-described method, and a method called texture mapping may be used. That is, each vertex of the object in the virtual three-dimensional space is given in advance texture coordinates specifying texture information to be attached. Then, after performing the above-described calculation on the viewpoint coordinate system, the texture information is read from the texture information memory using the texture coordinates as an address, and the texture information is attached to the polygon, thereby obtaining an image of all the dots in the polygon. Compute information.

By calculating image information using such a technique called texture mapping, first,
There is an advantage that the load on the arithmetic processing portion can be significantly reduced. In addition, various unprecedented image effects can be obtained by using this method.
Examples thereof include the following.

That is, the video camera 43 is attached to the side mirror 41 provided on the racing car 30. Then, as shown in FIG. 14, the rear real space image 105 captured by the video camera 43 and the rear virtual view image 106 are image-combined to form a side mirror display image 108.
Is formed, and this side mirror display image 108 is attached to a portion of the side mirror 41 of the display image 104 viewed from the player 50 by using a texture mapping method.
With such a configuration, the side mirror 41 has the configuration shown in FIG.
Similarly to the display image 104 shown in (b), an image in which the rear real space image 105 and the rear virtual view image 106 are combined is displayed. Thus, for example, the racing car 31 of the opponent player chasing from behind.
And the background image in the back and my racing car 30
Can be viewed by the side mirror 41, and a virtual world closer to the real world can be created.

FIG. 15 shows another example of the video effect using the texture mapping method. In this example, as shown in FIG. 15A, the racing car 30 of the player 50 and the racing car 31 of the opponent player 52 are installed in independent dome portions, respectively. A video camera 26 is installed in the dome, and the heads 28 of the players 50 and 52 are photographed from all sides. The photographing data of the player's head 28 is shown in FIG.
As shown in (b), the texture is mapped to the head of the opponent player 52 on the virtual view image 102 by using the texture mapping method. In this way, when the player 50 looks back, the real image of the head of the opponent player can be seen, and the sense of reality of the game can be greatly enhanced. In particular, when the racing car 31 of the opponent player 52 catches up with the own racing car 30 and lines up side by side, it is possible to see a face, etc., which is proud of the victory of the opponent player 52, thereby stimulating each player's consciousness of competition. A game with a sense of reality and tension can be provided.

In FIG. 15, the head 28 of each player is photographed by the video camera 26 from four directions.
It is sufficient that there is video information from at least the front of the player's head 28, and video information from other directions may be the one created by the virtual three-dimensional space calculation unit 70.
Further, the photographing means of the head 28 of each player is not limited to the video camera 26. For example, before starting the game play, the photographing means may be photographed by a still image camera and registered and used.

FIG. 16 shows an example of a simple block diagram for synthesizing an image by the texture mapping method described above.

That is, when synthesizing images by the texture mapping method, the field-of-view image calculation unit 76 is configured, for example, as shown in FIG.

The image information of each object output from the virtual three-dimensional space calculation unit 72 includes the vertex coordinates VX, VY, VZ of the polygons constituting each object and the texture vertex coordinates VTX given to each vertex of the polygon. , VTY, etc. Here, the texture coordinates TX and TY specify texture information to be attached to each polygon, and the texture information includes the texture coordinates TX and TY
Is stored in the texture storage unit 308 as an address. And the texture vertex coordinates VTX and VTY are
The texture coordinates TX and TY represent the texture coordinates at the vertex position of each polygon.

The vertex coordinates VX, VY, VZ, the texture vertex coordinates VTX, VTY, etc.
00 is input. Then, the vertex coordinate transformation unit 300 performs coordinate transformation such as perspective transformation, and outputs the result to the sorting circuit 302. Sorting circuit 3
In 02, the processing priority of each polygon is set. The priority is set so that, for example, a polygon closer to the viewpoint of the player is preferentially processed, and the subsequent processing is performed according to this priority.

The processor unit 304 calculates the display coordinates and texture coordinates T of all the dots in the polygon from the vertex coordinates and texture vertex coordinates of each polygon after the coordinate transformation.
X and TY are required. Then, the obtained texture coordinates TX and TY are written in the field buffer unit 306 using the display coordinates described above as an address.

When displaying an image, texture coordinates TX and TY are read from the field buffer unit 306, texture information is read from the texture storage unit 308 using these as addresses, and a virtual view image is formed. Thereafter, the display image synthesizing device 80 performs image synthesis with the real space video.

With the above configuration, the side mirror display image 108 is displayed on the side mirror 41 by, for example, the following arithmetic processing. Here, the side mirror 41
For example, the image is painted in blue, and an image calculated by the image synthesizing device is set to be displayed at this position.

In this case, first, image synthesis of the side mirror display image 108 shown in FIG. 14 is performed. That is, in FIG. 16, the calculation of the rear virtual view image 106 is performed by the virtual three-dimensional space calculation unit 72 and the view image calculation unit 76. The rear real space image 105 is photographed by the image camera 43. Then, the virtual view image 10 behind this
6 and the rear real space image 105 are synthesized by the display image synthesis device 80, and the side mirror display image 108 is synthesized.
Are image-combined.

Next, the side mirror display image 108
Is returned to the texture storage unit 308 as shown in FIG. Then, in the storage area of the texture storage unit 308, the image information of the side mirror display image 108 is written at the position of the texture coordinate corresponding to the side mirror 41. After that, the display image synthesizing device 80 performs image synthesis of the virtual view image 102 on which the side mirror display image 108 is displayed and the real space image 100 captured by the image camera 10. As a result, an image in which the side mirror display image 108 is displayed can be formed on the side mirror 41 of the display image 104.

When the real image of the head of the opponent player is projected by the configuration shown in FIG. 16, the image captured by the video camera 26 is written directly to the texture storage unit 308 as shown in FIG. . In other words, the video data captured by the video camera 26 is written in real time at the position of the texture coordinates corresponding to the head of the opponent player in the storage area of the texture storage unit 308. This makes it possible to obtain a display image in which a real image of the head of the opponent player 52 is projected on the virtual field-of-view image 102 viewed from the player 50.

In the case where the still image camera is used, a picture of the opponent player is stored as image data in the texture storage unit 308 when registering before starting the game.

Further, the texture mapping method in the present embodiment is not limited to the structure shown in FIG. 16, and any method of texture mapping can be used.

According to the present embodiment having the above-described configuration, it is possible to provide a game full of a sense of reality, which cannot be experienced in a conventional driving game.

That is, first, in the conventional driving game, a game screen for displaying a circuit or the like is displayed on the player 50.
Because it was set only on the front of the, there was still a lack of realism. On the other hand, in the present embodiment, since the player 50 can see all directions, for example, 360 degrees, the sense of reality of the game is greatly improved. In particular, for example, when the present embodiment is applied to a driving game by a plurality of players, the player 50 can actually look back or use the side mirror 41 and the rearview mirror having the above-described configuration to make a chase of the chasing opponent. The player can be confirmed, and the sense of presence can be further increased. In this case, if the real space image of the opponent player is attached by texture mapping as described above, it is possible to further increase the realism and tension of the game.

In the present embodiment, the player 50
It is possible to ride on a real racing car 30 in a real three-dimensional space, and moreover, the real racing car 3
0 can be run freely in a virtual three-dimensional space for games, and a virtual world closer to the real world can be experienced. That is, the player 50 checks the movements of the racing car 30, the tires 32, the rear wing 34, the opponent player and the like on the side mirror 41 in the actual three-dimensional space while actually checking the racing car 30 with his / her own eyes. Can be operated.
Further, since the operation units operated by the player 50, that is, the steering wheel 40, the instrument panel 44, the accelerator, the brake, the shift gear, and the like can be operated while actually confirming with the user's own eyes through the video camera 10 as described above. Therefore, the operability is greatly improved, and the sense of reality of the game is also greatly improved. In this case, if the control of the attitude control unit 24 and the sound output to the speaker 22 are changed according to the game situation, a more realistic game can be provided.

FIG. 3 shows an example of the virtual experience apparatus when the first embodiment is applied to a flight simulator.

In FIG. 3A, a blue mat (hereinafter referred to as a blue mat) is attached to the right window 2 and the left window 3 in the cockpit 45, and the method described above is applied to the blue mat. Display image 1
04 is fitted. This state is shown in FIG. Here, in FIG. 3B, a virtual view image 102 is fitted to a real space image 100 in the left window 3 direction, and a display image 10 is displayed.
4 is synthesized.

In this case, the image displayed on the display image 104 looks different between the pilot 46 and the instructor 48. This is because the virtual field-of-view image 102 uses the spatial sensor 12 for the player to
This is because the direction of view of each instructor 48 is detected and calculated. Therefore, it is possible to create an image that can be seen by the instructor 48 but cannot be seen by the pilot 46, that is, an image effect that differs depending on the direction of view, and a virtual world with a more realistic feeling can be realized. In this regard, such a video effect cannot be produced by a conventional method, that is, a method of simply providing a CRT display on a window, or a method of displaying a CRT image on a window by a half mirror.

Further, in this embodiment, the pilot 46
Can simulate the operation of an airplane while viewing the operation panel 47, the control stick 42, the face of the instructor 48, and the like in the cockpit 45 with the video camera 10. Therefore, the operation panel 47 of an airplane, which is generally considered to be complicated and difficult to operate, can be operated by the video camera 10 while actually seeing it with its own eyes, so that operation errors can be dramatically reduced.

That is, for example, according to a system in which all of the operation panel 47, the control stick 42, and the like are formed by image synthesis, the operation feeling for the pilot 46 is poorly realistic. Moreover, according to this method, it is necessary to attach a data glove to the pilot 46, and to detect what operation the pilot 46 has performed on the operation panel 47 and the control stick 42 based on an operation signal from the data glove. As a result, the scale of the system becomes enormous. In this way, the system has a drawback that the operation feeling of the pilot 46 is not good even though the scale is enormous, and there are many operation mistakes of the pilot 46.

In this respect, in this embodiment, the cockpit 4
5 has the same structure as the cockpit of a real airplane, so that such inconvenience does not occur. Further, in the present embodiment, since the instructor 48 next to the instructor can be seen,
The instructor 48's instructions and the like can be accurately grasped, and the angry face of the instructor 48 when an operation error is made can be seen, so that a more realistic flight simulator can be realized.

2. Second Embodiment FIG. 5 shows a block diagram of the second embodiment. FIG. 6 shows an example of a virtual experience device in which this is applied to a role playing game.

As shown in FIG. 5, this second embodiment is
In the first embodiment, the operation unit 38 is changed to a weapon 58 for battle with an enemy character.
Attach a space sensor 64 for battle weapons to the virtual 3
The game result calculation section 68 is built in the dimension space calculation section 72.

In the role playing game to which the second embodiment is applied, a setting as shown in FIG. 6A, that is, a plurality of players 50, 52 having a fighting weapon 58, for example, a sword 60, is provided. Are set to defeat the enemy character 66 in a team. Each player wears the head mounted body 9 having the same configuration as that of the first embodiment, and plays a game while watching the display image 104 projected on the image display unit 20. .

The inside of the attraction room 110 is entirely painted in blue, so that the above-described blue mat composition can be performed. In addition, the ceiling of the attraction room 110 has 3
A space sensor 12 for detecting dimensional information and a space sensor 64 for detecting three-dimensional information of a sword 60 as a battle weapon are attached. This attraction room 110
The space is almost the same area as the game space where the player actually plays the game, and the enemy character 66 is defeated while walking the players 50, 52, and the like in the attraction room 110 by themselves.

In FIG. 6B, a real space image 100
Is an image from the image camera 10 attached to the player 50. As shown in FIG.
At 0, the own hand 54, the second player 52 in front of the user, and the like are displayed with the blue attraction room 110 as a background. A virtual view image 10 created by the same method as in the first embodiment is added to this image.
2 are synthesized to form a display image 104. In this case, in addition to the enemy character 66, a maze, a door, a floor, and the like that form the game space are also displayed in the virtual view image 102.

With this configuration, the player 50 can defeat the enemy character 66 while assembling with the second player 52 while watching the display image 104. In this case, communication with the second player 52 is performed through the speaker 22, for example. Also, the enemy character 66
Is determined by the space sensor 64 for the fighting weapon attached to the sword 60.

That is, the three-dimensional information, for example, the position and direction of the sword 60 are detected by the battle weapon space sensors 64, 64. Then, based on the three-dimensional information of the sword 60, the virtual three-dimensional space calculation unit 72 shown in FIG. 5 accurately calculates whether or not the sword 60 has successfully attacked the enemy character 66. Then, when the game is successful, the enemy character 66 in the game space is erased or the like, and this is displayed on the image display device 20 such as the players 50 and 52 in real time. As a result, each player can check whether or not the enemy character 66 has fallen, and when it has fallen, the player advances to the next maze to defeat a new enemy character 66. As described above, in the present embodiment, it is possible to change the game progress in the game space formed by the skill of the players 50, 52, and the like, and to provide a virtual experience device that does not get tired even when playing repeatedly. .

In the present embodiment, the game score of each player, that is, the number of defeated enemy characters 66 is determined by using the game score calculation unit 68 based on the three-dimensional information from the battle space sensor 64 described above. The result is also output to the image display device 20 of each player. As a result, the game results of each player, the game results of each team, and the like can be confirmed after the game ends or during the game in real time. As a result, since each player can compete in real time with each player's skill, the fun as a game can be dramatically increased.

Further, in the present embodiment, the real space video taken by the video camera 10 is used as the video information of the second player 52 seen from the player 50, but the present embodiment is not limited to this. . That is, for example, the second
The player 52 of the virtual
The display may be performed using a character synthesized by the dimension space calculation unit 72. By doing so, it is possible to obtain a video effect such as evolving the second player as the game progresses and defeats the enemy character 66, and the fun of the game can be further increased.

As described above, the following method can be used as a method of changing the real space image of another player into a character image obtained by image synthesis. For example, a battle suit of a different color for each player, that is, attraction room 1
A battle suit of a different color from the inside of 10 is mounted, and image synthesis is performed by the same method as the blue mat method described in the first embodiment. That is, the second player is fitted with a red-colored battle suit, and only the red-colored portion is extracted from the video camera 10 by a method similar to the above-described method, and is set as an empty dot. next,
By superimposing a character image that is to be displayed to be superimposed on another player, an image in which the character image of the second player 52 is superimposed on the real space video 100 can be obtained. After that, only the blue color portion is extracted from the overwritten image, this is set as an empty dot, and the virtual view image 102 on which the enemy character 66, the maze, and the like are displayed is overwritten. As a result,
It is possible for the second player 52 to obtain the display image 104 changed to the character image. As a method of changing the image of the second player 52 as described above, for example, the above-described texture mapping method may be used.

In this embodiment, the sword 60 is used as the battle weapon 58. However, this embodiment is not limited to this, and any battle weapon such as a light gun can be used. When a light gun or the like is used, for example, information as to whether or not the player has shot the light gun is required. In this case, as shown in FIG. Is input and used for the calculation of the game space.

FIG. 7A shows a virtual experience apparatus when this embodiment is applied to a ride type attraction facility.

In this attraction, the player 50,
52 is mounted on a carrier 116, and the carrier 116 moves on a rail 118 laid in the attraction room 110. Various dioramas 114 are provided in the attraction room 110, and a background mat 112 painted in blue is provided inside the attraction room 110. Then, the players 50 and 52
Shoots down the enemy character 66 with the light gun 62 and competes for the game. FIG. 7B shows the state of image synthesis in this case. The method of this image synthesis is the same as in the first embodiment.

When the present embodiment is used in an attraction facility, the following advantages are mainly obtained.

First, in conventional attraction facilities, once the attraction facility is constructed, it is very difficult to change the contents of the attraction because the facility is large and expensive. For this reason,
It is necessary to take measures to prevent the player from getting tired of playing the same game several times. In this regard, in the present embodiment, it is possible to easily cope with this by changing the virtual image of the game space projected on the background mat 112 while using the diorama 114 in common. Further, as described above, by providing the game result calculation unit 68 in the virtual three-dimensional space calculation unit 72, this can be dealt with more effectively.

That is, the number of enemy characters 66 shot down by each player is calculated in real time by the game score calculation section 68, and the image of each player is calculated according to the number of enemy characters 66 shot down, that is, according to the skill of the player. The virtual image 102 displayed on the display device 20 is changed.
For example, for a player with high skill, the number of enemy characters 66 displayed in the virtual view image 102 is increased,
Also, the enemy character 66 to be displayed is made strong.
Further, a configuration may be adopted in which several kinds of courses of the game space to be displayed on the virtual view image 102 are provided in advance, and one of the courses is selected according to the game result. As a result, a player riding this attraction facility can experience a virtual experience of a different game no matter how many times he or she rides, and can provide an attraction facility that does not get tired even after repeated playing.

Another advantage of the present embodiment is that, despite the fact that an attraction facility that does not get tired as described above can be realized, a real space image can be used for the diorama 114 and the like in the attraction room. It is to be able to realize attractions full of reality.

For example, when the transporter 116 is of a so-called roller coaster type, an attraction full of thrilling and speed can be realized when the diorama 114 seen from the player is a real space image. That is, if all the visual images viewed by the user are created by image synthesis, the inherent speed, tension and power of the roller coaster cannot be sufficiently reproduced. In this regard, in the present embodiment, the face of another player adjacent to the roller coaster can also be viewed as a real space image, and the function inherent to the roller coaster can be exhibited more.

Also, when the diorama 114 touches the player's hand, it is possible to provide a more realistic attraction when the video is a real space video. This point is shown, for example, in an attraction to which the present embodiment is applied as shown in FIG.

The attraction shown in FIG.
This is an attraction that allows you to experience flying and fighting in a simulated manner.

In this attraction, as shown in FIG. 8A, a plurality of players get into the spacecraft cabin 120 of the spacecraft 134. The spacecraft cabin 120 is made very similar to a real spacecraft, and is provided with, for example, a cockpit and a combat seat. In this case, in particular, the control stick 132 and the operation panel 130 that the player directly touches are
The artillery 144 is elaborately made to look very real.

The players aboard the spacecraft cabin 120, according to their roles, the pilot, the co-pilot,
It is placed in the cockpit, combat seat, etc. as a shooter. And
The pilot 146 and the co-pilot 147 arranged in the cockpit,
The spacecraft 134 is controlled by the control column 132 and the control panel 130 while avoiding the meteorite 140 and the like projected on the cockpit window 122 by the blue mat method described above. In this case, in the present embodiment, as described above, the space sensor 12 is attached to each player, the direction of the view is calculated for each player, and the view image obtained by the calculation is displayed on the image display device 2.
0 is displayed. As a result, the appearance of the meteorite 140 approaching the spacecraft 134 is determined by the pilot 146 and the first officer.
47 and the shooter 148 can be set to look different from each other, so that an attraction full of a sense of reality and a sense of reality can be provided. In addition, Pilot 146, Copilot 1
47 can operate the spacecraft while operating the control stick 132 and the operation panel 130 which are made very similar to the real thing, so that the player can play as if he were operating a real spacecraft.

Shooters 148, 149 arranged in the battle seat
Is operated by a battle weapon 144 which is a battle weapon.
4. The spacecraft 142 projected on the right window 125 by the blue mat method is shot down. The game score in this case is calculated by the game score calculation unit 68, and is displayed in real time during the game or as a game result of all the crew members after the game is over.

As shown in FIG. 8B, the attitude and acceleration G of the spacecraft 134 on which the player rides are controlled by the attitude control unit 138 using hydraulic pressure or the like in accordance with the game progress and the player's operation signals. Thus, the configuration is such that the sense of reality is further increased.

As described above, in the present embodiment, since each player can play a different role in a very near-real spacecraft integrally, it is possible to provide an attraction full of realism without getting tired.

In the above-described embodiment, the image display unit 20 and the image synthesizing unit (the virtual visual field image calculating device 70, the display image synthesizing device 80, and the like in FIG. 5) are arranged at different places. Although the configuration is such that these are connected, the present embodiment is not limited to this, and a configuration in which all the components including the image combining unit are incorporated in the image display unit 20 may be used.
As shown in FIG. 9, a configuration may be employed in which the two-way radios 160 are used to connect them.

This walking type attraction is shown in FIG.
(A) As shown in (b), an attraction in which the player 50 equipped with the head-mounted body 9, the two-way radio 160, and the light gun 62 defeats the enemy character 66 while traveling in the maze 150 by himself. It is. Thus maze 15
In an attraction in which the player goes within 0, it is desirable that the player's movement has more freedom.
Therefore, in the present embodiment, the connection with the image synthesizing unit is performed by the two-way wireless device 160. In this case, as the two-way wireless method, for example, infrared rays or the like may be used.

Here, the maze 150 has a window 152, a picture 1
54, an elevator 156, a door 158, and the like. Then, a blue mat is laid on the entire surface of the window 152, and a blue mat is laid on only the portion where the enemy character 66 appears in the picture 154.

The elevator 156 and the door 158 are
The inside is painted blue. Thus, when the player 50 presses the elevator button to open the elevator or turns the door knob to open the door, the enemy character 66 can jump out of the door. This makes it ideal for attractions in games such as ghost houses. In this case as well, in the present embodiment, the elevator buttons, door knobs, and the like are genuine, and the player 50 can operate while watching the actual image, so that a more realistic virtual world can be enjoyed. Becomes

3. Third Embodiment FIG. 10 is a block diagram showing the third embodiment. FIG. 11 shows an example of a virtual experience apparatus in which the present invention is applied to golf. Although the case where the present invention is applied to golf is shown here, the present embodiment is not limited to this and can be applied to various sports.

As shown in FIG. 10, the third embodiment is different from the first embodiment in that the operation unit 38 is changed to ball video cameras 84 and 86 and moving object detection devices 88 and 90. This is an embodiment suitable mainly for ball games using a ball.

FIG. 11A shows an example of indoor golf to which the second embodiment is applied. In this embodiment, when the player 50 is in the play room 162,
The present invention relates to a virtual experience device that allows the user to experience a virtual world with almost the same feeling as real golf.

In FIG. 11A, the play room 16
2 are all painted in blue and the player 50
Is equipped with a head-mounted body 9 having the same configuration as that of the first embodiment. Then, the player 50 makes a shot of the real golf ball 166 using the real golf club 164 as if he were at a real golf course.

In this case, in the play room 162, the image of the golf course created by the virtual three-dimensional space calculation unit 72 is displayed via the image display device 20, and the player 50 is as if the player is at a real golf course. You can have a feeling like that. When the player 50 faces forward, a virtual view image 102 showing a green 162 of the golf course is projected in the direction of the line of sight, as shown in FIG. 11B.

Then, the player 50 moves the golf ball 16
When the shot of No. 6 is performed, the shot golf ball 1
66 is photographed by the ball video cameras 84 and 86. Then, based on the photographed image, the moving object detecting device 8
The coordinates of the center of gravity of the golf ball 166 are calculated by using the data 8 and 90.
Is calculated by estimating the trajectory of. Next, the virtual three-dimensional space calculation unit 72 draws the trajectory of the golf ball on the golf course set in the virtual world based on the estimated trajectory data of the golf ball 166,
And image display on the image display device 20. This allows the player 50 to see the golf ball 166 shot by the player 50 flying toward the green 162 on the display image 104.

Next, a method of detecting a moving object of the golf ball 166 will be described.

The ball video cameras 86 and 89 are for continuously photographing the golf ball 166, and are set at different positions and different photographing angles in the play room 162 as shown in FIG. The photographing data is stored in the data extracting unit 9 in the moving body coordinate detecting units 88 and 90.
2 and 94 are input as frame advance frame data.
In the data extracting units 92 and 94, only the photographing data of the ball is extracted by the background processing. That is, the golf ball 16
The difference between the frame data not shown and the frame data shown in No. 6 is obtained, and this is overwritten in an image memory called a frame buffer. As a result, image data in which only the image of the golf ball 166 is projected can be obtained.

Next, two-dimensional coordinates of the position of the golf ball 166, for example, the position of the center of gravity, are obtained from the obtained image data in the coordinate detectors 96 and 98, and output to the virtual three-dimensional space calculator 72. .

The virtual three-dimensional space calculation unit 72 obtains three-dimensional coordinates in the virtual three-dimensional space from the two two-dimensional coordinates of the golf ball 166 detected by the moving object detection devices 88 and 90. That is, the virtual three-dimensional space calculation unit 72 previously stores the set positions and set angles of the ball video cameras 84 and 86 in the virtual three-dimensional space, and stores the stored data and the two detected two-dimensional data. From the coordinates, three-dimensional coordinates in a virtual three-dimensional space can be obtained. Then, from the obtained three-dimensional coordinates, the trajectory of the golf ball 166 in the virtual three-dimensional space is calculated using, for example, spline interpolation or the like, and is output to the view image calculation unit 76 together with the background data of the golf course. Note that the method of estimating the trajectory of the golf ball 166 is not limited to the above, and various methods can be used. For example,
This may be estimated from the striking direction and the initial speed at the time of impact, or may be estimated from the sound at the moment of striking.

For example, not only the image viewed from the player 50 but also the image viewed from the green 162 can be displayed on the image display device 20 by, for example, a switching signal of the player 50. That is, to perform this, the viewpoint position may be set on the green 162 when the viewpoint image calculation unit 76 performs the viewpoint conversion. By doing so, the player 50 actually makes the golf ball 166
2 and the golf ball 166 flying on the green 162 can be seen at the same time, and it is possible to enjoy a feeling of play that cannot be obtained in actual golf play.

According to the present embodiment having the above configuration, the player 50 can enjoy golf in the play room 162 as if he or she were actually at a golf course.
In particular, in a game using such a ball, for example, golf, the player 50 needs to accurately recognize the moment when the golf ball 166 is shot by the club 164. That is, if the club 164 and the golf ball 166 at the moment of this shot are created by, for example, image synthesis, the sense of play is significantly deteriorated, the sense of reality is reduced, and moreover, accurate sports play can be reproduced. Can not.

On the other hand, in the present embodiment, the golf club 164, the golf ball 166, the player's own swing, and the like can be seen from the real space image obtained through the image camera 10, so that such a problem occurs. Therefore, a more accurate and realistic golf play can be enjoyed. Therefore, for example, the virtual experience device for ball game practice is optimal.

The present invention is not limited to the above embodiment, and various modifications can be made within the scope of the present invention.

For example, in the present embodiment, the blue mat method is used as a method of synthesizing the virtual visual field image and the real space image, but the image synthesis in the present invention is not limited to this. For example, various methods can be used, such as a method using a color other than blue, a method using multiple colors and combining them to form an image, and a method using texture mapping.

The virtual experience apparatus to which the present invention is applied is not limited to the one shown in the present embodiment, but can be applied to various experience apparatuses. For example, it is applied to various experience devices such as a helicopter flight simulator, a virtual human body surgery experience device, a virtual studio, a virtual zoo, a virtual design system, a virtual telephone, a virtual communication conference, a virtual baseball experience device, a virtual ski, a virtual soccer, etc. It is possible to

[0119]

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a first embodiment.

FIG. 2 is a schematic explanatory diagram showing a case where the first embodiment is applied to a driving game.

FIG. 3 is a schematic explanatory diagram showing a case where the first embodiment is applied to a flight simulator.

FIG. 4 is a schematic explanatory view showing a shape of a head mounted body.

FIG. 5 is a block diagram illustrating a configuration of a second embodiment.

FIG. 6 is a schematic explanatory diagram showing a case where the second embodiment is applied to a role playing game.

FIG. 7 is a schematic explanatory diagram showing a case where the second embodiment is applied to a ride-type attraction.

FIG. 8 is a schematic explanatory diagram showing a case where the second embodiment is applied to an attraction of a spaceship experience.

FIG. 9 is a schematic explanatory diagram showing a case where the second embodiment is applied to a walking attraction.

FIG. 10 is a block diagram illustrating a configuration of a third embodiment.

FIG. 11 is a schematic explanatory view showing a case where the third embodiment is applied to a golf experience device.

FIG. 12 is a schematic explanatory diagram for describing coordinate conversion in the present embodiment.

FIG. 13 is a schematic block diagram illustrating an example of a configuration of a display image composition device.

FIG. 14 is a schematic explanatory diagram for describing a method of displaying an image on a side mirror.

FIG. 15 is a schematic explanatory diagram illustrating a method of attaching a real space image to a face of an opponent player by a texture mapping method.

FIG. 16 is a schematic block diagram illustrating an example of a configuration for realizing a texture mapping method.

[Explanation of symbols]

 Reference Signs List 9 head-mounted body 10 video camera 12 space sensor 20 image display device 38 operation unit 50 player 58 battle weapon 64 battle weapon sensor 68 game performance calculation unit 70 virtual view image calculation unit 72 virtual 3D space calculation unit 76 view image Calculation unit 80 Display image synthesizing device 84, 86 Ball image camera 88, 90 Moving object detecting device 100 Real space image 102 Virtual field-of-view image 104 Display image

──────────────────────────────────────────────────の Continued on the front page (51) Int.Cl. ⁶ Identification code FI G09B 9/30 G06F 15/62 360 15/72 350

Claims (9)

[Claims] A texture storage unit that stores a texture to be mapped to an object in a virtual three-dimensional space; a unit that writes an image captured by an imaging unit as the texture in the texture storage unit; and a storage unit that stores the texture in the texture storage unit. Image synthesizing means for mapping the texture to be rendered to an object in a virtual three-dimensional space, and synthesizing an image viewed from a given viewpoint in the virtual three-dimensional space. 2. The image synthesizing apparatus according to claim 1, wherein an image captured in real time by the image capturing means is mapped to an object in a virtual three-dimensional space in real time. 3. The image synthesizing apparatus according to claim 1, wherein an image picked up by the image pickup means before the start of game play is mapped to an object in a virtual three-dimensional space. 4. The image synthesizing apparatus according to claim 1, wherein an image obtained by imaging the player by the imaging means is mapped to an object in a virtual three-dimensional space. 5. The image synthesizing apparatus according to claim 4, wherein a plurality of images obtained by imaging the player from a plurality of directions by a plurality of imaging means are mapped to an object in a virtual three-dimensional space. 6. The player head according to claim 1, wherein an image of the player's head obtained by imaging the head of the player by the imaging means is converted into an object representing the player's head in a virtual three-dimensional space. An image synthesizing device, which performs mapping. 7. The method according to claim 1, wherein an image of the first player imaged by the first imaging means is mapped to a first object in a virtual three-dimensional space, and
Mapping the image of the second player imaged by the imaging means on a second object in the virtual three-dimensional space, displaying the second object on a first screen viewed by the first player, An image synthesizing apparatus, wherein the first object is displayed on a second screen viewed by the player. 8. The image synthesizing apparatus according to claim 1, wherein the object in the virtual three-dimensional space is constituted by polygons. 9. A texture storage unit for storing a texture to be mapped to an object in a virtual three-dimensional space, an image captured by an imaging unit is written as the texture, and the texture stored in the texture storage unit is stored in a virtual three-dimensional space. An image synthesis method, which maps an object in a space and synthesizes an image viewed from a given viewpoint in a virtual three-dimensional space.