JP2016192029A - Image generation system and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image generation system and a program that can generate a more appropriate image than an existing image by reducing influences caused by failed tracking of an eye point.SOLUTION: The image generation system includes: an object space setting unit performing processing of setting an object space; an eye point information operating unit setting tracking eye point information obtained by tracking the eye point of a player as eye point information of the player; and an image generating unit generating an image which can be seen from the eye point of the player in the object space, based on the eye point information, the eye point information operating unit further performing processing of estimating the eye point information of the player if the eye point being tracked is in a lost state.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an image generation system, a program, and the like.

  2. Description of the Related Art Conventionally, a system that generates a projection image projected on a curved screen (dome-shaped screen) by a projection device (projector) is known. For example, as a conventional technique for projecting a projection image with little distortion on a curved screen, there is a technique disclosed in Patent Document 1.

  In addition, a stereoscopic image generation system is also in the spotlight as a system for generating an image that enhances the sense of reality by another approach. As a conventional technique of a stereoscopic image generation system, for example, there is a technique disclosed in Patent Document 2. Furthermore, an image generation system that generates a game image or the like and displays it on a head-mounted display device (head mounted display, HMD) is also known.

JP 2003-85586 A JP 2004-126902 A

  In such an image generation system, it is important to accurately specify the viewpoint of the player. In particular, in a system that generates projection images and stereoscopic images, an unnatural image for the player is generated if the viewpoint of the actual player is different from the viewpoint of the player in the object space used for image generation. End up.

  In this case, for example, a system that tracks the player's viewpoint using some device such as an imaging unit and generates an image at the tracked viewpoint can be considered. However, in such a system, it is not always possible to accurately track the viewpoint, and if the tracking fails, an image that is very unnatural and uncomfortable for the player may be generated.

  According to some aspects of the present invention, it is possible to provide an image generation system, a program, and the like that can reduce the influence of a viewpoint tracking failure and generate a more appropriate image than before.

  One aspect of the present invention includes an object space setting unit that performs an object space setting process, a viewpoint information calculation unit that sets tracking viewpoint information obtained by viewpoint tracking of the player as viewpoint information of the player, and the viewpoint information And an image generation unit that generates an image that can be seen from the viewpoint of the player in the object space, and the viewpoint information calculation unit is configured to estimate the viewpoint information of the player when the tracking viewpoint is in a lost state. It relates to the image generation system that performs. The present invention also relates to a program that causes a computer to function as each of the above-described units, or a computer-readable information storage medium that stores the program.

  According to one aspect of the present invention, tracking viewpoint information obtained by viewpoint tracking is set as player viewpoint information, and an image that is visible from the player's viewpoint in the object space is generated based on the set viewpoint information. . When the tracking viewpoint is lost, the player viewpoint information is estimated, and an image that is visible from the player viewpoint in the object space is generated based on the viewpoint information obtained by the estimation process. It becomes like this. This makes it possible to provide an image generation system or the like that can generate a more appropriate image even when viewpoint tracking fails.

  In the aspect of the invention, the viewpoint information calculation unit may specify the tracking viewpoint information obtained from the captured image obtained by photographing the player as the viewpoint information of the player.

  In this way, the tracking viewpoint information can be obtained using the captured image and an image that can be seen from the viewpoint of the player can be generated in the object space simply by providing an imaging unit that captures the player.

  In the aspect of the invention, the viewpoint information calculation unit may perform the viewpoint information estimation process when the tracking viewpoint is in a lost state based on the viewpoint history information of the player.

  In this way, for example, viewpoint information can be estimated using viewpoint history information obtained in a situation similar to the current situation. Accordingly, when the tracking viewpoint is in the lost state, more appropriate viewpoint information can be set as the player's viewpoint information, and an image that can be seen from the player's viewpoint in the object space can be generated.

  In the aspect of the invention, the viewpoint information calculation unit may be configured to detect the tracking viewpoint in a lost state based on at least one of the viewpoint history information, movement information of a moving body operated by the player, and operation information of the player. The viewpoint information may be estimated.

  In this way, not only the viewpoint history information but also the movement information of the moving body and the operation information of the player can be added to estimate the player's viewpoint information when the tracking viewpoint is in the lost state. Can be set as viewpoint information.

  In the aspect of the invention, the viewpoint information calculation unit may perform the tracking based on the viewpoint history information when the moving body operated by the player moves on a course or a map in the past in the same area. The viewpoint information may be estimated when the viewpoint is in a lost state.

  In this way, if you use viewpoint history information when you are located in the same area in the past in a course or map, you can set the viewpoint information in a similar situation as the viewpoint information of the player when the tracking viewpoint is lost, Appropriate images can be generated.

  In one aspect of the present invention, the course is a round course, and the viewpoint information calculation unit is based on the viewpoint history information when the moving body operated by a player has passed through the same area in the past in the round course. The viewpoint information may be estimated when the tracking viewpoint is in a lost state.

  In this way, if viewpoint history information when passing through the same area in the past in the circuit course is used, viewpoint information in a similar situation can be estimated as the viewpoint information of the player in the lost state of the tracking viewpoint. . In addition, since the moving body circulates the circuit course, viewpoint history information can be acquired for each circuit, for example, and there is an advantage that viewpoint information to be a candidate target can be increased.

  Further, according to an aspect of the present invention, the viewpoint information calculation unit may include a movable mechanism, and the viewpoint information calculation unit may perform the viewpoint information estimation process when the tracking viewpoint is in a lost state based on operation information of the movable mechanism.

  In this way, when the player's viewpoint changes due to the movement of the movable mechanism, the viewpoint information of the player is estimated based on the movement information of the movable mechanism, and is used as the player's viewpoint information when the tracking viewpoint is in the lost state. It becomes possible to set.

  In the aspect of the invention, the viewpoint information calculation unit obtains the motion information of the movable mechanism based on player operation information or game situation information, and based on the motion information, the tracking viewpoint lost state You may perform the estimation process of the said viewpoint information at the time.

  In this way, the operation information of the movable mechanism is obtained based on the player operation information and the game situation information, and the viewpoint information estimated based on the operation information is used as the player viewpoint information when the tracking viewpoint is in the lost state. It becomes possible to set.

  In the aspect of the invention, the viewpoint information calculation unit may estimate the viewpoint information when the tracking viewpoint is in a lost state based on player information including at least one of the player's physique information, gender information, and age information. Processing may be performed.

  In this way, when the tracking viewpoint becomes lost, viewpoint information can be estimated taking into account the player's physique, gender, age, etc., so a more appropriate viewpoint that reflects the player's information can be used. It becomes possible to generate images.

  In the aspect of the invention, the viewpoint information calculation unit may perform a process of selecting viewpoint information based on the player information from among a plurality of candidate viewpoint information. You may perform as an estimation process.

  In this way, it becomes possible to estimate the viewpoint information in the lost state of the racking viewpoint by selecting the viewpoint information according to the player's physique, gender, age, etc. from the plurality of candidate viewpoint information.

  In the aspect of the invention, the viewpoint information calculation unit may perform the estimation process of the viewpoint information when the tracking viewpoint is lost based on the result information of the physical simulation process of the player's motion.

  This physical simulation process includes a simulation process related to human behavior that reacts to changes in video and movable mechanisms.

  In this way, it is possible to identify the player's movement and to estimate the player's viewpoint information when the tracking viewpoint is in a lost state by the physical simulation process, so that an image with a viewpoint similar to the viewpoint of the player in the real world can be obtained. Can be generated.

  In the aspect of the invention, the image generation unit may generate a projection image based on the shape information of the projection screen constituted by one curved surface or a plurality of surfaces and the viewpoint information of the player. Good.

  In this way, by using the shape information and viewpoint information of the projection screen, an appropriate projection image reflecting the shape of the projection screen can be generated. Even when the tracking viewpoint is in a lost state, the projection image is generated using the viewpoint information obtained by the estimation process, so that it is possible to generate a projection image with less discomfort and unnaturalness.

  Also, in one aspect of the present invention, the image generation unit forms a straight line connecting the intersection position between the light beam emitted from the pixels on the drawing buffer through the optical system of the projection apparatus and the projection screen and the viewpoint position of the player. The color of the pixel on the drawing buffer may be determined as the line of sight of the virtual camera.

  In this way, it becomes possible to generate a projection image for projection by performing distortion correction processing in units of pixels.

  In the aspect of the invention, the image generation unit may correspond to the object based on an intersection position between the projection screen and a straight line connecting the vertex position of the object in the object space and the viewpoint position of the player. A vertex position on the drawing buffer for the drawing object may be obtained, and the drawing object may be drawn in the drawing buffer based on the vertex position.

  In this way, it becomes possible to generate a projection image for projection by performing distortion correction processing in units of vertices.

  In the aspect of the invention, the image generation unit may generate a first viewpoint image and a second viewpoint image for stereoscopic viewing based on the viewpoint information of the player.

  In this way, stereoscopic viewing can be realized by displaying the first viewpoint image and the second viewpoint image for stereoscopic viewing on the player. Even when the tracking viewpoint is in the lost state, the first viewpoint image and the second viewpoint image are generated using the viewpoint information obtained by the estimation process, so that stereoscopic vision with less sense of incongruity and unnaturalness is generated. Can be realized.

An example of the game system to which the image generation system of this embodiment is applied. The vertical sectional view of the game system to which the image generation system of this embodiment is applied. The structural example of a movable mechanism. 1 is a configuration example of an image generation system of the present embodiment. FIGS. 5A to 5C are explanatory diagrams of representative viewpoints used for generating a projection image. 6A and 6B are explanatory diagrams of viewpoint tracking. 7A and 7B are also explanatory diagrams of viewpoint tracking. FIG. 8A to FIG. 8C are explanatory diagrams of the tracking viewpoint lost state. The flowchart which shows the process example of the method of this embodiment. FIGS. 10A to 10C are explanatory diagrams of a method using viewpoint history information. FIG. 11A to FIG. 11C are explanatory diagrams of a method using viewpoint history information in a game in which a moving body is moved on a circuit course. Explanatory drawing of the method of selecting viewpoint information using the movement information etc. of a mobile body from several viewpoint historical information. The flowchart which shows the process example of the method of estimating a player's viewpoint information based on viewpoint history information. The flowchart which shows the process example of the method of estimating a player's viewpoint information based on the operation information of a movable mechanism. FIGS. 15A to 15C are explanatory diagrams of a method for estimating the viewpoint information of the player based on the operation information of the movable mechanism. The flowchart which shows the process example of the method of estimating the viewpoint information of a player based on player information. The flowchart which shows the process example of the method of estimating a player's viewpoint information by a physical simulation process. FIG. 18A to FIG. 18C are explanatory diagrams of a method for estimating the player's viewpoint information by physical simulation processing. Explanatory drawing of the distortion correction method per pixel of a drawing buffer. Explanatory drawing of the method of using a surrogate plane for distortion correction per pixel. The figure which shows the relationship between the drawing buffer by the method of using a substitute plane for distortion correction per pixel, a UV map, and a render texture. Explanatory drawing of the distortion correction method in the vertex unit of an object. 23A to 23C are explanatory diagrams of a method for generating a stereoscopic image. Explanatory drawing of the method of producing | generating the image displayed on HMD.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Configuration FIG. 1 shows a configuration example of a game system (game device) to which the image generation system of this embodiment is applied. FIG. 2 is a vertical sectional view of the game system.

  The game system shown in FIGS. 1 and 2 is a business game system installed in an amusement facility for a player to play a racing game. The game system includes a seat 10 on which a player sits, a control board 16, a curved screen 20 on which a projection image that is a game image is projected, a projection device 30 that projects a projection image (video) on the screen 20, An imaging unit 40 for viewpoint tracking, a movable mechanism 60, and a speaker (not shown) that outputs game sound are provided. Further, a handle 50, a brake pedal 52, an accelerator pedal 54, a shift lever 56, a card reader 58, and a coin insertion slot 59 are provided for the player to input operation information.

  The movable mechanism 60 changes the direction (posture) and position of the sheet 10 that is a movable object. For example, the movable mechanism 60 performs operations for causing the player to experience acceleration / deceleration, cornering load, and road surface unevenness. The movable mechanism 60 operates according to an operation signal from the control board 16.

  When the seat 10 is in the basic position / basic posture, the position / posture is set by the movable mechanism 60 so that the assumed normal viewing direction of the seated player faces the center of the screen 20. In the present embodiment, the front direction of the player who is seated is the assumed normal viewing direction. The screen 20 is formed in a convex shape with respect to the front direction (assumed normal viewing direction) of the player seated on the seat 10 at the basic position / posture.

  The control board 16 is mounted with various processors such as a CPU, GPU or DSP, and various memories such as an ASIC, VRAM, RAM or ROM. The control board 16 performs various processes for realizing a racing game based on programs and data stored in the memory, operation information of a player who operates the handle 50, and the like.

  The projection device 30 (projector) is supported by the housing frame 14 so that the projection center direction thereof faces the vicinity of the center of the screen 20 at a position above the seat 10 and not interfering with the player seated on the seat 10. is set up. In other words, the projection center direction is set so as to face the intersection point between the assumed normal viewing direction of the player and the screen 20. Further, the projection device 30 is provided with a wide-angle lens (for example, a fish-eye lens, that is, a super-wide-angle lens with an angle of view exceeding 180 degrees) as a projection lens, and a projection image is projected onto the projection area of the screen 20 through the wide-angle lens. Projected to the whole.

  The player sits on the seat 10 and operates the steering wheel 50, the accelerator pedal 52, the brake pedal 54, and the shift lever 56 while listening to the game sound from the speaker while watching the game screen displayed on the screen 20. Enjoy the racing game.

  In the race game of the present embodiment, a game space is configured by arranging background objects such as a race course in an object space (virtual three-dimensional space). In the object space, various objects such as a player car and other player cars are arranged, and a virtual camera is arranged at the viewpoint position of the player. Then, the image of the object space viewed from the virtual camera is projected (displayed) on the screen 20 by the projector 30 as a game image.

  FIG. 3 shows a configuration example of the movable mechanism 60 (movable housing). The movable mechanism 60 includes a movable object and an actuator that moves the movable object. In FIG. 3, the movable object is the seat 10, and the actuators are the electric cylinders 73 and 74. The electric cylinders 73 and 74 linearly move the rod portion as indicated by A1 and A2 based on an operation signal (operation information in a broad sense) that is an electric signal from the control board 16 (processing portion) in FIG. The electric cylinders 73 and 74 have a stepping motor and a ball screw, and the linear motion of the rod portion is realized by rotating the ball screw with the stepping motor. Then, the rod portions of the electric cylinders 73 and 74 are linearly moved in this manner, thereby realizing an operation of changing the direction (posture) and position of the seat 10 as shown in A3 and A4 of FIG.

  Specifically, a base 62 is provided on the bottom 12 of the chassis of the game system, and hinges 63 and 64 are provided on the base 62. One ends of the electric cylinders 73 and 74 are attached to the base 62 by hinge portions 63 and 64. Specifically, the hinge portions 63 and 64 support one ends of the electric cylinders 73 and 74 so as to be rotatable around the X axis that is the horizontal direction. In addition, you may attach the hinge parts 63 and 64 to the base 62 so that rotation around the Y-axis which is a perpendicular direction is possible. A mounting plate 80 is provided on the back side of the back portion of the seat 10, and hinge portions 83 and 84 are provided on the mounting plate 80. The other ends of the electric cylinders 73 and 74 are attached to the attachment plate 80 by hinge portions 83 and 84. Specifically, the hinge portions 83 and 84 support the other ends of the electric cylinders 73 and 74 so as to be rotatable around the X axis.

  Hinge portions 65 and 66 are provided on the base 62, and one ends of support portions 75 and 76 are attached to the hinge portions 65 and 66. The other ends of the support portions 75 and 76 are attached to the seat portion (back surface) of the seat 10. For example, the support portions 75 and 76 support the seat portion of the seat 10 so as to be rotatable around the Y axis that is the vertical direction.

  In the movable mechanism 60 of FIG. 3, when the rod portion of the electric cylinder 73 linearly moves in the extension direction and becomes long, and when the rod portion of the electric cylinder 74 linearly moves in the retraction direction and becomes short, the seat 10 (seat portion) becomes Y For example, it rotates to rotate clockwise around the axis. Further, when the rod portion of the electric cylinder 73 is linearly moved in the retracting direction to be shortened and the rod portion of the electric cylinder 74 is linearly moved to be in the extension direction to be long, for example, the seat 10 is operated to rotate counterclockwise. By operating the movable mechanism 60 in this way, the player can experience cornering of a moving body such as a car.

  In order to allow the player to experience road surface irregularities and the like, the electric cylinders 73 and 74 may be controlled so that the rod portion linearly moves with a minute stroke distance.

  3 is an example of the configuration of the movable mechanism 60, and the movable mechanism 60 of the present embodiment is not limited to the configuration of FIG. 3, and various modifications can be made. For example, the movable mechanism 60 is a mechanism (machine) that performs an operation that changes at least one of the viewpoint position and the line-of-sight direction of the player. The movable mechanism 60 operates based on operation information (for example, an operation signal that is an electrical signal) from the image generation system (processing unit) of the present embodiment. As the movable mechanism 60 operates, at least one of the position and direction (orientation) of the seat 10 on which the player sits (in a broad sense, the movable object or the riding section on which the player rides) changes, and as a result, At least one of the viewpoint position and the line-of-sight direction of the player changes.

  FIG. 4 shows an example of a block diagram of the image generation system of the present embodiment. Note that the configuration of the image generation system according to the present embodiment is not limited to that shown in FIG. 4, and various modifications may be made such as omitting some of the components (each unit) or adding other components. .

  The imaging unit 40 (camera) includes an optical system such as a lens and an imaging element. The lens is a lens that projects an image on an image sensor, for example. The imaging element is realized by, for example, a CCD or CMOS sensor.

  The movable mechanism 60 (working housing) is a mechanism (machine) for operating a movable object such as the sheet 10 and the details thereof are as shown in FIG.

  The operation unit 160 is for a player to input operation information. 1 and 2, the operation unit 160 can be realized by the handle 50, the brake pedal 52, the accelerator pedal 54, the shift lever 56, an operation button (not shown), and the like.

  The storage unit 170 serves as a work area for the processing unit 100, the communication unit 196, and the like, and its function can be realized by a RAM (DRAM, VRAM) or the like. Then, the game program and game data necessary for executing the game program are held in the storage unit 170.

  An information storage medium 180 (a computer-readable medium) stores programs, data, and the like, and functions thereof are an optical disk (CD, DVD, BD), HDD (hard disk drive), and SSD (solid state drive). Alternatively, it can be realized by a memory (ROM or the like). The processing unit 100 performs various processes of the present embodiment based on a program (data) stored in the information storage medium 180. That is, in the information storage medium 180, a program for causing a computer (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute processing of each unit). Is memorized.

  The display unit 190 displays an image generated according to the present embodiment. In the case of FIGS. 1 and 2, the display unit 190 can be realized by an LCD in a liquid crystal projector, a DMD in a DLP projector, or the like. The sound output unit 192 outputs the sound generated by the present embodiment, and its function can be realized by a speaker, headphones, or the like.

  The auxiliary storage device 194 (auxiliary memory, secondary memory) is a storage device used to supplement the capacity of the storage unit 170, and can be realized by a memory card such as an SD memory card or a multimedia card.

  The communication unit 196 communicates with the outside (for example, another image generation system, a server, or a host device) via a wired or wireless network, and functions as a communication ASIC, a communication processor, or the like. It can be realized by hardware and communication firmware.

  The processing unit 100 (processor) performs game processing, image generation processing, sound generation processing, or the like based on operation information from the operation unit 160, a program, or the like. The processing unit 100 performs various processes using the storage unit 170 as a work area. The functions of the processing unit 100 can be realized by hardware such as various processors (CPU, GPU, etc.), ASIC (gate array, etc.), and programs.

  The processing unit 100 includes a game calculation unit 102, an object space setting unit 104, a moving body calculation unit 106, a viewpoint information calculation unit 108, a virtual camera control unit 110, an image generation unit 120, and a sound generation unit 130. Various modifications may be made such as omitting some of these components (each unit) or adding other components.

  The game calculation unit 102 performs game calculation processing. Here, as a game calculation, a process for starting a game when a game start condition is satisfied, a process for advancing the game, a process for calculating a game result, or a process for ending a game when a game end condition is satisfied and so on.

  The object space setting unit 104 performs an object space setting process in which a plurality of objects are arranged. For example, various objects (polygons, free-form surfaces) representing display objects such as moving objects (people, animals, robots, cars, airplanes, marine aircraft, etc.), maps (terrain), buildings, courses (roads), trees, walls, water surfaces, etc. Or an object configured with a primitive surface such as a subdivision surface) is set in the object space. In other words, the position and rotation angle of the object in the world coordinate system (synonymous with direction and direction) are determined, and the rotation angle (rotation angle around the X, Y, and Z axes) is determined at that position (X, Y, Z). Place the object. Specifically, the object data storage unit 172 of the storage unit 170 stores object data such as the position, rotation angle, moving speed, moving direction, etc. of the object (part object) in association with the object number. . The object space setting unit 104 performs a process of updating the object data for each frame, for example.

  The moving object calculation unit 106 performs control processing for moving a moving object (moving object) such as a person, an animal, a car, or an airplane. Also, control processing for operating the moving body is performed. That is, based on operation information input by the player through the operation unit 160, a program (movement / motion algorithm), various data (motion data), etc., a moving object (object, model object) is moved in the object space, Performs control processing to move the moving body (motion, animation). Specifically, a simulation process for sequentially obtaining movement information (position, rotation angle, speed, or acceleration) and motion information (part object position or rotation angle) of a moving body for each frame (1/60 second). I do. A frame is a unit of time for performing a moving / movement process (simulation process) and an image generation process of a moving object.

  The virtual camera control unit 110 performs control processing of a virtual camera (viewpoint, reference virtual camera) for generating an image that can be seen from a given (arbitrary) viewpoint in the object space. Specifically, a process for controlling the position (X, Y, Z) or rotation angle (rotation angle around the X, Y, Z axes) of the virtual camera (process for controlling the viewpoint position, the line-of-sight direction, or the visual field range). I do. The virtual camera control of the virtual camera control unit 110 can be realized based on the viewpoint information of the player obtained by the viewpoint information calculation unit 108, for example.

  The image generation unit 120 performs drawing processing based on the results of various processing (game processing and simulation processing) performed by the processing unit 100, thereby generating an image and outputting the image to the display unit 190. Specifically, geometric processing such as coordinate transformation (world coordinate transformation, camera coordinate transformation), clipping processing, perspective transformation, distortion correction transformation, or light source processing is performed, and drawing data ( The position coordinates, texture coordinates, color data, normal vector, α value, etc.) of the vertexes of the primitive surface are created. Based on the drawing data (primitive surface data), the object (one or a plurality of primitive surfaces) after the geometry processing is stored in a drawing buffer 176 (a buffer that can store image information in units of pixels such as a frame buffer and a work buffer). draw. Thereby, an image that can be seen from the virtual camera (given viewpoint) in the object space is generated. Note that the drawing processing performed by the image generation unit 120 can be realized by vertex shader processing, pixel shader processing, or the like.

  The sound generation unit 130 performs sound processing based on the results of various processes performed by the processing unit 100, generates game sounds such as BGM, sound effects, or sounds, and outputs the game sounds to the sound output unit 192.

  And the image generation system of this embodiment contains the object space setting part 104, the viewpoint information calculating part 108, and the image generation part 120, as shown in FIG.

  The object space setting unit 104 performs an object space setting process. That is, a process for arranging and setting a plurality of objects in the object space is performed.

  The viewpoint information calculation unit 108 sets tracking viewpoint information obtained by the viewpoint tracking of the player as the viewpoint information of the player. For example, the viewpoint information of the player in the real space is specified by a viewpoint tracking method such as eye tracking, face tracking, or head tracking. The tracking viewpoint information is viewpoint information obtained by viewpoint tracking. As a viewpoint tracking method for obtaining the tracking viewpoint information, various known methods can be employed.

  The player's viewpoint information is information including at least one of the player's viewpoint position and line-of-sight direction. The viewpoint information may include information on a plurality of viewpoint positions and line-of-sight directions, and other information (for example, information on the binocular interval).

  The image generation unit 120 generates an image that can be seen from the viewpoint of the player in the object space based on the identified viewpoint information. For example, based on the viewpoint information in the real space, the viewpoint position and the line-of-sight direction in the object space corresponding to the viewpoint information are specified, and an image that can be seen from the viewpoint is generated. The generated image may be a projection image configured by one curved surface or a plurality of surfaces and projected on a screen, or may be a stereoscopic image that realizes stereoscopic vision. The stereoscopic image may be displayed on a flat display, or the stereoscopic image may be projected on a screen formed of one curved surface or a plurality of surfaces. Alternatively, an image displayed on an HMD (head-mounted display device) may be used.

  In this embodiment, the viewpoint information calculation unit 108 performs the viewpoint information estimation process (determination process) of the player when the tracking viewpoint is in a lost state. For example, when the player's viewpoint (the viewpoint in the real world) cannot be captured for some reason and the tracking viewpoint information cannot be acquired (lost state), the player's viewpoint information is estimated.

  For example, in a state where the viewpoint tracking is successful, the viewpoint information calculation unit 108 sets (identifies) the tracking viewpoint information acquired by the viewpoint tracking as the viewpoint information of the player. Then, the image generation unit 120 generates an image that can be seen from the viewpoint of the player in the object space, based on the set viewpoint information. On the other hand, when the viewpoint tracking is in a lost state, the viewpoint information calculation unit 108 performs a process for estimating the viewpoint information of the player. Then, the image generation unit 120 generates an image that can be seen from the viewpoint of the player in the object space, based on the viewpoint information obtained by the estimation process. In other words, an image that is visible at the viewpoint position in the object space corresponding to the estimated viewpoint of the player is generated.

  For example, the viewpoint information calculation unit 108 sets tracking viewpoint information obtained from a captured image obtained by photographing the player as the viewpoint information of the player. For example, as shown in FIGS. 1 and 2, in the present embodiment, an imaging unit 40 that photographs the player is provided. The imaging unit 40 is installed, for example, at a position where at least the player's face can be photographed. That is, it is installed such that the shooting direction (optical axis direction) faces the player (player's face) at a position where the player can be seen from the front. Then, the viewpoint information calculation unit 108 sets the tracking viewpoint information obtained from the captured image (image data) from the imaging unit 40 as the viewpoint information of the player. Even if the imaging unit 40 is a normal camera, the viewpoint position including the depth direction can be estimated to some extent from the size of the face and the distance between both eyes, but a plurality of color cameras are used for more accurate determination. It is good also as a structure which can acquire a position also with respect to a distance direction. Further, in addition to a normal color camera, an object provided with a distance measuring camera that can acquire distance information may be used.

  The viewpoint information calculation unit 108 performs viewpoint information estimation processing when the tracking viewpoint is in a lost state based on the viewpoint history information of the player. The viewpoint history information (viewpoint movement history information) is a viewpoint information group obtained by viewpoint calculation processing such as viewpoint tracking in the past, for example. This viewpoint history information is stored in the viewpoint history information storage unit 177 of the storage unit 170. The viewpoint history information may be information generated by a device different from the image generation system of the present embodiment. In this case, the viewpoint history information may be acquired via the network or the auxiliary storage device 194 and stored in the viewpoint history information storage unit 177.

  For example, the viewpoint information calculation unit 108 performs viewpoint information estimation processing when the tracking viewpoint is lost based on at least one of viewpoint history information, movement information of a moving body operated by the player, and operation information of the player. The moving body is an object that moves in the object space based on the operation of the player. The movement information of the moving body is the position, speed, acceleration, angular velocity, angular acceleration, or the like of the moving body. Note that the velocity and acceleration here are generally vector quantities. Alternatively, the moving information of the moving body may be a moving path of the moving body. The player operation information is information obtained by the player operating the operation unit 160. This operation information may be operation history information that is past operation information of the player.

  For example, the viewpoint history information storage unit 177 stores a plurality of candidate viewpoint history information, and the viewpoint information calculation unit 108 selects the plurality of viewpoints based on the movement information of the moving object and the operation information of the player. The viewpoint history information (viewpoint information) is selected from the history information, and the player's viewpoint information is set (estimated) based on the selected viewpoint history information.

  For example, the viewpoint information calculation unit 108 selects the first viewpoint history information (first viewpoint information) from the plurality of viewpoint history information when the position of the moving object is the first position, If the position is 2, the second viewpoint history information (second viewpoint information) is selected. In addition, the viewpoint information calculation unit 108, when the speed (or acceleration, angular velocity, angular acceleration) of the moving body is the first speed (first acceleration, angular velocity, angular acceleration) among the plurality of viewpoint history information. Selects the first viewpoint history information (first viewpoint information), and if it is the second speed (second acceleration, angular velocity, angular acceleration), the second viewpoint history information (second viewpoint information) ) Is selected. The viewpoint information calculation unit 108 also sets the first viewpoint history information when the player has performed a first operation (for example, turning the steering wheel to the left or stepping on the accelerator pedal) from among a plurality of viewpoint history information. When (first viewpoint information) is selected and the player is performing a second operation (for example, turning the steering wheel to the right or pressing the brake pedal), the second viewpoint history information (second viewpoint information) is selected. ) Is selected.

  In addition, when the moving body operated by the player moves on the course or the map, the viewpoint information calculation unit 108 determines the tracking viewpoint based on the viewpoint history information when it is located in the same area (course block or map block) in the past. Estimating viewpoint information in the lost state. Here, the course is set as a route on which a moving body such as a car should move in the object space. This course is used for car racing games and the like. The course has a given width, for example, in a direction intersecting (orthogonal) with its traveling direction. Course shape data, running condition setting data at each position of the course, and the like are stored in the course data storage unit 178 as course data. The map is a field in which a moving body such as a character moves in the object space. The map terrain data, game condition setting data at each position of the map, and the like are stored in the map data storage unit 179 as map data.

  The moving object calculation unit 106 performs calculation processing for moving the moving object on the course or the map based on the operation information of the player from the operation unit 160. When the moving object is located in the first area (first course block, first map block) of the course or the map, the viewpoint information calculation unit 108 moves the moving object to the first area in the past. Read viewpoint history information when is located. For example, the viewpoint history information associated with the first area is read. Based on the read viewpoint history information, the viewpoint information of the player when the moving body is located in the first area is specified. Similarly, when the moving body is located in the second area of the course or the map (second course block, second map block), the viewpoint information calculation unit 108 moves to the second area in the past. Read viewpoint history information when the moving body is located. For example, the viewpoint history information associated with the second area is read. Based on the read viewpoint history information, the viewpoint information of the player when the moving body is located in the second area is specified.

  In this case, a round course can be used as the course. The circuit course is a course in which a moving body can make many laps such as one lap, two laps, and three laps in a competitive game or the like. The viewpoint information calculation unit 108 performs viewpoint information estimation processing in the lost state of the tracking viewpoint based on viewpoint history information when the moving body operated by the player has passed through the same area in the past in this circuit course. For example, when the current number of laps is the m-th lap, based on the viewpoint history information when passing (positioning) the same area (course block) in the first lap to the (m-1) -th lap, The player's viewpoint information when passing through the area is specified. For example, the viewpoint information is selected from the first to m−1th viewpoint history information corresponding to the 1st to m−1th laps based on, for example, movement information of the moving body or operation information of the player. Thus, the viewpoint information of the player when passing through the area is specified.

  Moreover, the image generation system (game system) of this embodiment has the movable mechanism 60 as demonstrated in FIGS. In this case, the viewpoint information calculation unit 108 performs viewpoint information estimation processing when the tracking viewpoint is in a lost state based on the operation information of the movable mechanism 60. The operation information of the movable mechanism 60 is information for setting the operation state of the movable mechanism 60. For example, the motion information is information for specifying the posture state (rotation state) and position (movement position) of the sheet 10 that is a movable object. The operation control of the movable mechanism 60 is performed by the processing unit 100.

  For example, when it is determined that the seat 10 that is the movable object is in the first posture state (first position) based on the operation information of the movable mechanism 60, the viewpoint information calculation unit 108 Is set as the viewpoint information of the player in the lost state, which is assumed when the player is in the first posture state (first position). When it is determined that the sheet 10 that is the movable object is in the second posture state (second position) based on the operation information of the movable mechanism 60, the sheet 10 is in the second posture state (second position). 2) is set as the viewpoint information of the player in the lost state.

  For example, the viewpoint information calculation unit 108 obtains the operation information of the movable mechanism 60 based on the player operation information or the game situation information, and estimates the viewpoint information when the tracking viewpoint is in the lost state based on the obtained operation information. I do.

  For example, the operation information of the movable mechanism 60 is obtained based on operation information such as the player turning the handle 50 to the right or to the left in FIG. That is, the operation information for setting the operation state (rotation state, etc.) of the movable mechanism 60 is obtained. The movable mechanism 60 operates based on an operation signal that is this operation information.

  For example, when the player turns the handle 50 to the right, the movable mechanism 60 is operated so that, for example, the seat 10 rotates clockwise in order to make the player feel cornering in the right direction. Then, the viewpoint position and line-of-sight direction assumed when cornering in the right direction are set (estimated) as the viewpoint information of the player. When the player turns the handle 50 to the left, the movable mechanism 60 is operated so that, for example, the seat 10 rotates counterclockwise so that the player can feel the cornering in the left direction. Then, the viewpoint position and the line-of-sight direction assumed when cornering in the left direction are set (estimated) as the viewpoint information of the player.

  Further, based on the game situation information, for example, if it is determined that the moving body of the player is traveling on an uneven road surface, the seat 10 is slightly moved up and down in the Y-axis direction of FIG. The movable mechanism 60 (electric cylinders 73 and 74) is operated. Then, the viewpoint position and line-of-sight direction assumed when traveling on an uneven road surface are set (estimated) as player viewpoint information. For example, the player's viewpoint position is raised or lowered according to the unevenness of the road surface.

  Note that the operation information is information acquired when the player operates the operation unit 160, for example. The game situation information is information representing the game situation at that time, and is information that changes as the game progresses. For example, in a game in which a moving body moves, the game situation information is information representing a running situation such as a running environment of the moving body. In a game using a character (RPG game, martial arts game, action game) or the like, the game situation information is information based on information indicating the current situation of the character in the game, character status parameters, or the like. Further, as the game status information, for example, information indicating the progress status of the game or information indicating the achievement level of the game (map clear status, acquired points, etc.) can be assumed.

  The viewpoint information calculation unit 108 may perform viewpoint information estimation processing when the tracking viewpoint is in a lost state, based on player information including at least one of the player's physique information, gender information, and age information. For example, when the player has a first physique (for example, high height, heavy weight, etc.) and a second physique (for example, low height, light weight, etc.), the player in the lost state Different viewpoint information. Alternatively, the player's viewpoint information in the lost state is different depending on whether the player is male or female. Alternatively, the player's viewpoint information in the lost state is different depending on whether the player is an adult or a child.

  Specifically, the viewpoint information calculation unit 108 performs a process of selecting viewpoint information based on player information from among a plurality of candidate viewpoint information as an estimation process of viewpoint information when the tracking viewpoint is in a lost state. For example, when the player has the first physique, the viewpoint information calculation unit 108 selects viewpoint information (viewpoint history information) prepared for the first physique from a plurality of candidate viewpoint information, Set the viewpoint information of the player in the lost state. In the case of the second physique, the viewpoint information (viewpoint history information) prepared for the second physique is selected, and the viewpoint information of the player in the lost state is set. Alternatively, when the player is a male, the viewpoint information calculation unit 108 selects viewpoint information (viewpoint history information) prepared for the male and sets the viewpoint information of the player in the lost state. In the case of a woman, the viewpoint information (viewpoint history information) prepared for the woman is selected and the viewpoint information of the player in the lost state is set. Alternatively, when the player is an adult, the viewpoint information (viewpoint history information) prepared for the adult is selected, and the viewpoint information of the player in the lost state is set. If it is a child, the viewpoint information (viewpoint history information) prepared for the child is selected, and the viewpoint information of the player in the lost state is set.

  The viewpoint information calculation unit 108 may perform viewpoint information estimation processing when the tracking viewpoint is in a lost state based on the result of the physical simulation processing of the player's motion. For example, a physical simulation process of the operation of the player (model object) is performed based on the operation information of the player, the game situation information, or the operation information of the movable mechanism 60 obtained from the operation information and the game situation information. For example, based on operation information, game situation information, operation information of the movable mechanism 60, and the like, how the motion of the player changes is obtained by a known physical simulation process. Then, based on the obtained motion of the player (model object), the viewpoint information of the player when the tracking viewpoint is lost is estimated. For example, the motion information when the moving body performs right cornering or left cornering is obtained by physical simulation processing, and the movement of the player's viewpoint position or gaze position at that time is specified based on the motion information To do. Then, the identified viewpoint position and line-of-sight direction are estimated as the viewpoint information of the player in the lost state.

  The image generation unit 120 generates a projection image based on the shape information of the projection screen constituted by one curved surface or a plurality of surfaces and the viewpoint information of the player. The projection screen shape information is, for example, information representing the shape of the projection screen by a mathematical expression (for example, an equation of an ellipsoid) or a polygon model. The image generation unit 120 generates a projection image by performing distortion correction processing for reflecting the shape of the projection screen formed by one curved surface or a plurality of surfaces.

  That is, the image generation unit 120 generates a projection image for projection. The generated projection image is projected onto the screen 20 by the projection device 30 shown in FIGS. As a result, the player can view the image of the object space viewed from the virtual camera as a game image. The projection image for projection is, for example, a projection image generated by projection mapping processing. Projection mapping is a method of projecting with the projection device 30 taking into account the state (shape, etc.) of the object (screen) to be projected, the state (position, direction, etc.) of the projection device 30 and the state of the viewpoint (position, direction, etc.). .

  Specifically, the image generation unit 120 includes an intersection position between a light beam emitted from a pixel (projection image pixel) on the drawing buffer 176 through an optical system (such as a wide-angle lens) of the projection device 30 and the projection screen; The color of the pixel on the drawing buffer 176 is determined using the straight line connecting the viewpoint positions as the line of sight of the virtual camera. For example, using this straight line as the line of sight of the virtual camera, the pixel color is determined from the information in the object space. Specifically, the color of the pixel is determined according to the intersection position of this straight line and the object in the object space (the position of the point on the object where this straight line first arrives in the object space). This process can also be realized by the ray tracing method, but the drawing load is large and it is difficult to implement in real time. Therefore, a more practical method can be realized by saving the rendering result to a render texture on a flat screen that is as close as possible to a curved screen (this will be referred to as a “proxy plane”). Good. Then, the projection image is generated by determining the color of the pixel in this way.

  Alternatively, the image generation unit 120 obtains a straight line connecting the vertex position of the object in the object space and the viewpoint position, and obtains an intersection position between the straight line and the projection screen. Then, based on the obtained intersection position, the vertex position on the drawing buffer 176 for the drawing object corresponding to the object is obtained. Then, a projection image is generated by drawing a drawing object in the drawing buffer 176 based on the obtained vertex position.

  Note that the projection screen is, for example, a virtual screen for generating a projection image that is arranged and set in an object space that is a virtual three-dimensional space, corresponding to the screen 20 in FIGS. In the image generation unit 120, distortion correction processing (also referred to as projection mapping processing) is performed in accordance with the shape of the projection screen. This projection screen can be composed of one curved surface or a plurality of surfaces (curved surface, flat surface).

  The drawing object is a two-dimensional object corresponding to the three-dimensional object to be drawn. For example, a three-dimensional object means an object arranged in a three-dimensional space (object space), and is an object having, for example, a three-dimensional coordinate value (X, Y, Z coordinate value) as a coordinate value of its vertex. On the other hand, a drawing object is an object having, for example, a two-dimensional coordinate value (X, Y coordinate) as a coordinate value of its vertex. The drawing buffer 176 is a buffer capable of storing image information in units of pixels, such as a frame buffer and a work buffer.

  The image generation unit 120 may generate a first viewpoint image and a second viewpoint image for stereoscopic viewing based on the viewpoint information of the player. For example, the first viewpoint image is a left-eye image, and the second viewpoint image is a right-eye image. For example, the viewpoint information of the left-eye virtual camera (first viewpoint virtual camera) and the right-eye virtual camera (second viewpoint virtual camera) are set based on the player's viewpoint information (viewpoint position, line-of-sight direction). To do. Then, a left-eye image (first viewpoint image) is generated based on the left-eye virtual camera, and a right-eye image (second viewpoint image) is generated based on the right-eye virtual camera. Note that, as a method for actually observing a stereoscopic image, a spectacle method (specifically, a polarization method, a time division method, a spectroscopic method, etc.) or a naked eye method may be considered.

2. Next, the method of this embodiment will be described in detail. In the following, the case where the method of the present embodiment is applied to a system that generates a projection image on a projection screen will be mainly described, but the present embodiment is not limited to this. As will be described later, the method of the present embodiment is applicable to various image generation systems such as a system that generates a stereoscopic image and a system that generates a display image on an HMD.

2.1 Viewpoint Tracking In order to generate a projection image on the projection screen, it is necessary to perform distortion correction processing to reflect the shape of the projection screen. In order to perform this distortion correction processing accurately, viewpoint information such as the player's viewpoint position is required. Details of the distortion correction processing will be described later.

  Until now, in order to realize such distortion correction processing, as shown in FIG. 5A, the representative viewpoint RVP of the player is set, and distortion correction processing for generating a projection image is performed. It was. For example, in FIG. 5A, the representative viewpoint RVP is set to a position at a height HV from the reference position RP on the bottom surface of the sheet 10. This representative viewpoint RVP does not exactly match the player's actual viewpoint. For example, the first player who is tall and the second player who is short have different viewpoint positions, but in FIG. 5A, these first and second players use the same representative viewpoint RVP. Then, a distortion correction process is performed to generate a projection image. This is because a slight difference in the viewpoint position is considered not so great for the generated projection image.

  However, for example, when the movable mechanism 60 as described with reference to FIG. 3 is provided, the movable mechanism 60 operates, so that the viewpoint position VP of the player is obtained as shown in FIGS. 5B and 5C. And the line-of-sight direction VL change. In such a situation where the movable mechanism 60 operates and the player's viewpoint position VP and line-of-sight direction VL change, it is unnatural if a projection image is generated by the technique using the representative viewpoint RVP as shown in FIG. An image with a large sense of discomfort is displayed. That is, if a projection image subjected to the distortion correction processing calculated at the representative viewpoint RVP in FIG. 5A is projected on the projection screen, the player may feel a great sense of incongruity.

  Therefore, in the present embodiment, a technique is adopted in which the viewpoint information of the player is set by capturing the viewpoint of the player by viewpoint tracking, and an image such as a projection image is generated using the set viewpoint information. Specifically, the tracking viewpoint information obtained by the player's viewpoint tracking is set as the player's viewpoint information, and an image that is visible from the player's viewpoint in the object space is generated based on the set viewpoint information.

  The viewpoint tracking in this case is desirably performed based on the captured image of the imaging unit 40 installed as shown in FIGS. That is, the tracking viewpoint information obtained from the captured image obtained by photographing the player is set as the viewpoint information of the player, and an image such as a projection image is generated. FIG. 6A shows an example of the imaging unit 40 having a configuration suitable for viewpoint tracking.

  For example, the imaging unit 40 in FIG. 6A is provided with first and second cameras 45 and 46. By using the first and second captured images captured by the first and second cameras 45 and 46, viewpoint tracking for capturing the viewpoint of the player can be realized. For example, by providing a plurality of cameras 45 and 46 in the imaging unit 40, information in the depth direction of the player who is the subject in the real space can be obtained. For example, using the captured images of the camera 45 and the camera 46, it is possible to obtain the player's viewpoint position on a plane orthogonal (intersect) in the depth direction (the optical axis direction of the imaging unit 40). Further, the position of the viewpoint of the player in the depth direction can be obtained using the parallax information of the captured images of the cameras 45 and 46, and thereby the three-dimensional position of the viewpoint can be obtained. Alternatively, the three-dimensional position of the viewpoint may be obtained by using a camera having a color image sensor as the camera 45 and using a camera having a depth sensor as the camera 46. Furthermore, even with only a camera having a single color image sensor, the face size and binocular spacing are measured in advance at a fixed position, the face size estimated from information such as age, gender, and physique By calculating using the distance between both eyes, the three-dimensional position of the viewpoint can be obtained, although the accuracy is slightly reduced.

  In FIG. 6B, eyeglasses 200 (a mounting member in a broad sense) are prepared, and the eyeglasses 200 are provided with a left eye marker MKL corresponding to the left eye of the player and a right eye marker MKR corresponding to the right eye. That is, the left eye marker MKL is provided at a position corresponding to the left eye portion of the glasses 200, and the right eye marker MKR is provided at a position corresponding to the right eye portion. These left-eye and right-eye markers MKL and MKR have different shapes.

  Then, the player wearing the glasses 200 is photographed by the imaging unit 40, and the position information of the left eye and right eye of the player is acquired based on the captured image. That is, the player is imaged by the imaging unit 40, and image recognition processing is performed on the obtained captured image to recognize the shapes of the left-eye and right-eye markers MKL and MKR in FIG. Then, for example, by detecting the positions of the left eye and right eye of the player based on the result of shape recognition, the viewpoint information of the player is acquired, and viewpoint tracking is realized. In particular, when performing a stereoscopic display using glasses, the brightness of the portion viewed from the imaging unit 40 may be reduced due to the presence of the stereoscopic glasses, which may make eye tracking difficult. By attaching to the glasses, eye tracking can be performed more accurately.

  Alternatively, viewpoint tracking may be realized by eye tracking as shown in FIG. For example, by eye tracking, the positions and shapes of the left eye and right eye pupils of the player are recognized, the left eye and right eye positions ERP, ELP, the line-of-sight directions ERD, ELD, etc. are specified, and the player viewpoint information is acquired. Realize tracking. This eye tracking can be realized, for example, by photographing the left eye and the right eye of the player by the imaging unit 40 and performing image recognition processing such as a pupil on the captured image.

  Alternatively, viewpoint tracking may be realized by face tracking as shown in FIG. For example, the imaging unit 40 captures the player's face and performs face image recognition processing. Then, based on the result of the image recognition process, the position and direction of the player's face are specified, and the viewpoint position VP and line-of-sight direction VL of the player are obtained to realize viewpoint tracking.

  Note that the viewpoint tracking method of the present embodiment is not limited to the method described with reference to FIGS. 6A to 7B, and various known viewpoint tracking methods that can specify the viewpoint information of the player can be employed.

  In this way, if viewpoint information is acquired by viewpoint tracking and an image such as a projection image is generated using the acquired viewpoint information, a more natural and less uncomfortable image can be generated.

  However, if viewpoint tracking is successful, appropriate viewpoint information can be acquired and a natural image can be generated, but if viewpoint tracking fails and the tracking viewpoint is lost, it is appropriate Can not get the correct viewpoint information. For this reason, an uncomfortable image or a broken image may be generated.

  For example, when the movable mechanism 60 operates by rotating the seat 10 as shown in FIG. 8A, the player covers his / her eyes with his hand as shown in FIG. 8B due to this operation. In some cases, the player may bend forward as shown in FIG. When such a situation occurs, the imaging unit 40 cannot capture the player's face, the player's viewpoint cannot be captured, and the tracking viewpoint becomes lost. Alternatively, there may be a case where the viewpoint of the spectator watching the game is mistakenly tracked due to a change or movement of the player's posture, and as a result, the viewpoint position suddenly jumps. This is a so-called “misrecognition state”. For example, a threshold value for the amount of movement of the viewpoint position between frames is set, and processing such as a recognition error when the amount of change in the viewpoint position exceeds the threshold value, etc. By doing so, it can be returned to the lost state, so that it will be described below including the lost state.

  When such a tracking viewpoint lost state occurs, the player's viewpoint information cannot be acquired, or the reliability of the acquired player's viewpoint information is significantly reduced, and an image such as an appropriate projection image cannot be generated. End up.

2.2 Estimating Viewpoint Information in Lost State In order to solve the above-described problems, in the present embodiment, processing for estimating the player's viewpoint information in the lost state of the tracking viewpoint is performed, and the player's viewpoint information in the object space is determined. A technique for generating an image that can be seen from the viewpoint is adopted.

  FIG. 9 is a flowchart showing a processing example of the technique of this embodiment. First, it is determined whether or not the frame is updated (step S1). For example, it is determined whether it is time to update the frame image. If it is the frame update timing, the captured image is acquired by the imaging unit 40 (step S2). That is, the photographed image is acquired although the face of the player is photographed. Then, viewpoint tracking is performed based on the acquired captured image to obtain tracking viewpoint information (step S3). That is, viewpoint tracking such as that described with reference to FIGS. 6A to 7B is performed to set viewpoint information such as the viewpoint position and line-of-sight direction of the player.

  Next, it is determined whether or not the tracking viewpoint is in a lost state (step S4). For example, as described with reference to FIGS. 8B and 8C, viewpoint tracking has failed due to, for example, that the face of the player cannot be captured by the imaging unit 40, and tracking viewpoint information is obtained in step S3. It is determined whether or not the lost state is not possible. If the tracking viewpoint is in a lost state, viewpoint information estimation processing is performed (step S5). For example, processing for estimating the player's viewpoint information is performed based on various types of information such as viewpoint history information, player operation information, game situation information, operation information of the movable mechanism 60, or player information. Based on the viewpoint information, an image that can be seen from the viewpoint of the player in the object space is generated (step S6).

  For example, when the tracking viewpoint is not in the lost state, an image that can be seen from the viewpoint of the player in the object space is generated based on the tracking viewpoint information obtained in step S3. On the other hand, if the tracking viewpoint is in a lost state, an image that is visible from the player's viewpoint in the object space is generated based on the viewpoint information estimated in step S5.

  At this time, when generating an image to be projected onto the curved screen 20 as shown in FIG. 1, a distortion correction process as described later is performed to generate a projection image. When generating a stereoscopic image, based on viewpoint information (tracking viewpoint information or viewpoint information obtained by estimation processing), a left-eye virtual camera (first viewpoint in a broad sense), a right-eye virtual camera (in a broad sense) Is set to generate a left-eye image for stereoscopic viewing (first viewpoint image in a broad sense) and a right-eye image (second viewpoint image in a broad sense).

  According to the method of the present embodiment described above, an image that can be seen from the viewpoint of the player is generated based on the tracking viewpoint information obtained by viewpoint tracking. Therefore, for example, even when the viewpoint position VP or the line-of-sight direction VL of the player in the real world changes greatly due to the movement of the movable mechanism 60 as shown in FIGS. Since the image is generated based on the viewpoint information of the player captured by the tracking, it is possible to display an image on which the player hardly feels unnaturalness or discomfort. For example, in the case of a projected image on a curved screen 20 as shown in FIG. 1, it is possible to generate a projected image on which distortion correction processing has been performed more appropriately. Also, when generating a stereoscopic image, a stereoscopic image that does not contradict the position of the player's viewpoint in the real world can be displayed on the player.

  In this embodiment, when an image is generated using such viewpoint tracking, for example, a situation has occurred in which the tracking viewpoint is in a lost state as shown in FIGS. 8B and 8C. Even in this case, the viewpoint information of the player is estimated based on various information such as viewpoint history information, player operation information, game situation information, operation information of the movable mechanism 60, or player information. Since the image is generated based on the viewpoint information obtained by the estimation process, it is possible to prevent a situation in which an uncomfortable image that is far from the viewpoint position of the real world player is displayed on the player, for example. . Accordingly, it is possible to provide an image generation system that can generate a more appropriate image even when viewpoint tracking fails.

2.3 Estimation Process Based on Viewpoint History Information Next, a specific example of the viewpoint information estimation process of the present embodiment will be described. In this embodiment, based on the player's viewpoint history information, viewpoint information estimation processing is performed when the tracking viewpoint is in a lost state. The viewpoint information estimation process based on this viewpoint history information will be described.

  FIG. 10A shows an example of viewpoint history information. 10A, for each index (I1, I2, I3...), The viewpoint position (VP1, VP2, VP3...) And the line-of-sight direction (VL1, VL2, VL3...・) And other viewpoint information are associated. As the index of the viewpoint history information, various indexes such as a frame number, a time stamp, a course block number, and a map block number can be assumed.

  In FIG. 10B, the moving body MV operated by the player is moving on the course CS. That is, based on the operation information input by the operation unit 160, a calculation process for moving the moving body MV is performed, and thereby the moving body MV moves along the course CS. In the path LC1, the moving body MV moves on the course CS by so-called out / in / out, and in the path LC2, the moving body MV moves out / out / out. For example, the route LC1 is the first round route of the course CS, and the route LC2 is the second round route.

  In FIG. 10C, VPHT1 is viewpoint history information on the route LC1 in the first round, and VPHT2 is viewpoint history information on the route LC2 in the second round. For example, each index (I1, I2, I3...) Is a frame number (time stamp). In this case, when the moving body MV moves on the route LC1 in the first round of the circuit course CS, the player's viewpoint position (VP11, VP12, VP13... ), The line-of-sight directions (VL11, VL12, VL13...) Are associated with each frame number (I1, I2, I3...) And stored as viewpoint history information VPHT1. Similarly, when the moving body MV moves on the route LC2 in the second lap of the course CS, the player viewpoint positions (VP21, VP22, VP23...) Obtained by viewpoint tracking in each frame, The line-of-sight directions (VL21, VL22, VL23...) Are stored as viewpoint history information VPHT2 in association with each frame number (I1, I2, I3...). In these routes LC1 and LC2, since the course of the moving body MV is different, the viewpoint position and the line-of-sight direction stored in association with each frame number (each index) in the viewpoint history information VPHT1 and VPHT2 are also different. .

  For example, when the moving body MV operated by the player passes the same corner shown in FIG. 10B in the third lap of the circuit course CS, the viewpoint tracking is in a lost state. Specifically, the lost state where the player's face and the like cannot be captured by the imaging unit 40 due to the state shown in FIGS. 8B and 8C and the viewpoint information of the player cannot be obtained by viewpoint tracking. Suppose that it becomes a state.

  In this case, in this embodiment, the viewpoint information of the player when the tracking viewpoint is in the lost state is estimated using the viewpoint history information VPHT1 and VPHT2 shown in FIG. Specifically, the viewpoint information in the lost state is estimated based on the viewpoint history information VPHT1 and VPHT2 and the movement information of the moving object MV or the operation information of the player. For example, based on the movement information such as the position, velocity, acceleration, angular velocity or angular acceleration of the moving body MV in the corner on the third lap of the course CS and the viewpoint history information VPHT1, VPHT2, the viewpoint in the lost state Estimate information. For example, based on the position of the moving body MV, the degree of proximity between the moving path LC3 in the third turn of the moving body MV and the moving paths LC1 and LC2 in FIG. Then, for example, based on viewpoint history information (VPHT1 or VPHT2) associated with the movement path closer to the movement path LC3 of the third lap among the movement paths LC1 and LC2 of the first and second laps. Thus, viewpoint information such as the player's viewpoint position and line-of-sight direction in the lost state is obtained. Alternatively, the weighting coefficients WT1 and WT2 corresponding to each of the movement paths LC1 and LC2 are set based on the degree of proximity to the movement path LC3 in the third lap, and the viewpoint history information VPHT1 and the weighting coefficients WT1 and WT2 The weighting process of VPHT2 is performed, and the viewpoint information of the player in the lost state is obtained.

  Alternatively, when it is determined that the speed (or acceleration, angular velocity, angular acceleration) of the moving object MV is high, the viewpoint history information VPHT2 of the movement path LC2 is adopted, or the weighting coefficient of the viewpoint history information VPHT2 is increased. Thus, the viewpoint information of the player in the lost state is obtained. On the other hand, when it is determined that the speed (or acceleration, angular velocity, angular acceleration) of the moving body MV is slow, the viewpoint history information VPHT1 of the movement path LC1 is adopted, or the weighting coefficient of the viewpoint history information VPHT1 is increased. Thus, the viewpoint information of the player in the lost state is obtained. In this way, the viewpoint information when the tracking viewpoint is in the lost state based on the viewpoint history information when the moving body MV operated by the player has passed through the same corner (area in a broad sense) in the past in the course CS. Can be estimated.

  In the circuit course CS in FIG. 11A, a plurality of course blocks are set for the course CS. For example, for the corner indicated by E1 in FIG. 11A, course blocks C1 to C11 are set as shown in FIG. Each of the course blocks C1 to C11 is divided by lines in a direction intersecting (orthogonal) with the traveling direction of the course CS. In the viewpoint history information of FIG. 11C, this course block (course block number) is used as an index, and viewpoint information such as the viewpoint position and the line-of-sight direction is stored. That is, the index in the viewpoint history information in FIG. 10A is a course block in FIG.

  The course block may be a block to which information other than the viewpoint information is further associated. For example, shape data for specifying the shape of each course block, running condition setting data in each course block, and the like may be stored in association with the course block together with the viewpoint information. The shape data is data such as the course width of each course block in a direction intersecting (orthogonal) with the traveling direction, for example. The travel condition setting data is, for example, data for setting the travel conditions of the moving body in each course block, and is, for example, road surface data indicating the degree of friction and slipperiness of the road surface. The course data storage unit 178 in FIG. 4 stores course data in which such various types of information are associated with each course block. Note that a technique for associating various information with a course block and storing it as course data is disclosed in, for example, Japanese Patent Application Laid-Open No. 2000-348216.

  FIG. 12 is an example of viewpoint history information generated in a course CS as shown in FIGS. 11 (A) and 11 (B). For example, the viewpoint history information VPHT1 of FIG. 12 is generated based on viewpoint tracking or the like in traveling of the moving body in the first lap of the circuit course CS. The viewpoint history information VPHT2 is generated based on viewpoint tracking or the like in traveling in the second lap. In this way, viewpoint history information VPHT1 to VPTN is generated when the moving body makes N turns around the course CS.

  Then, for example, when the tracking viewpoint lost state as described with reference to FIGS. 8B and 8C occurs during traveling of the moving body on the (N + 1) th lap of the course CS, a plurality of viewpoint histories in FIG. The viewpoint information is selected based on the information VPHT1 to VPHTN (first to Nth viewpoint history information) and the movement information of the moving body (or the operation information of the player) at that time. The selected viewpoint information is estimated as the viewpoint information of the player when the tracking viewpoint is in the lost state. In this way, the viewpoint information in the lost state of the tracking viewpoint is estimated based on the viewpoint history information when the moving body has passed the same course block (area in a broad sense) in the past in the circuit CS of the circuit. become.

  FIG. 13 shows a flowchart of viewpoint information estimation processing based on viewpoint history information. First, movement information such as the position and speed of the moving body is calculated (step S11). This is realized by performing movement calculation processing of the moving body based on the player operation information input from the operation unit 160. Then, the course block where the moving body is located is specified (step S12). For example, when the moving body is moving in the corner indicated by E1 of the course CS in FIG. 11A, it is specified in which of the course blocks C1 to C11 in FIG. 11B the moving body is located. Is done. The course block is specified based on the information on the position of the moving body obtained in step S11.

  Next, viewpoint information corresponding to movement information (or player operation information) of the moving body is selected from a plurality of viewpoint information associated with the identified course block in a plurality of viewpoint history information (step S13). The selected viewpoint information is estimated as viewpoint information when the tracking viewpoint is in a lost state (step S14).

  For example, in FIG. 12, when the number of laps of the moving body is N + 1, the course block specified in step S12 in the viewpoint history information VPHT1 to VPHTN obtained in the 1st to Nth laps. The viewpoint information corresponding to the movement information of the moving body (player operation information) is selected from the plurality of associated viewpoint information.

  For example, it is assumed that the course block in which the moving body is located in the tracking viewpoint lost state is the course block C1 in FIG. In this case, in the viewpoint history information VPHT1 to VPTN, movement information such as the speed and acceleration of the moving body is selected from the viewpoint information such as the viewpoint positions VP11 to VPN1 and the line-of-sight directions VL11 to VLN1 associated with the course block C1. The viewpoint information corresponding to is selected.

  Specifically, in FIG. 12, each course block is associated with the speed of the moving body when it has passed the course block in the past (moving information in a broad sense, or may be player operation information). It has been. For example, in the viewpoint history information VPHT1, VPHT2,..., VPHTN, the speed VC11, VC21,..., VCN1 when passing through the course block C1 in the first to Nth laps is associated with the course block C1. ing. When the speed of the moving body in the course block C1 is VCX in the (N + 1) th lap, for example, the speed VCX is selected from the speeds VC11 to VCN1 associated with the course block C1 of the viewpoint history information VPHT1 to VPHTN. Select the speed closest to.

  For example, it is assumed that the speed VC11 associated with the course block C1 in the viewpoint history information VPHT1 is closest to the speed VCX. In this case, the viewpoint position VP11 and the line-of-sight direction VL11 associated with the course block C1 in the viewpoint history information VPHT1 are selected and estimated as the player viewpoint information when the tracking viewpoint is in the lost state. Alternatively, it is assumed that the speed VCN1 associated with the course block C1 in the viewpoint history information VPHTN is closest to the speed VCX. In this case, the viewpoint position VPN1 and the line-of-sight direction VLN1 associated with the course block C1 in the viewpoint history information VPHTN are estimated as the viewpoint information of the player when the tracking viewpoint is in the lost state.

  By doing in this way, viewpoint information in a lost state can be estimated based on viewpoint history information (viewpoint information) when the moving body has passed through the same area in the past in the circuit course. That is, the viewpoint information estimation process in the lost state as described with reference to FIGS. 10A to 10C can be realized.

  For example, in the cornering of the (N + 1) th cycle, if the speed of the moving body MV is high, there is a high possibility that the moving body MV travels along a route such as LC2 in FIG. Therefore, in this case, by selecting the viewpoint information of the viewpoint history information at the same time from the past viewpoint history information of the first to N-th laps at the same speed, at such a high speed. The viewpoint information when cornering can be estimated as the viewpoint information of the player in the lost state. Therefore, more appropriate viewpoint information estimation processing can be realized.

  In the above description, the viewpoint information is selected based on the speed of the moving body. However, the viewpoint information may be selected based on the acceleration, angular velocity, angular acceleration, and the like of the moving body. Alternatively, the viewpoint information may be selected based on the player's operation information (for example, the angle at which the steering wheel is turned, the depression amount of the accelerator pedal or the brake pedal, etc.). That is, the player viewpoint information when the tracking viewpoint is in the lost state may be estimated based on the player operation history information.

2.4 Estimation Process Based on Movement Information of Movable Mechanism In the present embodiment, viewpoint information estimation processing when the tracking viewpoint is in a lost state is performed based on the movement information of the movable mechanism. FIG. 14 shows a flowchart of viewpoint information estimation processing based on the operation information of such a movable mechanism.

  First, player operation information is acquired (step S21). Then, based on the acquired player operation information and game situation information, the movable mechanism 60 is operated (step S22). For example, when the player operates the handle 50, the brake pedal 52, the accelerator pedal 54, or the shift lever 56 of FIG. 1, the movable mechanism 60 is operated based on the operation information. Alternatively, the movable mechanism 60 is operated so as to correspond to the game situation. For example, in the game process, as the game progresses, a process that changes (updates) a game parameter that represents a game situation is performed, and the movable mechanism 60 is operated based on information about the game parameter.

  Specifically, in FIG. 1, when the player turns the handle 50 to the right (when turned to the right), the movable mechanism 60 is operated so that the seat 10 (seat portion) rotates clockwise. Specifically, the electric cylinders 73 and 74 are operated so that the rod portion of the electric cylinder 73 in FIG. 3 is long and the rod portion of the electric cylinder 74 is short. Thus, for example, it is possible to give the player a feeling that the car tail drifts to the left and the car drifts by cornering at the right corner.

  On the other hand, when the player turns the handle 50 to the left (when turned to the left), the movable mechanism 60 is operated so that the seat 10 rotates counterclockwise. Specifically, the electric cylinders 73 and 74 are operated so that the rod portion of the electric cylinder 73 becomes shorter and the rod portion of the electric cylinder 74 becomes longer. Thereby, for example, the tail of the car flows in the right direction, and it is possible to give the player a feeling that the car has drifted by cornering at the left corner.

  Alternatively, when the running situation that is one of the game situation information is a running situation that runs on an uneven road surface, the movable mechanism 60 is operated so that the seat 10 is slightly shaken. Specifically, the rod portions of the electric cylinders 73 and 74 are linearly moved with a minute stroke distance. Thereby, it is possible to give the player a feeling as if the vehicle is traveling on a rough road with unevenness.

  In FIG. 14, it is determined whether or not the tracking viewpoint is in the lost state (step S23). If it is determined that the tracking viewpoint is in the lost state, the player in the lost state is determined based on the operation information of the movable mechanism 60. The viewpoint information is estimated (step S24).

  For example, in FIG. 15A, the direction of the seat 10 when the seat 10 is stopped is CL, the viewpoint position of the player is VP, and the line-of-sight direction is VL. In this case, in FIG. 15B, for example, the player turns the handle 50 of FIG. 1 to the right, so that the seat 10 rotates clockwise (clockwise when the direction facing the player is the front direction) The movable mechanism 60 is operating. In FIG. 15C, for example, since the player has turned the handle 50 to the left, the movable mechanism 60 is operated so that the seat 10 rotates counterclockwise.

  In this embodiment, the player's viewpoint position VP and line-of-sight direction VL when the movable mechanism 60 is operated as shown in FIGS. 15B and 15C are measured in advance using viewpoint tracking or the like. . Then, the viewpoint information such as the measured viewpoint position VP and the line-of-sight direction VL is stored in association with the operation information of the movable mechanism 60. For example, it is stored in the storage unit 170 of FIG. For example, for each operation information such as each rotation angle of the seat 10 (operation information of the movable object in a broad sense) or each movement amount of the rod portion of the electric cylinders 73 and 74 (operation information of the actuator in a broad sense), Measurement results (or simulation results) of viewpoint information corresponding to each operation information are stored in association with each other. When the tracking viewpoint becomes lost, the viewpoint information associated with the operation information (operation status) of the movable mechanism 60 at that time is read from the storage unit 170, and the read viewpoint information is converted into the lost viewpoint information. Estimated as the viewpoint information of the player in the state.

  For example, when the tracking viewpoint is in a lost state when the seat 10 is rotated clockwise as shown in FIG. 15B, the measurement result (simulation result) at this rotation angle (in the rotation state). Is read from the storage unit 170 and is estimated as the viewpoint information of the player in the lost state. Similarly, when the tracking viewpoint is in the lost state when the seat 10 is rotated counterclockwise as shown in FIG. 15C, the measurement result (simulation result) at this rotation angle (in the rotation state). ) Is read from the storage unit 170 and estimated as the player's viewpoint information in the lost state.

  In this way, when the tracking viewpoint is in the lost state, the player's viewpoint can be set to the viewpoint according to the operating state of the movable mechanism 60 at that time. Therefore, even when the tracking viewpoint is in a lost state, it is possible to generate an image that is less uncomfortable and looks more natural and can be displayed on the player.

2.5 Estimation Processing Based on Player Information In this embodiment, viewpoint information estimation processing in a lost tracking viewpoint state is performed based on player information including at least one of the player's physique information, gender information, and age information. May be. Specifically, the process of selecting viewpoint information based on the player information from among a plurality of candidate viewpoint information is performed as the viewpoint information estimation process in the lost state. FIG. 16 shows a flowchart of viewpoint information estimation processing based on such player information.

  First, player information such as the player's physique, sex, and age is acquired (step S31). This can be acquired, for example, when the player inputs player information using the operation unit 160 of FIG. Alternatively, the player information may be acquired using an IC card or the like as the auxiliary storage device 194. For example, when the player touches the IC card with the card reader 58 of FIG. 1, player information associated with the IC card is acquired. Specifically, the medium ID of the IC card is sent to a server system (not shown) via a network, and player information (user information, user information, and the like) stored in association with the medium ID from the storage unit (database) of the server system. Personal information) is read out. The read player information is received by the image generation system (game system) via the network, and the player information is acquired. Alternatively, the player information itself may be stored in an IC card (an auxiliary storage device in a broad sense). In addition, there is information that can be acquired by the photographing apparatus without the player inputting it himself. For example, information such as “the height from the ground to the viewpoint in the basic posture when the player sits on the seat” is often not accurately known to the player himself / herself. However, the information is acquired to a certain degree of accuracy by a method such as “return the seat position to a fixed position every time before playing, and acquire the viewpoint position when the player sits there for the first time and the posture is stable”. be able to.

  Then, it is determined whether or not the tracking viewpoint is in a lost state (step S32). If it is determined that the tracking viewpoint is in a lost state, a plurality of pieces of candidate viewpoint information acquired based on operation information of the movable mechanism 60 and the like are determined. Viewpoint information corresponding to the player information is selected (step S33). Then, the selected viewpoint information is estimated as the viewpoint information of the player when the tracking viewpoint is in the lost state (step S34).

  For example, when estimating the player's viewpoint information in the lost state based on the operation information of the movable mechanism 60 as shown in FIGS. 15 (A) to 15 (C) described above, FIGS. A plurality of candidate viewpoint information is prepared as viewpoint information (VP, VL) of C). For example, viewpoint information for a person with a high height (for a person with a large physique) and viewpoint information for a person with a short height (for a person with a small physique) are prepared. For example, the viewpoint position VP is high for a person who is tall, and the viewpoint position VP is low for a person who is short.

  When it is determined that the height of the player is high based on the physique information of the player information, for example, a viewpoint for a person with high height is selected from candidate viewpoint information associated with the operation information of the movable mechanism 60. Information is selected as viewpoint information when the tracking viewpoint is in a lost state. On the other hand, when it is determined that the height of the player is low based on the physique information of the player information, for example, a viewpoint for a person with a short height is selected from candidate viewpoint information associated with the operation information of the movable mechanism 60. Information is selected as viewpoint information when the tracking viewpoint is in a lost state.

  In this way, when the tracking viewpoint is in a lost state, it is possible to select an appropriate viewpoint information corresponding to the physique such as the height of the player, generate an image, and display it on the player.

  As viewpoint information (VP, VL) in FIGS. 15B and 15C, for example, viewpoint information for men and adults and viewpoint information for women and children are prepared. For example, when the seat 10 is rotated as shown in FIGS. 15B and 15C, it is expected that the change in the line-of-sight direction VL is small for men and adults, and the change in the line-of-sight direction VL is expected for women and children. Is expected to be large.

  Then, when it is determined that the player is a male or an adult based on the gender information or the age information of the player information, for example, from the candidate viewpoint information associated with the operation information of the movable mechanism 60, the male Or adult viewpoint information is selected as the viewpoint information when the tracking viewpoint is in a lost state. On the other hand, when it is determined that the player is a woman or a child based on the gender information or age information of the player information, for example, from among the candidate viewpoint information associated with the operation information of the movable mechanism 60, Viewpoint information for children and children is selected as viewpoint information when the tracking viewpoint is in a lost state.

  For example, when preparing male or adult viewpoint information, the movable mechanism 60 is caused to perform operations such as actually sitting on the seat 10 by a male or adult and rotating the seat 10. Then, the viewpoint information measured at that time may be stored in the storage unit 170 in association with the operation information of the movable mechanism 60. Similarly, when preparing viewpoint information for women or children, the movable mechanism 60 is caused to perform operations such as actually sitting on the seat 10 by a woman or child and rotating the seat 10. Then, the viewpoint information measured at that time may be stored in the storage unit 170 in association with the operation information of the movable mechanism 60.

  In this way, when the tracking viewpoint is in a lost state, it is possible to select appropriate viewpoint information according to the gender and age of the player, generate an image, and display it on the player.

  In the above description, the case where the viewpoint information is selected from the plurality of candidate viewpoint information based on the operation information of the movable mechanism 60 has been described. However, the present embodiment is not limited to this. For example, based on movement information (position, velocity, acceleration, angular velocity, angular acceleration, etc.) of a moving body, player operation information, or game situation information, viewpoint information corresponding to player information is selected from a plurality of candidate viewpoint information Thus, the viewpoint information of the player when the tracking viewpoint is in the lost state may be estimated. For example, when the viewpoint information is selected by the method described with reference to FIGS. 10A to 13, the viewpoint information may be estimated in consideration of the player information, and the viewpoint information of the player in the lost state may be estimated.

2.6 Estimation Processing Based on Results of Physical Simulation Processing In this embodiment, viewpoint information estimation processing when the tracking viewpoint is in a lost state may be performed based on the results of physical simulation processing of player operations. FIG. 17 shows a flowchart of viewpoint information estimation processing based on the result of such physical simulation processing.

  First, player operation information is acquired (step S41). Then, the movable mechanism 60 is operated based on the player operation information and game situation information (step S42). For example, as described above, the movable mechanism 60 is operated based on operation information on the handle 50 and the like. Alternatively, the movable mechanism 60 is operated according to a game situation such as a road surface situation.

  Next, a physical simulation process of the movement of the model object of the player is executed based on the movement information of the movable mechanism 60 (step S43). For example, information on the force acting on the player is specified based on the operation information of the movable mechanism 60 and the like, and a physical simulation process such as a dynamic calculation using the information on the force is executed.

  Then, it is determined whether or not the tracking viewpoint is in the lost state (step S44). If it is determined that the tracking viewpoint is in the lost state, the viewpoint information of the model object obtained based on the physical simulation process is used as the tracking viewpoint lost. Estimated as viewpoint information of the player at the time (step S45). For example, in step S43, viewpoint information such as the viewpoint position and the line-of-sight direction of the model object is obtained at any time by performing a physical simulation process of the movement of the model object of the player based on the movement information of the movable mechanism 60 and the like. ing. When it is determined that the tracking viewpoint is in the lost state, the viewpoint information of the model object is estimated to be the viewpoint information of the player, and an image in the lost state of the tracking viewpoint is generated.

  For example, as shown in FIG. 18A, the physical simulation process is executed in a state where the model object MB that models the player is seated on the sheet object SB that models the seat 10. For example, when the sheet 10 is rotated by the movement of the movable mechanism 60, the sheet object SB corresponding to the sheet 10 is turned clockwise or counterclockwise as shown in FIGS. Rotate around. Then, a physical simulation process for obtaining the motion information of the model object MB when the sheet object SB rotates in this way is executed. By obtaining the motion information of the model object MB in this way, the viewpoint position VP and the line-of-sight direction VL of the model object MB when the sheet object SB rotates can be specified as the viewpoint information of the model object MB. The identified viewpoint information is estimated as the viewpoint information of the player when the tracking viewpoint is in the lost state.

  When performing such a physical simulation process, it is desirable to use the player information described with reference to FIG. For example, based on the physique information, gender information or age information of the player information, a player model object used for the physical simulation process is selected from a plurality of candidate model objects. Then, a physical simulation process is executed using the selected model object, and the viewpoint information of the player is obtained.

  In this way, it is possible to execute physical simulation processing according to the player's physique, gender, age, etc., so even when the tracking viewpoint is lost, select appropriate viewpoint information according to the player's physique, gender, and age. Thus, an image can be generated and displayed on the player.

  In the above description, the case where the viewpoint information is obtained by performing the physical simulation process based on the operation information of the movable mechanism 60 has been described, but the present embodiment is not limited to this. For example, the viewpoint information may be obtained by performing physical simulation processing based on movement information (position, velocity, acceleration, angular velocity, angular acceleration, etc.) of the moving object, player operation information, or game situation information.

  Various processes can be assumed as the physical simulation process of the present embodiment. For example, a model object is composed of a plurality of parts (bones and joints), and a force vector acting on the model object is obtained by operating the movable mechanism 60. Then, a dynamic calculation for obtaining the motion of each part of a plurality of parts (head, chest, waist, leg, hand) by applying the obtained force vector to the model object is performed as a physical simulation process. Then, viewpoint information such as the viewpoint position and line-of-sight direction of the model object is obtained based on the obtained movement of the part such as the head, and is estimated as the viewpoint information of the player when the tracking viewpoint is in the lost state. The physical simulation processing method is disclosed in, for example, Japanese Unexamined Patent Application Publication Nos. 2009-140519, 2004-30502, and 2009-37572. For example, as an example of a virtual human simulation method, a character is approximated by a single rigid model, and dynamic calculation is performed when an external force is applied to the single rigid model. In addition, key frames representing character movement are arranged in a multi-dimensional space by a technique called multi-dimensional key frame animation. Then, multidimensional coordinates are determined according to the state of the single rigid model of the character. Furthermore, the final posture / movement of the character is determined in consideration of human-like behaviors such as trying to maintain the posture or balancing when an external force is applied.

  Thus, various processes can be assumed as the physical simulation process used in the present embodiment.

3. Image Generation Methods Next, various image generation methods to which the method of this embodiment can be applied will be described.

3.1 Generation of Projected Image First, a case where an image generated by the image generation system of the present embodiment is a projected image projected on a projection screen will be described. In the following, the case where the projection screen is a curved screen (dome-shaped screen) will be mainly described as an example, but the present embodiment is not limited to this. A projection screen refers to any screen that is not a single plane. For example, the projection screen may be a screen composed of one curved surface or a plurality of surfaces (plane, curved surface). That is, as the screen, a screen configured by one curved surface, a screen configured by a plurality of planes, a screen including a curved surface portion and a plane portion, and the like can be assumed. Further, in the case of a screen that is a single plane, it can be done by a simpler process as in the case of a flat panel display that is not a projection type.

3.1.1 Distortion Correction Method in Pixel Unit First, a method for generating a projection image for projection will be described. For example, when the projection screen is a screen composed of one curved surface or a plurality of surfaces, in this embodiment, a distortion correction process based on the shape information of the projection screen is performed to generate a projection image. Hereinafter, a specific example of a distortion correction method performed when generating a projection image will be described.

  When projecting an image onto a dome-shaped (curved surface) screen, if the position of the projection device and the position of the player (observer) are separated from each other, image distortion becomes conspicuous. Therefore, by taking this distortion into consideration in advance and generating a projection image from the projection apparatus (an image drawn in the drawing buffer of the projection apparatus), it is possible to present an image without distortion as viewed from the player.

  In this case, if the screen is a single plane, the distortion becomes linear (distortion due to the perspective), and thus correction can be easily performed by using only one projective transformation matrix.

  However, in the case of a screen that is not a single plane (a screen composed of one curved surface or a plurality of surfaces), distortion that is not linear is added, so a simple correction method that uses only one projection transformation matrix is not possible. The distortion cannot be corrected, and it is necessary to perform fine correction according to the portion of the image.

  As a technique for realizing such distortion correction, a technique for performing distortion correction in units of pixels of the drawing buffer and a technique for performing distortion correction in units of vertices of a three-dimensional object are conceivable. First, a distortion correction method for each pixel will be described.

In the distortion correction method for each pixel of the drawing buffer, the processes (1), (2), (3), and (4) shown in FIG.
(1) The pixel PX (XP, YP) on the drawing buffer (frame buffer) obtains a light ray RY emitted through the lens of the projection device.
(2) The position of the intersection PS (XS, YS, ZS) where the ray RY intersects the screen SC is obtained. For example, when the screen SC is represented by an equation such as an elliptical equation, the intersection PS is obtained using an equation representing the straight line RY and an equation representing the screen SC. A mathematical expression representing the screen SC is shape information of the screen SC.
(3) The color of the intersection PS (XS, YS, ZS) needs to be a color when the player (observer) observes the object space (virtual space). Therefore, a straight line LV connecting the position VP of the virtual camera VC corresponding to the player's viewpoint and the position of the intersection PS (XS, YS, ZS) is obtained.
(4) Using the straight line LV as the line of sight of the virtual camera VC, the color of the pixel of the projected image on the drawing buffer is determined from the information in the object space. For example, the position of the point PB (XB, YB, ZB) on the three-dimensional object OB that reaches first in the object space is obtained, and the color of the pixel of the projection image on the drawing buffer is determined accordingly.

  In this case, in the improvement method of the distortion correction method in pixel units, instead of obtaining the color of the intersection PB (XB, YB, ZB) between the straight line LV and the three-dimensional object OB in the last (4), The color of the pixel of the projection image is determined by using the color of the intersection (render texture coordinates (U, V)) between the plane (render texture) drawn in advance and the straight line LV. A render texture can be created by selecting a plane as close as possible to the projection plane (hereinafter referred to as a proxy plane) in advance and performing a normal drawing method, that is, drawing with the plane as the projection plane.

  FIG. 20 shows an example of such proxy planes PL1 and PL2. Thus, the number of proxy planes is not limited to one, and in this case, two are used. Point PP is the intersection of straight line LV and proxy plane PL1 (PL2).

  The reference position of the render texture need only be calculated once, as long as the viewpoint, the position of the projection device, and the position and shape of the screen do not change. As a typical method for storing the data, for each pixel of the drawing buffer, a pixel value (texel value) at which position (U, V) of the render texture is referred to one sheet. There is a method to keep it as a texture. This texture is called a “UV map”. FIG. 21 shows the relationship among the drawing buffer, UV map, and render texture of the projection apparatus.

  Obtaining and storing the corresponding points (UV coordinates) of the render texture of FIG. 21 for all the pixels on the drawing buffer requires a large resource. If there is no room to prepare such a large resource, the corresponding point (XP, YP) on the drawing buffer and the corresponding point (U, V) on the render texture for the representative vertex appropriately arranged on the screen SC. And make a mesh with triangular polygons connecting them. When the triangle polygon is drawn, the (U, V) coordinates recorded as information at the vertices of the triangle polygon are referred to, and the (U, V) coordinates interpolated therefrom are used for the points inside the triangle polygon. Like that. In this way, resources can be saved significantly.

3.1.2 Distortion Correction Method for Each Vertex Next, a method for correcting distortion for each vertex of a drawing object will be described. In this method, the vertex of the three-dimensional object in the object space is converted into a point on the drawing buffer of the projection apparatus. Specifically, the processes (1), (2), (3), and (4) shown in FIG. This corresponds to performing the processing of the method of FIG. 19 in the reverse order.
(1) A straight line LV connecting the position of the vertex V (XV, YV, ZV) of the three-dimensional object OB in the object space and the position VP of the virtual camera VC corresponding to the viewpoint of the player is obtained.
(2) The position of the intersection PS (XS, YS, ZS) between the obtained straight line LV and the screen SC is obtained. For example, when the screen SC is represented by an equation such as an equation of an ellipsoid, the intersection PS is obtained using an equation representing the straight line LV and an equation representing the screen SC. A mathematical expression representing the screen SC is shape information of the screen SC.
(3) A straight line LR connecting the position of the intersection PS (XS, YS, ZS) and the position of the projection device PJ is obtained.
(4) The position of the point PD (XD, YD) on the drawing buffer corresponding to the straight line LR is obtained. This point PD corresponds to the vertex of the drawing object OBD on the drawing buffer corresponding to the three-dimensional object OB. When obtaining the point PD from the straight line LR, information on the optical system such as the lens characteristics and arrangement of the projection device PJ is used.

  Thereafter, a projection image can be generated on the drawing buffer by performing a rasterizing process for connecting the vertices of the drawing object OBD and painting colors.

  In this case, it is desirable to use the following method. That is, vertex division processing is performed on the three-dimensional object OB in the object space with the vertex division number set by the vertex division number setting unit. Then, an intersection position between the straight line connecting the vertex position of the three-dimensional object OB after the vertex division process and the position of the virtual camera VC and the projection screen SC is obtained. Then, based on the obtained intersection position, the vertex position in the drawing buffer for the drawing object OBD corresponding to the three-dimensional object OB is obtained. Then, based on the obtained vertex position of the drawing object OBD, the drawing object OBD is drawn in the drawing buffer to generate a projection image for projection. By adopting such a method, for example, a straight line in the object space is drawn as a straight line on the drawing buffer, and as a result, a situation in which a distorted image is observed can be suppressed.

  As described above, in a system that generates a projection image on a projection screen, setting the viewpoint position is important. For example, in FIGS. 19 and 22, if the position of the viewpoint VP is not accurately specified, the distorted image correction process becomes inaccurate, and an unnatural image is generated.

  In this regard, in the method of the comparative example, a representative viewpoint RVP as shown in FIG. 5A is set, and the representative viewpoint RVP is regarded as the viewpoint of the player, and distortion correction processing is performed to generate a projection image. It was.

  However, the method of this comparative example is based on the premise that there is little change in the player's viewpoint position. Therefore, for example, when the movable mechanism 60 as shown in FIG. 3 is provided in the image generation system, the position of the player's viewpoint in the real world is changed by the operation of the movable mechanism 60, and the representative viewpoint in FIG. The position of RVP will not match. That is, the position of the viewpoint VP used for the distortion correction processing in FIGS. 19 and 21 does not match the viewpoint of the player in the real world. For this reason, for example, in a situation where the movable mechanism 60 operates and the viewpoint position and the line-of-sight direction of the player change greatly, it is unnatural if a projection image is generated by the technique using the representative viewpoint RVP as shown in FIG. An image with a large sense of discomfort is displayed.

  Accordingly, in the present embodiment, viewpoint tracking as shown in FIGS. 6A to 7B is performed to capture the viewpoint of the player in the real world, and the viewpoint obtained by viewpoint tracking is changed to FIG. , Distortion correction processing for generating a projection image is performed by setting the viewpoint VP in FIG. At this time, the UV map required in the case of the distortion correction method in units of pixels shown in the above 3.1.1 is created in real time according to the change in the viewpoint position. By doing so, compared with the method using the representative viewpoint RVP in FIG. 5A, the difference between the viewpoint used for the distortion correction processing and the viewpoint of the player in the real world is reduced, and more natural images are displayed to the player. become able to.

  However, in the viewpoint tracking method, when viewpoint tracking is successful, appropriate viewpoint information can be acquired and an image with less sense of incongruity can be generated. However, in the situation of FIGS. 8B and 8C, etc. When the viewpoint tracking fails due to the occurrence and the tracking viewpoint is in a lost state, there is a problem that appropriate viewpoint information cannot be acquired.

  In this regard, in the present embodiment, as described with reference to FIGS. 9 to 18C, when the tracking viewpoint is in a lost state, viewpoint history information, movement information of the moving body, operation information, operation information of the movable mechanism, or player information A method for estimating the player's viewpoint information by effectively utilizing various information such as the above is employed. Therefore, it is possible to prevent a situation in which an uncomfortable image that is far from the viewpoint position of the real-world player is displayed on the player, and to provide an image generation system that can generate a more appropriate image. .

3.2 Stereoscopic image, head-mounted display device The method of the present embodiment is also effective in a system that generates a stereoscopic image. Specifically, by applying the method of the present embodiment to an image generation system that generates a first viewpoint image and a second viewpoint image for stereoscopic viewing based on the viewpoint information of the player, appropriate stereoscopic viewing is achieved. An image can be generated.

  Note that the method of this embodiment can also be applied to the generation of a stereoscopic projection image. For example, FIG. 23A to FIG. 23C are explanatory diagrams of a method for generating a stereoscopic projection image on the projection screen SC. For example, as shown in FIG. 23A, a left-eye virtual camera VCL (first viewpoint virtual camera in a broad sense) and a right-eye virtual camera VCR (second viewpoint virtual camera in a broad sense) are prepared. Then, by using each of the left-eye virtual camera VCL and the right-eye virtual camera VCR to generate a projection image by the method described with reference to FIGS. 19 to 22 and the like, a stereoscopic projection image can be generated.

  In this case, the positions of the left-eye virtual camera VCL and the right-eye virtual camera VCR can be set to detect the positions of the left eye and right eye of the player by viewpoint tracking or the like. Even in the conventional method (when the positions of the left eye and right eye of the player cannot be detected), the viewpoint position of the player can be estimated to some extent according to the projection position of the three-dimensional object to be rendered. Accordingly, the positions of the left-eye virtual camera VCL and the right-eye virtual camera VCR can be set accordingly. For example, as shown in FIGS. 23B and 23C, the positions of the left-eye virtual camera VCL and the right-eye virtual camera VCR according to the projection position D1 (or D2) of the three-dimensional object to be rendered ( By changing the (viewpoint position), an effect such as a head tracking could be created in a pseudo manner. However, when the movable mechanism 60 is provided in the image generation system, the operation of the movable mechanism 60 is also performed. The position of the player's viewpoint in the real world may change significantly, resulting in a display of an image that is unnatural or uncomfortable, as in the case of non-stereoscopic viewing. Therefore, by detecting the positions of the left eye and the right eye of the player by viewpoint tracking or the like, it is possible to present a stereoscopic image that is more natural than before.

  In such a system for generating a stereoscopic image, when a tracking viewpoint lost state occurs, there is a possibility that a failure such as displaying a broken stereoscopic image may occur. In this regard, in the present embodiment, even when the tracking viewpoint is in a lost state, the viewpoint information of the player is estimated based on various information, and a stereoscopic image is generated based on the estimated viewpoint information. Therefore, it is possible to prevent a situation in which a broken stereoscopic image is generated, and it is possible to provide an image generation system suitable for a stereoscopic image.

  The method of the present embodiment is also effective for application to a display image generation method of a head-mounted display device (HMD).

  For example, in FIG. 24, the HMD 200 is provided with a plurality of LEDs 231 to 236. The LEDs 231 to 234 are provided on the front side of the player, and the LED 235 and the LED 236 (not shown) are provided on the back side of the player. These LEDs 231 to 236 emit (emit) light in a visible light band, for example. Specifically, the LEDs 231 to 236 emit light of different colors. Then, the light of these LEDs 231 to 236 is imaged by an imaging unit installed on the front side of the player. That is, the spot light of these LEDs 231 to 236 appears in the captured image of the imaging unit. Then, by performing image processing of the captured image, the position (three-dimensional position, movement) of the player's head (HMD) is detected.

  For example, an imaging unit 40 as shown in FIG. 6A is used as the imaging unit. The imaging unit 40 is provided with first and second cameras 45 and 46. By using the first and second captured images of the first and second cameras 45 and 46, the player's The position of the head in the depth direction can be detected. Further, a motion sensor (not shown) is provided for the HMD 200, and the rotation angle (gaze direction) of the player's head can be detected based on the motion detection information of the motion sensor. Therefore, by using such an HMD 200, regardless of the direction of all 360 degrees around the player, the corresponding image in the virtual three-dimensional space (the virtual corresponding to the player's viewpoint). The image that can be seen from the camera) can be displayed on the display unit of the HMD 200. In addition, you may use infrared LED instead of visible light as LED231-236. Further, the position and movement of the player's head may be detected by other methods such as using a distance measuring sensor such as a depth camera.

  The HMD 200 is provided with a headband 260 and the like so that the player can stably wear the HMD 200 on the head with a better wearing feeling. In addition, the HMD 200 is provided with a headphone terminal (not shown). By connecting a headphone 270 to the headphone terminal, the player can play game sound that has been subjected to, for example, three-dimensional sound (three-dimensional audio) processing. It becomes possible to listen. Then, the player inputs the operation information by operating the operation unit 160 of FIG. 4 or performing a head movement operation or a head swing operation, and enjoys game play. The whirling movement and the swinging movement can be detected by a motion sensor of the HMD 200 or the like.

  Even in such an image generation system of the HMD 200, the viewpoint tracking of the player may be lost. That is, the occurrence of some situation makes it impossible to detect the player's viewpoint position and line-of-sight direction, and viewpoint tracking may become lost. For example, there is a possibility that the imaging unit cannot detect the light from the LEDs 231 to 236 in FIG. 24, or the gaze direction corresponding to the rotation angle of the player's head cannot be detected due to a temporary malfunction of the motion sensor. is there.

  When such a tracking viewpoint lost state occurs, an image based on incorrect tracking viewpoint information is displayed on the display unit of the HMD 200, and the player feels uncomfortable or unnatural, but displays an image. There is a fear. For example, although the player is actually facing the front direction, an image when facing the right direction or the left direction is displayed on the display unit of the HMD 200, or the viewpoint position of the player in the object space is assumed. There is a possibility that the situation is set such that the position is far from the viewpoint position.

  In this regard, in this embodiment, even when the tracking viewpoint in the HMD 200 is lost, various information such as viewpoint history information, movement information of the moving body, or operation information of the movable mechanism can be effectively used to obtain the player's information. A method for estimating viewpoint information can be employed. Therefore, it is possible to prevent a situation in which an uncomfortable image that is far from the detected viewpoint direction or line-of-sight position is displayed on the display unit of the HMD 200, and to generate a more appropriate image for HMD The system can be provided.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described at least once together with a different term having a broader meaning or the same meaning in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. In addition, a viewpoint tracking method, a viewpoint information estimation method, a movable mechanism operation control method, a physical simulation method, a projection image generation method, a stereoscopic image generation method, an HMD image generation method, and the like are also described in this embodiment. However, the present invention is not limited thereto, and techniques equivalent to these are also included in the scope of the present invention. The present invention can be applied to various games. Further, the present invention can be applied to various image generation systems such as an arcade game system and a large attraction system in which a large number of players participate.

VP viewpoint position, VL line-of-sight direction, VPHT1 to VPHTN viewpoint history information,
MV mobile, CS course, SC screen, PJ projector (projector),
VC virtual camera, OB object, OBD drawing object,
10 seats, 14 housing frames, 16 control boards, 20 screens,
30 projection device, 40 imaging unit, 45, 46 first and second cameras,
50 steering wheel, 52 brake pedal, 54 accelerator pedal,
56 shift lever, 58 card reader, 59 coin slot,
60 movable mechanism, 62 base, 63, 64, 65, 66 hinge part,
73, 74 Electric cylinder, 75, 76 support part,
80 Mounting plate, 83, 84 Hinge part,
100 processing unit, 102 game calculation unit, 104 object space setting unit,
106 moving body calculation unit, 108 viewpoint information calculation unit, 110 virtual camera control unit,
120 image generation unit, 130 sound generation unit, 160 operation unit, 170 storage unit,
172 Object data storage unit, 176 drawing buffer,
178 Course data storage unit, 179 map data storage unit, 180 information storage medium,
190 display unit, 192 sound output unit, 194 auxiliary storage device, 196 communication unit,
200 HMD, 231 to 236 LED, 260 headband,
270 headphones

Claims (16)

An object space setting unit for performing object space setting processing;
A viewpoint information calculation unit that sets tracking viewpoint information obtained by the viewpoint tracking of the player as the viewpoint information of the player;
An image generation unit that generates an image that is visible from the viewpoint of the player in the object space based on the viewpoint information;
Including
The viewpoint information calculation unit includes:
An image generation system that performs estimation processing of the viewpoint information of a player when a tracking viewpoint is in a lost state. In claim 1,
The viewpoint information calculation unit includes:
An image generation system, wherein the tracking viewpoint information obtained from a captured image obtained by photographing a player is set as the viewpoint information of the player. In claim 1 or 2,
The viewpoint information calculation unit includes:
An image generation system that performs estimation processing of the viewpoint information when the tracking viewpoint is in a lost state, based on viewpoint history information of a player. In claim 3,
The viewpoint information calculation unit includes:
The viewpoint information is estimated when the tracking viewpoint is lost based on at least one of the viewpoint history information, movement information of a moving object operated by the player, and operation information of the player. Generation system. In claim 3 or 4,
The viewpoint information calculation unit includes:
When the moving body operated by the player moves on a course or a map, the viewpoint information is estimated when the tracking viewpoint is lost based on the viewpoint history information when the player is located in the same area in the past. An image generation system characterized by the above. In claim 5,
The course is a circuit course,
The viewpoint information calculation unit includes:
The viewpoint information is estimated when the tracking viewpoint is in a lost state based on the viewpoint history information when the moving body operated by the player has passed through the same area in the past in the circuit course. Image generation system. In any one of Claims 1 thru | or 6.
Including a movable mechanism,
The viewpoint information calculation unit includes:
An image generation system that performs an estimation process of the viewpoint information when the tracking viewpoint is in a lost state based on operation information of the movable mechanism. In claim 7,
The viewpoint information calculation unit includes:
The movement information of the movable mechanism is obtained based on player operation information or game situation information, and the viewpoint information is estimated when the tracking viewpoint is lost based on the movement information. Image generation system. In any one of Claims 1 thru | or 8.
The viewpoint information calculation unit includes:
An image generation system for performing estimation processing of the viewpoint information when the tracking viewpoint is in a lost state based on player information including at least one of physique information, gender information, and age information of the player. In claim 9,
The viewpoint information calculation unit includes:
An image generation system characterized in that a process of selecting viewpoint information from a plurality of candidate viewpoint information based on the player information is performed as an estimation process of the viewpoint information when the tracking viewpoint is in a lost state. In any one of Claims 1 thru | or 10.
The viewpoint information calculation unit includes:
An image generation system, wherein the viewpoint information is estimated when the tracking viewpoint is in a lost state based on a result of a physical simulation process of a player operation. In any one of Claims 1 thru | or 11,
The image generation unit
An image generation system that generates a projection image based on shape information of a projection screen constituted by one curved surface or a plurality of surfaces and the viewpoint information of a player. In claim 12,
The image generation unit
The pixel on the drawing buffer, with a line connecting the intersection position of the light beam emitted from the projection system optical system and the projection screen and the viewpoint position of the player as a virtual camera line of sight An image generation system characterized by determining a color. In claim 12,
The image generation unit
Based on the intersection position of the projection screen and the straight line connecting the vertex position of the object in the object space and the viewpoint position of the player, the vertex position on the drawing buffer for the drawing object corresponding to the object is obtained, An image generation system for drawing the drawing object in the drawing buffer based on the vertex position. In any one of Claims 1 thru | or 14.
The image generation unit
An image generation system for generating a first viewpoint image and a second viewpoint image for stereoscopic viewing based on the viewpoint information of the player. An object space setting unit for performing object space setting processing;
A viewpoint information calculation unit that sets tracking viewpoint information obtained by the viewpoint tracking of the player as the viewpoint information of the player;
As an image generation unit that generates an image that can be seen from the viewpoint of the player in the object space based on the viewpoint information,
Make the computer work,
The viewpoint information calculation unit includes:
A program for performing estimation processing of the viewpoint information of a player when a tracking viewpoint is in a lost state.