JP2015231445A - Program and image generating device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To achieve a new technique for improving usability when achieving gesture input using an HMD.SOLUTION: In a scene where gesture input is required, a body 24 to be selected correlated with operation contents of the gesture input is displayed on a display screen of an HMD while being fixed or nearly fixed to a predetermined position in a visual field. The body 24 to be selected is re-displayed by remaining in the visual field even if players 2 shake their heads. Thus, selection contents and targets of the gesture input are not missed.

Description

  The present invention relates to a program for causing a computer to control display.

  A head mounted display (hereinafter referred to as “HMD”), which is a head-mounted display, is known as a key device for providing a realistic and powerful video experience. For example, techniques employed for attractions such as amusement parks and video games are known (see, for example, Patent Documents 1 to 3).

  A technique for recognizing a user's gesture while using an HMD is also known (see, for example, Patent Document 4).

Japanese Patent Laid-Open No. 5-305181 JP 2000-210468 A JP 2003-125313 A JP 2012-181809 A

  An object of the present invention is to realize a new technique for improving usability when realizing gesture input using an HMD.

A first invention for solving the above-described problems includes an HMD (Head Mounted Display) worn on a user's head and a pointing device held by the user's hand or the user (hereinafter collectively referred to as “operation instruction”). A computer connected to imaging means (for example, a distance measuring sensor unit 1100 and a color image sensor 1104 in FIG. 1) installed to take an image of a real space including the HMD.
Calculation means for calculating a virtual position corresponding to the position of the operation instruction in the real space in a virtual space that is a display image space of the HMD, using a photographed image of the imaging means (for example, the process of FIG. 9). Unit 200, game calculation unit 202, hand position calculation unit 214, step S4 in FIG.
Input determination object arrangement control means (for example, the player auxiliary control unit 220 and the selected object control unit in FIG. 9) that arranges the input determination object (for example, the selection object 24 in FIG. 6) at a given position in the virtual space. 228, stereoscopic image generation control unit 232, step S112 in FIG. 14),
Based on the virtual position and the arrangement position of the input determination body, input determination means for determining an input to the input determination body (for example, the player auxiliary control unit 220, the gesture input determination unit 238 in FIG. 9, and FIG. 14). Steps S170 to S172),
It is a program to make it function as.

  The input determination body is a display body that is provided to the user as a standard for gestures when requesting gesture input. For example, a target position or a target area of the hand position, an operation start point to an end point, and an operation trajectory connecting these two points, a guide, or the like. In the case of displaying in the sense of indicating the target position and target area, from the user's standpoint, the input determination object appears to be the object of the selection operation and can be called “selected object”. Of course, depending on the situation where the gesture input is requested, it may also serve as a character or background object appearing during the game.

  According to the first invention, the input determination body to which the content for gesture input is assigned can be displayed in the display screen of the HMD. If you are not familiar with gesture input, it is difficult to understand how you can make the desired input on the spot, but the user can use the input judgment body as a guide for gesture input, so it will be requested smoothly at that time Will be able to input gestures. That is, it is possible to improve usability when realizing gesture input using the HMD.

The second invention is an HMD position / orientation determination means for determining the position and / or orientation of the HMD in the real space (for example, the HMD position / orientation calculation unit 210 in FIG. 9 and step S4 in FIG. 12),
Virtual space image generation means for generating an image of the virtual space whose field of view changes according to the position and / or orientation of the HMD determined by the HMD position and orientation determination means as the image of the virtual space to be displayed on the HMD (For example, the stereoscopic image generation control unit 232 in FIG. 9 and step S15 in FIG. 12), the computer is further functioned,
The input determination object arrangement control means is a program according to the first aspect of the invention that arranges the input determination object at a given position in the view following the change in the view.

  If the input determination object is displayed at a fixed position with respect to the virtual space, the input determination object will be out of sight when the user shakes his / her head. Depending on the game content, the input determination object may remain out of sight. Therefore, the user may forget that it is time to input an operation. For example, in FPS (First Person Shooting Game), such a situation may occur. However, according to the second invention, even if the user wears the HMD and shakes his / her head, the input determination body is displayed in the field of view of the HMD, so that there is no such worry.

  However, depending on the content of the game, it may be more convenient for the input determination body to be at a predetermined position in the virtual space. For example, it may be said that this is the case where a number or operation for unlocking a secret room key is input in an adventure game.

  Therefore, as a third invention, an HMD position / posture determination unit that determines the position and / or posture of the HMD in the real space, and an image of the virtual space that is displayed on the HMD are determined by the HMD position / posture determination unit. The computer further functions as a virtual space image generation unit that generates an image of the virtual space in which the field of view changes according to the position and / or posture of the HMD, and the input determination object arrangement control unit includes the field of view The program according to the first aspect of the present invention may be configured such that the position where the input determination body is arranged is a predetermined position in the virtual space, regardless of the change in the number.

  According to a fourth aspect of the invention, the virtual space image generation means generates an image in which a character (for example, the guide character 20 in FIG. 5) is placed in the virtual space, and the input determination body placement control means is a front input determination. It is a program of the 3rd invention which makes a body accompany a character.

  According to the fourth invention, for example, it is suitable for a scene where a response to a question from a character is input by gesture.

  According to a fifth aspect of the present invention, when the input determination object arrangement control means satisfies a given condition, the incident determination display of the input determination object is interrupted and the input determination object is positioned at a predetermined position in the field of view. The program of the fourth invention.

  According to the fifth aspect, since the input determination body is fixedly displayed (remaining display in the field of view) at a predetermined position in the field of view, the usability is excellent.

  For example, as a sixth invention, when the input determining body arrangement control means satisfies the given condition when the character is outside the field of view after being positioned within the field of view, It is also possible to configure the program of the invention.

  In this case, if it is applied on the condition that the character is once in the field of view, the input determination body is always in the field of view even if the user shakes his / her head and the character is out of the field of view (out of the screen). Therefore, it is not necessary to lose sight of the input judgment object and to remember that the input is requested.

According to a seventh aspect of the present invention, an instruction object (for example, the left hand object 14 or the right hand object 16 in FIG. 5) indicating that the virtual instruction position is a position corresponding to the position of the operation instruction object in the real space is provided in the HMD. Instruction object display control means (for example, hand object control unit 226 in FIG. 9, step S4 in FIG. 12, step S52 in FIG. 13) to be displayed on
As a program according to any one of the second to sixth inventions for causing the computer to function.

  According to the seventh aspect, the pointing object can be displayed at the position of the virtual space corresponding to the position of the operation pointing object in the real space. Therefore, it becomes easy to perform a gesture input such that the operation instruction object is brought close to the input determination body depending on the display of the instruction object.

  In an eighth aspect of the invention, the input determination means determines an input to the input determination body based on a relative positional relationship between the display position of the input determination body and the display position of the pointing object (for example, FIG. 14 Steps S170 to S172) are the programs of the sixth invention.

  According to the eighth invention, the user can input to the input determination object by bringing the operation instruction object close to the input determination object or directing the operation instruction object to the input determination object, thereby realizing excellent usability. .

  According to a ninth aspect of the present invention, when viewed from the HMD, when the virtual position is a hidden position in the virtual space, first image processing means (for example, FIG. 9) performs image processing so that the pointing object is exposed. 14 is a program according to the seventh or eighth invention for causing the computer to further function as the expression processing unit 230, step S122 in FIG.

  According to the ninth aspect, the visibility of the pointing object can be improved.

  According to a tenth aspect of the present invention, when viewed from the HMD, when the input determination body is in a hidden position in the virtual space, second image processing means (for example, performing image processing so that the input determination body is exposed) 9 is a program according to any one of the second to ninth inventions for causing the computer to further function as the expression processing unit 230 in FIG. 9 and step S120 in FIG.

  According to the tenth aspect, the visibility of the input determination body can be improved.

In an eleventh aspect of the invention, the virtual space image generation means generates an image in which characters are arranged in the virtual space,
Character target operation input determination means for determining operation input to the character based on the virtual position (for example, gesture input determination unit 238 in FIG. 9, step S8 in FIG. 12),
Means for invalidating the character target operation input determining means when the input determining body is displayed, and means for validating the character target operation input determining means when not displaying (for example, invalid character gesture input in FIG. 9) 13 is a program according to any one of the second to tenth inventions for causing the computer to function as the control unit 240, step S94 in FIG. 13, and step S174 in FIG.

  According to the eleventh aspect, when there is a gesture input target other than the input determination object in the virtual space, the input to the input determination object is disabled by invalidating the gesture input to other than the input determination object. It is possible to prioritize and prevent unintended input.

  With regard to the relationship between the coordinate system of the real space and the coordinate system of the virtual space, as a twelfth invention, a coordinate system initial setting means (for example, a diagram) for initializing the coordinate system of the virtual space to correspond to the real space. It is preferable that the program of any one of the first to eleventh inventions for further functioning the computer is configured as 16 coordinate system initial setting units 204).

  For example, as a thirteenth invention, the coordinate system initial setting means configures the program according to the twelfth invention, wherein the initial setting is executed with reference to the position of the operation instruction object in the captured image of the imaging means. This is more preferable because the accuracy of the operation instruction is improved.

  Further, the invention is not limited to the case where the relative positional relationship between the HMD and the operation instruction object is controlled in the same manner in the real space and the virtual space. As a fourteenth invention, the calculation means includes the HMD in the real space and the A program according to any one of the first to thirteenth inventions, comprising virtual position correcting means (for example, virtual position correcting unit 206 in FIG. 16) for correcting the virtual position based on a relative positional relationship with an operation instruction. It may be configured.

  Further, as a fifteenth aspect of the invention, when the relative position relationship based on the orientation of the HMD satisfies a predetermined correction condition, the virtual position correction unit follows a correction calculation associated with the correction condition. A program according to a fourteenth aspect of the invention for correcting the virtual position can be configured.

A sixteenth aspect of the invention is connected to an HMD mounted on a user's head and an imaging unit installed to photograph a real space including the operation instruction object and the HMD, and generates an image to be displayed on the HMD. An image generation apparatus for performing
A calculation unit that calculates a virtual position corresponding to the position of the operation instruction in the real space in a virtual space that is a display image space of the HMD, using the captured image of the imaging unit (for example, the control in FIG. 1). A substrate 1050, the processing unit 200 in FIG. 9, the game calculation unit 202, the hand position calculation unit 214, and step S4 in FIG.
Input determination object arrangement control means for arranging an input determination object at a given position in the virtual space (for example, control board 1050 in FIG. 1, player auxiliary control unit 220 in FIG. 9, selected object control unit 228, stereoscopic view) The image generation control unit 232, step S112 in FIG. 14, and
Based on the virtual position and the arrangement position of the input determination body, input determination means for determining an input to the input determination body (for example, the control board 1050 in FIG. 1, the player auxiliary control unit 220 in FIG. 9, gesture input) Determination unit 238, steps S170 to S172 in FIG.
Is an image generating apparatus.

  According to the sixteenth aspect, the same effect as in the first aspect can be obtained.

The figure which shows the structural example of a game system. The figure which shows the structural example of a game controller. The figure which shows an example of a structure of a headset. The figure explaining the coordinate system of ranging space and the element which measures and is recognized. The figure which shows an example of the virtual space for producing | generating the stereoscopic vision image (game image) displayed on HMD, and the object arrange | positioned in virtual space. (1) The figure which shows the example of the stereoscopic vision image (game image; game screen) displayed on HMD, and (2) the player in the real world at that time, and its surrounding state. (1) The figure which shows the example of the stereoscopic vision image (game image; game screen) displayed on HMD, and (2) the player in the real world at that time, and its surrounding state. (1) The figure which shows the example of the stereoscopic vision image (game image; game screen) displayed on HMD, and (2) the player in the real world at that time, and its surrounding state. The functional block diagram which shows the function structural example of a game system. The figure which shows the example of the program and data which a memory | storage part memorize | stores. The figure which shows the data structural example of virtual space control data. The flowchart for demonstrating the flow of a process in a game system. The flowchart for demonstrating the flow of a player assistance process. 14 is a flowchart continued from FIG. The figure which shows the modification of a structure of a game system. The figure which shows the modification of a function structure. The figure which shows the modification of the virtual space for producing | generating the stereoscopic vision image (game image) displayed on HMD, and the object arrange | positioned in virtual space.

FIG. 1 is a diagram illustrating an example of a configuration of a game system in the present embodiment.
A game system 1000 according to the present embodiment includes a game apparatus main body 1002, a touch panel 1004, a distance measuring sensor unit 1100, a game controller 1200, and a headset 1300 equipped with an HMD 1310. The HMD 1310 will be described as a non-transmissive (also referred to as an immersive) HMD, but may be a transflective or transmissive HMD.

  The game apparatus main body 1002 is a kind of computer and incorporates a control board 1050.

The control board 1050 is, for example,
1) Various microprocessors such as CPU (Central Processing Unit) 1051, GPU (Graphics Processing Unit), DSP (Digital Signal Processor),
2) Various IC memories 1052, such as VRAM, RAM, ROM,
3) A communication module 1053 for wireless or wired communication connection with peripheral devices such as the touch panel 1004, the distance measuring sensor unit 1100, the game controller 1200, and the headset 1300,
4) A driver circuit 1054 such as a touch panel 1004 is mounted.
Note that part or all of the control board 1050 may be configured by an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

  Then, the game apparatus main body 1002 reads the program and data stored in the IC memory 1052 by the control board 1050 and performs arithmetic processing, thereby controlling the game system 1000 in an integrated manner and executing a video game. it can.

  The touch panel 1004 is a video monitor equipped with a touch panel, and is disposed almost in front of the player 2.

  The distance measuring sensor unit 1100 is fixedly installed, for example, on the upper part of the touch panel 1004 so that the headset 1300 and the game controller 1200 are included in the shooting range in the front direction of the display surface of the monitor. Then, an image of the player 2 existing in front of the touch panel 1004 and surrounding objects can be taken, and the image data can be output to the game apparatus main body 1002. Further, by analyzing the image, the distance to the player 2 and surrounding objects can be calculated, and the distance information can be output to the game apparatus main body 1002. Note that the drawing in the drawing is a front view of the distance measuring sensor unit 1100.

  Specifically, the distance measuring sensor unit 1100 includes, for example, an infrared irradiator 1102 that irradiates infrared (IR) forward, a color image sensor 1104 that can capture an infrared reflected image from a front object, and infrared rays. One or a plurality of LSIs 1106 for analyzing the reflected image and calculating the relative distance to the object and performing image recognition processing are mounted. The color image sensor 1104 is a kind of imaging means.

The ranging sensor unit 1100 determines the representative point position in the captured image by recognizing each part of the body of the game controller 1200, the headset 1300, and the player 2 from the image captured by the color image sensor 1104. Then, the distance up to the representative point position can be measured. Such distance measurement can be realized by a known IR distance sensor technology.
It should be noted that the distance measuring technique employed in the distance measuring sensor unit 1100 is not limited to the IR type and can be employed as appropriate. For example, stereo shooting may be performed using the left and right color image sensors 1104, and the distance may be calculated from the parallax.

The distance measuring sensor unit 1100 is equipped with a known 6-axis sensor 1108.
The 6-axis sensor 1108 is a known sensor for motion tracing that detects movement in the front and rear (the optical axis direction of the color image sensor 1104), left and right, and up and down of the distance measuring sensor unit 1100, and rotation around each axis. Thus, for example, the acceleration in the three-axis direction and the angular velocity around the three axes are detected. Alternatively, the acceleration in the triaxial direction and the geomagnetic direction in the east, west, south, and north may be detected. From the detection result by the 6-axis sensor 1108, the direction of the depth direction axis (Z-axis) of the coordinate system of the ranging space can be known.

  The distance measuring sensor unit 1100 is appropriately equipped with a communication module for communicating with the game apparatus main body 1002, a battery, and the like.

FIG. 2 is a diagram illustrating a configuration example of the game controller 1200.
The game controller 1200 of the present embodiment is designed on the assumption that it is held and used with both hands. Of course, it may be designed on the premise of holding with one hand like a gun-type controller.
The game controller 1200 is provided with a light emitting marker 1202 on its outer surface. A speaker 1204, a vibrator 1206, a six-axis sensor 1208, a controller board 1250, a battery (not shown), and the like are incorporated.

  The light emitting marker 1202 is realized by, for example, an LED or the like, and light emission is controlled by the controller board 1250. Usually, it is always on.

  The 6-axis sensor 1208 detects motion of the game controller 1200 in the front and rear (X2 axis), left and right (Y2 axis), and up and down (Z2 axis) movements and rotations (Φc, θc, Ψc) around each axis. A well-known sensor for tracing, for example, detects acceleration in three axial directions and angular velocity around three axes. Alternatively, the acceleration in the triaxial direction and the geomagnetic direction in the east, west, south, and north may be detected.

  The controller board 1250 includes a CPU 1251, an IC memory 1252, a communication module 1253, an interface circuit 1254, and the like, and controls operation control of the game controller 1200 and communication with the outside. The interface circuit 1254 controls output of control signals to the speaker 1204 and the vibrator 1206 and input of signals from the operation button switch 1260, the direction input key 1262, the joystick 1264, and the like.

FIG. 3 is a diagram illustrating a configuration example of the headset 1300 in the present embodiment, and corresponds to (1) a front view and (2) a side view.
The headset 1300 is equipment that the player 2 wears on the head, provides image and sound information to the player 2 so as to be viewable, and detects the position and posture of the head for realizing head tracking. It has a function. For example, the headset 1300 includes headphones 1302, an HMD 1310 (Head Mounted Display), and a headset substrate 1350.

  The headphones 1302 are preferably so-called sealed stereo headphones.

The HMD 1310 can be realized by a non-transmissive / immersive head-mounted display whose viewing angle corresponds to a human viewing angle and its driver circuit.
For example, a communication IC that receives an image signal from the headset substrate 1350, an image display device and an optical element that display an image based on the received image signal to the eyes of the wearer with a visual angle equivalent to the naked eye, a six-axis sensor 1308, , An IC chip 1309 for calculating information on head acceleration and posture detected by the 6-axis sensor 1308, and a communication module 1352 for communication connection with the game apparatus main body 1002. Some of these may be included in the headset substrate 1350.

  The 6-axis sensor 1308 moves the HMD 1310, that is, the front and rear (X3 axis), left and right (Y3 axis), and upper and lower (Z3 axis) of the head of the player 2 and rotation (Φh, θh, Ψh) around each axis. This is a known sensor for detecting a motion trace, and detects, for example, acceleration in three axis directions and angular velocity around the three axes. Alternatively, the acceleration in the triaxial direction and the geomagnetic direction in the east, west, south, and north may be detected.

  The headset substrate 1350 is a control substrate that is connected to each part of the headset 1300 and performs various arithmetic processing in the headset and processing related to data communication control with the outside. For example, it can be realized by a CPU, an IC memory, an image processing LSI, a communication IC, an interface circuit, or the like.

  Further, light emitting markers 1312 are provided at three locations on the front surface of the HMD 1310. The light emitting marker 1312 is realized by, for example, an LED or the like, and light emission is controlled by the headset substrate 1350. Usually, it is always on. The light emitting marker 1312 is used as a marker for recognizing the presence and position of the HMD 1310 from an image captured by the color image sensor 1104 of the distance measuring sensor unit 1100.

FIG. 4 is a diagram for explaining the coordinate system of the distance measurement space and the elements to be measured / recognized.
In the coordinate system of the distance measurement space, the distance measurement sensor unit 1100 is the origin of coordinates, and the optical axis direction of the color image sensor 1104 is the depth axis (Z axis). The scale is set in the same way as in real space. Although the distance measurement space is also a real space, each axis direction of the coordinate system is defined according to the installation posture of the distance measurement sensor unit 1100. The installation posture of the distance measurement sensor unit 1100 is determined from the detection result of the 6-axis sensor 1108. Of course, since the horizontal direction and the vertical direction of the real space can be obtained from the detection result of the six-axis sensor 1109, the direction of the coordinate axis of the distance measuring space can also be set along the horizontal direction and the vertical direction.

  In the game system 1000 of the present embodiment, the position Ph of the HMD 1310, the position Pc of the game controller 1200, the right hand position Pr and the left hand position Pl of the player 2 in the distance measurement space by distance measurement by the distance measurement sensor unit 1100 are as necessary. (For example, with a predetermined period such as an interval of 100 ms).

  Specifically, the presence of the HMD 1310 is recognized by recognizing the light emitting marker 1312 of the HMD 1310 from the image taken by the color image sensor 1104 of the distance measuring sensor unit 1100, and the light emitting marker 1312 in the image coordinate system is captured. The position Ph of the HMD 1310 is determined from the current position and the distance measurement result of the position. The position Ph may be determined from the relative positional relationship, relative distance, and size of the light emitting marker 1312 with the distance measuring sensor unit 1100 as a reference. Similarly, for the game controller 1200, the position Pc of the game controller 1200 can be determined from the position where the light emitting marker 1202 appears in the image coordinate system and the distance measurement result at the position.

  Further, from the size / inclination / deformation of each of the plurality of light emitting markers 1312 in the image photographed by the color image sensor 1104 and the size / inclination / deformation of the figure formed by connecting the respective light emitting markers 1312, the orientation of the HMD 1310 ( [Phi] h, [theta] h, [Psi] h), in other words, the line-of-sight direction (front direction of the face) of the player wearing the HMD 1310 can be detected. Then, the posture at the position Ph and a change in the posture can be known by the 6-axis sensor 1308 of the HMD 1310. The same applies to the game controller 1200. From the size / tilt / deformation of each light emitting marker 1202 in the image taken by the color image sensor 1104, and the size / tilt / deformation of the figure formed by connecting each light emitting marker 1202 to the game. The attitude (Φc, θc, Ψc) of the controller 1200 can be detected. Further, the 6-axis sensor 1208 of the game controller 1200 can know the posture at the position Pc and the change in the posture.

  If the 6-axis sensor 1308 or 6-axis sensor 1208 includes a geomagnetic sensor or the like and can detect an attitude based on the geomagnetic direction, it can be used as correction information for attitude calculation using the light emitting marker 1312 and the light emitting marker 1202. The posture calculation can be realized without using the light emitting marker 1312 or the light emitting marker 1202.

  The left hand position Pl and the right hand position Pr, which are operation instructions at the time of gesture input, recognize from the image captured by the color image sensor 1104 an image of a portion where the left hand and the right hand are captured, and position coordinates of the image portion, It is determined from the distance measurement result relating to the position coordinates. Note that hand image recognition can be realized by appropriately using a known technique such as shape recognition.

  And the game system 1000 of this embodiment improves inconvenience at the time of using HMD with a home stationary game device based on these parameter values, and improves usability. Specifically, when the headset 1300 is worn and the game controller 1200 is not gripped, the game controller 1200 is relatively easy even when the headset 1300 is worn and the state of the outside world cannot be confirmed. Assisting the player by finding and guiding the player to hold it.

[Description of Principle]
One of the guides for easily finding the game controller 1200 and holding it in the hand is the guidance display by the HMD 1310.

  FIG. 5 is a diagram illustrating an example of a player auxiliary display virtual three-dimensional space for generating a stereoscopic image for player auxiliary display displayed on the HMD 1310 and objects arranged in the space.

  In this embodiment, the player auxiliary display virtual three-dimensional space is defined by a coordinate system in which coordinate axis directions are determined along the horizontal direction and the vertical direction of the real space, and the scale is set in the same manner as the real space. The coordinate origin is the same position as the origin of the distance measurement space. The player auxiliary display virtual three-dimensional space is an example of a virtual space, and is hereinafter abbreviated as “virtual space” as appropriate. In order to generate a game image to be displayed on the HMD 1310, a viewpoint coordinate system virtual space based on the virtual viewpoint for the HMD 1310 (virtual stereo camera 10) may be provided separately. In addition, the player auxiliary display virtual three-dimensional space may of course have a different scale from the real space, for example, when the player enjoys a game that enters a micro world such as an insect.

  In the player auxiliary display virtual three-dimensional space, a background object 4 such as a floor, a wall, and a ceiling, a virtual stereo camera 10, a controller object 12, a direction guide object 13, and a left-hand object are appropriately added to supplement the sense of direction of the space. 14 and the right hand object 16 are arranged. Further, when some kind of operation guidance display directed to the player 2 such as prompting an operation input is performed, the guidance character 20, the guidance display panel 22, and the selected object 24 are arranged.

  The virtual stereo camera 10 is a set of left and right virtual cameras for generating a stereoscopic image by the HMD 1310. The rendering image of the player auxiliary display virtual three-dimensional space photographed by the right camera of the set becomes an image for the right eye of the HMD 1310, and the rendered image photographed by the left camera becomes an image for the left eye.

  The virtual stereo camera 10 is disposed at a position Ph ′ of the HMD 1310 that is appropriately coordinate-converted from the distance measurement coordinate system to the coordinate system of the virtual space. The shooting angle of view of the virtual stereo camera 10 is set to the same level as the human eye, and the direction of the optical axis is the direction of the front-rear axis (X3 axis: FIG. 3) detected by the 6-axis sensor 1308 of the HMD 1310. The rotation around the optical axis is controlled to be the same as the rotation around the front-rear axis detected by the 6-axis sensor 1308. That is, the virtual stereo camera 10 is controlled to trace the movement of the head of the player 2. For such a function, a known head tracking technology related to the HMD can be used.

  The controller object 12 is an object (first form of the first guidance display) indicating the position where the game controller 1200 is present, and has a simplified design of the game controller 1200, for example. And it arrange | positions in the position Pc 'of the game controller 1200 by which coordinate conversion was suitably carried out from the ranging coordinate system to the coordinate system of virtual space. The size of the controller object 12 is the same as or similar to that of the real game controller 1200.

  The direction guidance object 13 is an object (second form of the first guidance display) that is controlled so as to point to the position of the controller object 12 in the virtual space. When the controller object 12 does not enter the field of view of the HMD 1310 (within the shooting angle of view), the arrangement position is automatically adjusted so that it is displayed at a predetermined position in the field of view of the HMD 1310. Specifically, the direction guidance object 13 is arranged and displayed at a position closest to the position PC ′ of the game controller 1200 on the periphery of the display screen of the HMD 1310 (see FIG. 6).

  The left hand object 14 and the right hand object 16 are pointing objects indicating positions in the virtual space corresponding to positions in the real space (ranging space) of the hand, which is an operation instruction for gesture input in the present embodiment. It is the 2nd guidance display for making it possible to take game controller 1200. Specifically, the left-hand object 14 and the right-hand object 16 are controlled to be placed at the left-hand position Pl ′ and the right-hand position Pr ′ of the player 2 that are appropriately coordinate-converted from the distance measurement coordinate system to the coordinate system of the virtual space, respectively. That is, the left hand object 14 and the right hand object 16 trace the movements of the left hand and the right hand of the player 2, respectively.

  The guide character 20, the guide display panel 22, and the selected object 24 are arranged on the back side as viewed from the virtual stereo camera 10, that is, in the front direction of the player 2.

Even if the position or line-of-sight direction of the virtual stereo camera 10 changes (that is, even if the head on which the headset 1300 is worn is moved), the arrangement positions of the guide character 20 and the guide display panel 22 in the virtual space do not change. . In particular, the guide character 20 can move in the virtual space, but the guide display panel 22 is arranged at a fixed position in the virtual space.
The guidance display panel 22 is, for example, a plate-like object that displays the purpose of the operation input or text or a diagram that prompts the operation input.

  The selection target 24 corresponds to, for example, a selection operation icon, and displays text or a figure indicating the associated options. The arrangement position of the selected object 24 is automatically controlled so as to be fixed or substantially fixed within the shooting angle of view of the virtual stereo camera 10 (that is, within the field of view seen by the HMD 1310) (details will be described later). The design of the selected object 24 can be set as appropriate, but in the present embodiment, the design corresponds to a “speech balloon” that displays the dialogue of the guide character 20.

6 to 8 are diagrams showing (1) an example of a stereoscopic image (game image; game screen) displayed on the HMD 1310, and (2) the player 2 in the real world at that time and the surrounding state. is there.
The images shown in FIGS. 6 to 8 are in a preparatory stage before the start of the game, and can be viewed with the player 2 wearing the headset 1300 with both hands and facing the touch panel 1004, that is, the front.

  Since the guide character 20, the guidance display panel 22, and the selected object 24 are initially arranged near the coordinate origin of the virtual space (see FIG. 5), a stereoscopic image (game screen) of the virtual stereo camera 10 as shown in FIG. ) These are displayed in front of the center of W2. Among these, the position in the virtual space is adjusted following the direction of the line of sight of the virtual stereo camera 10 so that the selected object 24 is always displayed at a predetermined position on the game screen (see FIG. 8).

  At this stage, since the player 2 has not yet held the game controller 1200, the left-hand object 14 and the right-hand object 16 can be displayed as shown in FIG. Since the controller object 12 does not fall within the shooting angle of view from the virtual stereo camera 10 in this state (see FIG. 6B), the direction guidance object 13 is displayed on the screen as the first guidance display. (See FIG. 6 (1)).

Although the player 2 has not yet held the game controller 1200, the player 2 can input an operation based on the position of his / her hand.
In the example of FIG. 6A, the player 2 is requested to confirm the intention to start the game. If there is an intention to start the game with the game controller 1200 in hand, the arm may be extended to the left side in an attempt to touch the left selected object 24 with the hand. As the left arm is extended, the left hand object 14 is moved and displayed. As shown in the stereoscopic image (game image) W4 shown in FIG. The display form is switched to another display form. In the example of FIG. 7A, the display color is changed. Then, when the state is maintained for a predetermined time or a predetermined action (for example, a hand is shaken on the spot), it is determined that the left hand object 14 has selected and input the left selection object 24, and the left selection object 24 is selected. The input of the option associated with is confirmed. That is, the intention to play the game is input by gesture. In FIG. 7A, the direction guide object 13 shows another example.

  If the player 2 rotates the neck leftward with reference to the direction guide object 13 from the state of FIG. 7A, the direction of the virtual stereo camera 10 in the virtual space also rotates leftward. The HMD 1310 displays the stereoscopic image (game screen) W6 of FIG.

  The direction guidance object 13 displayed in FIG. 6A and FIG. 7A is erased from the virtual space when the controller object 12 enters the shooting range of the virtual stereo camera 10 in the state of FIG. Then, it disappears from the stereoscopic image (game screen) W6. That is, the first guidance display is switched from the direction guidance object 13 to the controller object 12.

  Since the selected object 24 is fixedly displayed at a predetermined position of the stereoscopic image (game screen), it is not lost. Further, the selection target 24 includes a front direction guide portion 25 (in the example of FIG. 8A, a drawing instruction portion (drawing portion)) indicating the front direction (direction of the touch panel 1004) at the game play position of the player 2. Is prepared, and even if the player 2 shakes his / her head, the front direction is not lost.

  Depending on the orientation of the head of the player 2 (that is, the line-of-sight direction of the virtual stereo camera 10), a case where a part or all of the controller object 12 is hidden behind another object such as the selected object 24 may occur. Other objects are not particularly limited. For example, the guide character 20, the guide display panel 22, and objects constituting the background are also included. In this case, the controller object 12 is exposed. Specifically, the object that hides the controller object 12 is displayed semi-transparently as shown in FIG. 8A, or the model of the hidden object is temporarily changed so that the controller object 12 can be seen. For example, you might make a peephole or cut a corner. In the example of FIG. 8A, the selected object 24 is a hidden object.

  In the present embodiment, the horizontal direction and the vertical direction of the virtual space are the same as the real world (real space) where the player 2 is located, and the scale is also the same as the real world (real space). The shooting angle of view of the virtual stereo camera 10 is matched to the naked eye. Therefore, for the player 2 (user), the stereoscopic image seen by the HMD 1310 is recognized as a virtual reality image having the same sense of distance as the reality. Therefore, the player 2 can easily reach the game controller 1200 by extending his arm toward the position where the controller object 12 is visible. Even in the process of reaching the hand, as the arm is extended, the left hand object 14 (or the right hand object 16) is moved and displayed in the stereoscopic image to indicate the latest hand position. The game controller 1200 can be easily picked up.

  In this embodiment, the guidance to the game controller 1200 is provided in combination with the vibration of the game controller 1200 and the signal sound (guidance sound) from the headphones 1302 in addition to such stereoscopic vision.

  Specifically, as shown in FIG. 6B, when it is determined that the game controller 1200 is not gripped, the vibrator 1206 (see FIG. 2) is operated to vibrate the game controller 1200. The conditions for detecting that the robot is not gripped are, for example, 1) the signal detected by the 6-axis sensor 1208 does not change for a predetermined time or longer, and 2) the operation input from the operation button switch 1260 or the like is not detected for a predetermined time or longer. ) It can be a single or a combination of the conditions that the relative distance from the position Pc of the game controller 1200 to the left hand position Pl or the right hand position Pr is more than the reference value.

  The sound generated by the vibration of the game controller 1200 provides a clue for the player 2 to know which direction the game controller 1200 is. In addition, it is convenient that the game controller 1200 is immediately recognized by vibration when touched by a hand.

The headphone 1302 emits a signal sound in surround so that sound can be heard from the direction of the position Pc of the game controller 1200 as viewed from the position Ph of the HMD 1310.
The surround function itself is realized by a known technique. The player 2 audibly knows where the game controller 1200 is from the direction in which the signal sound can be heard, finds the game controller 1200, and can pick it up. At this time, the volume or pitch is changed according to the relative distance between the hand position (right hand position Pr, left hand position Pl) and the position Pc of the game controller 1200, or the period of the pulse sound is changed. It is more suitable to perform sound control such as.

[Description of functional configuration]
Next, a functional configuration example for realizing the present embodiment will be described.
FIG. 9 is a functional block diagram illustrating a functional configuration example of the game system 1000 according to the present embodiment. The game system 1000 according to the present embodiment includes an operation input unit 100, a processing unit 200, a sound output unit 390, an image display unit 392, a communication unit 394, a controller oscillation unit 396, and a storage unit 500. In FIG. 9, main functional blocks of the game apparatus main body 1002 have a configuration excluding the operation input unit 100, the sound output unit 390, the image display unit 392, and the controller oscillation unit 396.

  The operation input unit 100 is a means for inputting various operations. It can be realized by a known operation input device such as a joystick, a push switch, a lever, a volume, a pedal switch, a keyboard, a mouse, a track pad, a track ball, and a gesture input device. In the present embodiment, the touch panel 1004 and the game controller 1200 correspond to this (see FIG. 1).

  The operation input unit 100 of the present embodiment includes a head posture change detection unit 102 and a distance measurement unit 104.

  The head posture change detection unit 102 detects the posture and posture change of the player 2 and outputs a detection signal to the processing unit 200. For example, it can be realized by a known head tracking technique such as using a gyroscope or an acceleration sensor, or recognizing a human head from a captured image by an image sensor. In the present embodiment, the HMD 1310 (see FIG. 3) built in the headset 1300, more specifically, the 6-axis sensor 1308 and the IC chip 1309 correspond to this. It can be said that the distance measuring sensor unit 1100 corresponds to this as long as the configuration is obtained from the image processing of the HMD 1310 taken by the distance measuring sensor unit 1100.

  The ranging unit 104 captures the player 2 and its surroundings at a predetermined cycle, measures the distance to the existing object, and outputs the captured image data and the ranging result to the sequential processing unit 200. In the present embodiment, the distance measuring sensor unit 1100 corresponds (see FIG. 1).

  The processing unit 200 is realized by electronic components such as a microprocessor such as a CPU and a GPU, an ASIC (Application Specific Integrated Circuit), an FPGA, and an IC memory, and includes each function unit including the operation input unit 100 and the storage unit 500. Data input / output control between Then, various arithmetic processes are executed based on predetermined programs, data, and input signals from the operation input unit 100, and the operation of the game system 1000 is integratedly controlled. In the present embodiment, the control board 1050 corresponds to this (see FIG. 1).

  The processing unit 200 includes a game calculation unit 202, a sound signal generation unit 290, an image signal generation unit 292, and a communication control unit 294.

The game calculation unit 202 executes various processes related to game play.
For example,
1) Processing for arranging a background object, a player character object, and a virtual stereo camera 10 in a virtual space;
2) processing for controlling the movement of the player character object based on the operation input signal from the operation input unit 100;
3) Based on the detection information from the head posture change detection unit 102, the virtual stereo camera 10 is linked with the posture change of the head of the player 2 and the game space image captured by the virtual stereo camera 10 is rendered. Processing to display as a stereoscopic image on the HMD 1310;
4) Game end determination process,
5) Game score calculation processing,
6) Timekeeping processing can be performed. Of course, other processes can be executed as appropriate according to the game content.

  Then, the game calculation unit 202 of the present embodiment has an HMD position / orientation calculation unit 210, a controller position / orientation calculation unit 212, a hand position calculation unit 214, and a controller gripping determination for the purpose of improving usability in using the HMD. A unit 216 and a player auxiliary control unit 220.

The HMD position / orientation calculation unit 210 calculates the position and orientation of the HMD 1310 in real space as needed at predetermined intervals (for example, every 10 ms). In the present embodiment, the position Ph of the HMD 1310 (or the head of the player 2) is calculated from the image obtained by the distance measurement unit 104 and the distance measurement result, and the posture is based on the detection information from the head posture change detection unit 102. (Φh, θh, Ψh) is obtained.
As a method for obtaining the position Ph, a known technique can be used as appropriate. For example, three light emitting markers 1312 of the HMD 1310 shown in the image obtained by the distance measuring unit 104 are extracted, and the X-axis coordinate value and the Y-axis coordinate value are obtained from the coordinates in the intermediate position image. The Z-axis coordinate value may be determined from the distance measurement result corresponding to the coordinate value. As for the posture, a known technique for recognizing and calculating a predetermined geometric pattern printed on the front and side surfaces of the light emitting marker 1312 and the HMD 1310 from an image photographed by the color image sensor 1104 may be used. .

The controller position / orientation calculation unit 212 calculates the position Pc of the game controller 1200 at any given time (for example, every 10 ms). In the present embodiment, the position Pc of the game controller 1200 is calculated from the image obtained by the distance measuring unit 104 and the distance measurement result.
As a method for obtaining the position Pc, a known technique can be used as appropriate. For example, two light emitting markers 1202 of the game controller 1200 shown in the image obtained by the distance measuring unit 104 are extracted, and the X-axis coordinate value and the Y-axis coordinate value are determined from the coordinates in the image at the intermediate position of the extracted position. And the Z-axis coordinate value may be determined from the distance measurement result corresponding to the coordinate value. Alternatively, the game controller 1200 may be determined by recognizing the image from an image captured by the color image sensor 1104.

The hand position calculation unit 214 calculates a virtual position corresponding to the position of the operation input object for gesture input in the real space in a virtual space (player auxiliary display virtual three-dimensional space) serving as a display image space of the HMD 1310. In the present embodiment, the left hand position Pl and the right hand position Pr of the player 2 are calculated as needed at predetermined intervals (for example, every 10 ms). That is, the position of the hand of the player 2 is tracked.
As a method for calculating the hand position (the virtual position of the left hand and the right hand), a known technique can be used as appropriate. For example, a human hand is image-recognized from an image captured by the color image sensor 1104, an X-axis coordinate value and a Y-axis coordinate value are obtained from the coordinates of the recognized image, and a distance measurement result up to the hand corresponding to the coordinate value The Z-axis coordinate value may be determined from

  The controller grip determination unit 216 determines whether the player 2 has the game controller 1200 in hand. In other words, it can be detected that it is gripped. The determination method / detection method can be selected as appropriate. For example, when there is no operation input for a predetermined time (for example, 5 seconds), or when there is no change in the posture of the game controller 1200 for a predetermined time (for example, 5 seconds), it may be determined that it is not gripped. If a sensor for detecting the electrostatic capacity can be mounted on the grip portion of the game controller 1200, it may be detected whether or not the sensor is gripped from a change in the electrostatic capacity due to the contact of a human hand.

The player auxiliary control unit 220 performs main control for improving inconvenience when using the HMD and improving usability.
In the present embodiment, the virtual stereo camera control unit 221, the controller object control unit 222, the direction guidance object control unit 224, the hand object control unit 226, the selected object control unit 228, the expression processing unit 230, A stereoscopic image generation control unit 232, a controller vibration control unit 234, a pseudo surround control unit 236, a gesture input determination unit 238, and an anti-character gesture input invalidation control unit 240 are included.

  The virtual stereo camera control unit 221 controls the position and posture of the virtual stereo camera 10 in the virtual space so as to correspond to the position and posture of the HMD 1310 in the real space. That is, the position and posture of the virtual stereo camera 10 are controlled so as to interlock and follow the head position and the line-of-sight direction (that is, the face front direction) of the player 2 wearing the HMD 1310 in real space.

  The controller object control unit 222 arranges the controller object 12 in the virtual space and controls its display form. The display form is set to a standard form until the relative distance between the game controller 1200 and the hand of the player 2 satisfies a predetermined proximity condition, and when the proximity condition is satisfied, the display form is changed to a special form.

  The direction guidance object control unit 224 arranges the direction guidance object 13 in the virtual space and controls its movement and posture. Specifically, the position of the direction guide object 13 is controlled so that it always stays at a predetermined position in the field of view (game screen) of the HMD 1310, and the posture is controlled so as to indicate the direction of the game controller 1200 viewed from the HMD 1310.

  The hand object control unit 226 arranges the left hand object 14 and the right hand object 16 in the virtual space, and controls their movement. Specifically, control is performed so that the left hand object 14 and the right hand object 16 are positioned at positions in the virtual space corresponding to the relative positions of the left hand position and the right hand position with respect to the HMD 1310 in the real space.

  The selected body control unit 228 controls display of a display body that the user can use as a standard of gesture when requesting gesture input. As a result, since the gesture input is determined based on the positional relationship between the display body and the operation instruction object such as the player's hand, it can be said that the display control is performed on the “input determination body”. In the present embodiment, the selected object 24 associated with the operation input content is used as a kind of input determination object, and is arranged in the virtual space as a “speech balloon” of the guide character 20 to be the target reaching position of the player's hand. . Then, the position of the selected body 24 is controlled so that it always stays at a predetermined position in the field of view (game screen) of the HMD 1310, while the front of the selected body 24 is controlled to face the virtual stereo camera 10. Further, the front direction guide unit 25 performs control so as to indicate the direction of a predetermined position (for example, the coordinate origin position) in the virtual space. In this embodiment, by fixing the tip of the front direction guide portion 25 at a predetermined position of the guide character 20 and controlling the deformation of the object so as to be continuous with the selected object 24, it is as if the “balloon” expands and contracts. Although it is free, it always seems to point to the guide character 20.

  The display form of the selected object 24 can be set as appropriate according to the game content and the like. For example, it may be a hand movement start point to an end point, or a movement locus guide connecting the two points. Of course, it may also serve as a character or background that appears in the game.

  When there is another object between the virtual stereo camera 10 and the controller object 12 and part or all of the controller object 12 is hidden behind the other object, the exposure processing unit 230 becomes invisible. Visualization is performed by image processing that makes it appear so that it can be seen. For example, the object model is changed such that another object hiding the controller object 12 is made translucent or a peephole is made in the other object.

  Similarly, when there is another object between the virtual stereo camera 10 and the selected object 24 and a part or all of the selected object 24 is not visible behind the other object, the selected object. The image is processed and visualized so that 24 can be seen.

  Similarly, when there is another object between the virtual stereo camera 10 and the pointing object such as the left hand object 14 or the right hand object 16, and part or all of the pointing object is behind the other object and cannot be seen. The image is processed and visualized so that the pointing object can be seen.

  The stereoscopic image generation control unit 232 performs control for generating an image of a virtual space in which the field of view changes according to the position and orientation of the HMD 1310. Specifically, images of the virtual space taken by the left and right cameras of the virtual stereo camera 10 are rendered into the left-eye and right-eye images of the HMD 1310. That is, a stereoscopic image to be a game image (game screen) is generated. The stereoscopic image generation control unit 232 is an example of a virtual space image generation unit.

  The controller vibration control unit 234 performs control to vibrate the game controller 1200 when the player 2 is not holding the game controller 1200.

  The pseudo surround control unit 236 artificially surrounds stereo sound so that a signal sound can be heard from the direction of the game controller 1200 viewed from the HMD 1310 when the player 2 is not holding the game controller 1200. In the present embodiment, since the sound is provided to the player 2 by the headphones 1302, the known pseudo surround technology and virtual surround technology can be appropriately used.

The gesture input determination unit 238 determines an operation input to the object based on the virtual position (left hand position Pl, right hand position Pr) of the left hand or the right hand, which is the operation instruction, and the arrangement position of the object in the virtual space. To do.
In the present embodiment, the selection input of the selected object 24 can be determined based on the position of the operation instruction object in the virtual space and the arrangement position of the selected object 24. Specifically, when the relationship between the left hand position Pl ′ or the right hand position Pr ′ in the virtual space and the arrangement position of the selected object 24 satisfies the arrival determination condition, a selection input associated with the selected object 24 is input. Judgment is made. Similarly, the gesture input determination unit 238 can determine a gesture input for the guide character 20.

  When the selected object 24 is arranged and displayed, the character-to-character gesture input invalidation control unit 240 performs gesture input (for example, a guide character 20 in FIG. 5) as a target (for example, the guide character 20 in FIG. 5). The determination of an operation input such as stroking the guide character 20 is invalid, and is valid when the selected object 24 is not arranged or displayed. In other words, in addition to the selected object 24, the gesture input target arranged in the virtual space is temporarily excluded from the operation target, and only the gesture input to the selected target object 24 to be prioritized is permitted. Prevents unintended and undesired gesture input on objects.

  The sound signal generation unit 290 is realized by a known technique such as a microprocessor such as a digital signal processor (DSP) and its control program, and generates sound effects, BGM, various operation sounds, and the like based on the processing result by the game calculation unit 202. A sound signal including the game sound is generated and output to the sound output unit 390.

  The sound output unit 390 is realized by a device that outputs sound effects, BGM, and the like based on the sound signal input from the sound signal generation unit 290. In the present embodiment, the headphones 1302 mounted on the headset 1300 correspond to this (see FIG. 3).

  The image signal generation unit 292 can output image signals such as image data generated by the stereoscopic image generation control unit 232 and image data related to the setting screen of the game system 1000 to the image display unit 392.

  The image signal generation unit 392 displays various images based on the image signal input from the image signal generation unit 292. For example, it can be realized by an image display device such as a flat panel display, a cathode ray tube (CRT), or a projector. In the present embodiment, the touch panel 1004 (see FIG. 1) and the HMD 1310 (see FIG. 3) are applicable.

  The communication control unit 294 executes data processing related to data communication, and realizes data exchange with an external device via the communication unit 394.

  The communication unit 394 is realized by, for example, a wireless communication device, a modem, a TA (terminal adapter), a cable communication cable jack, a control circuit, and the like, and communicates with an external server system via a communication network. Realize. In addition, wireless communication with a wireless connection device such as the game controller 1200 is realized.

  The controller oscillation unit 396 causes the game controller 1200 to vibrate according to the vibration control signal output from the game calculation unit 202. In the present embodiment, the vibrator 1206 and the controller board 1250 correspond to this (see FIG. 2).

  The storage unit 500 stores programs and various data for realizing various functions for causing the processing unit 200 to control the game system 1000 in an integrated manner. Further, it is used as a work area of the processing unit 200, temporarily stores data input from the operation input unit 100, and temporarily stores calculation results executed by the processing unit 200. This function is realized by, for example, an IC memory such as a RAM and a ROM, a magnetic disk such as a hard disk, and an optical disk such as a CD-ROM and DVD. In this embodiment, the IC memory 1052 corresponds to this (see FIG. 1).

  FIG. 10 is a diagram illustrating an example of programs and data stored in the storage unit 500 according to the present embodiment. The storage unit 500 stores a system program 501, a game calculation program 502, game initial setting data 510, and play data 520. Of course, other information can be stored as appropriate.

The system program 501 is read and executed by the processing unit 200, thereby causing the game system 1000 to realize basic input / output functions necessary as a computer.
The game calculation program 502 is a program for causing the processing unit 200 to realize the function as the game calculation unit 202.

  The game initial setting data 510 stores game space, game rules, and various initial setting data that define various objects appearing in the game.

  The play data 520 stores data for player auxiliary control and various data describing the game progress status. In the present embodiment, the HMD position coordinates 521 (the position Ph of the HMD 1310), the HMD attitude data 522, the controller position coordinates 523 (the position Pc of the game controller 1200), and the left-hand position coordinates 525 in the distance measurement space that is a real space. (Left hand position Pl) and right hand position coordinates 526 (right hand position Pr). Also, virtual space control data 630 is included. Of course, information other than these can be included as appropriate.

  The virtual space control data 630 is data of a virtual three-dimensional space (virtual space) constituting the game space, and includes data for managing the arrangement of the player auxiliary display objects. For example, as shown in FIG. 11, virtual stereo camera control data 602, controller object control data 604, direction guidance object control data 606, left hand object control data 608, right hand object control data 610, guidance character control data. 612, guidance display panel control data 614, and selected object control data 616. Of course, information other than these can be included as appropriate.

[Description of operation]
FIG. 12 is a flowchart for explaining the flow of processing in the game system 1000, and is realized by executing the game calculation program 502.

  First, the game apparatus main body 1002 of the game system 1000 performs device authentication of each device of the touch panel 1004, the distance measuring sensor unit 1100, the game controller 1200, and the headset 1300 (step S2).

  By completing the device authentication, the distance measurement sensor unit 1100 sequentially inputs image data photographed at a predetermined cycle by the color image sensor 1104 and a distance measurement result based on the image to the game apparatus main body 1002. Further, a detection result by the 6-axis sensor 1208 from the game controller 1200 and a detection result by the 6-axis sensor 1308 of the HMD 1310 are input from the headset 1300 to the game apparatus main body 1002 at a predetermined cycle.

  Next, the game apparatus main body 1002 determines the position Ph of the HMD 1310, the position Pc of the game controller 1200, the left hand position Pl and the right hand position of the player 2 based on the image and the distance measurement result input from the distance measurement sensor unit 1100. A process for calculating Pr at any time in a predetermined cycle is started (step S4). That is, tracking of each of these elements in the real space is started.

  Further, a process for determining at any time whether or not the game controller 1200 is held by the hand of the player 2 is started at a predetermined cycle (step S6), and determination of a gesture input is started (step S8).

  Next, the game apparatus body 1002 displays a startup screen on the touch panel 1004 and waits for an operation input for starting the game. When a predetermined game start operation is input from the game controller 1200 (step S10), the game apparatus main body 1002 displays guidance for prompting the player 2 to wear the headset 1300 on the touch panel 1004 (step S12). ). Then, the acceptance of confirmation of the intention to play the game is started (step S14), and the stereoscopic image based on the state of the virtual space photographed by the virtual stereo camera 10 with the virtual stereo camera 10 placed at the initial position of the virtual space. Is displayed on the HMD 1310 (step S15), and player assist processing is executed (step S16).

13 to 14 are flowcharts for explaining the flow of the player auxiliary process.
As shown in FIG. 13, in the player assistance process, the game apparatus main body 1002 generates virtual space control data 630 (see FIG. 11), and the virtual space in the virtual space is linked to the position / posture of the HMD 1310 in the real space. Control of the movement and posture of the stereo camera 10 is started (step S50).

  Further, a left hand object 14 and a right hand object 16 (see FIG. 5) are arranged in the virtual space, and the objects 2 in the virtual space are linked to the positions and postures of the left hand and right hand of the player 2 in the real space. The movement / posture control is started (step S52).

  Next, when the game controller 1200 is not held by the player 2 (NO in step S60), the game apparatus main body 1002 places the controller object 12 (see FIG. 5) in the virtual space, and the real space of the game controller 1200. The movement / posture control of the object in the virtual space is started so as to be interlocked with the position / posture at (step S62).

Then, the display form adjustment of the controller object 12 is started (step S64).
Specifically, the relative distance between the controller object 12 and the left hand object 14 and the relative distance between the controller object 12 and the right hand object 16 are respectively calculated, and the display form of the controller object 12 (for example, according to the smaller relative distance) , Change display color, blink pattern, change display size, etc.).

  Next, the game apparatus body 1002 of the game system 1000 starts determining whether or not to display the direction guidance object 13 and starts arranging / releasing the object according to the determination result (step S66).

  Specifically, it is determined whether the controller object 12 is within the shooting range of the virtual stereo camera 10. If it is not within the shooting range, the direction guidance object 13 is regarded as requiring display, and the direction guidance object 13 pointing to the controller object 12 is immediately placed in a predetermined area within the shooting range. In other words, the arrangement position and orientation of the direction guide object 13 are controlled according to the direction or field of view of the HMD 1310. On the other hand, if it is not within the shooting range, the direction guidance object 13 is regarded as unnecessary to be displayed, and the direction guidance object 13 is not arranged.

  Along with this, it is assumed that the display form is successively changed and controlled according to the distance from the left hand object 14 or the right hand object 16 nearest to the virtual stereo camera 10. For example, the display color may be changed, or the dimensional ratio (for example, the length of an arrow) may be changed. If the design includes only arrows, the number of arrows may be increased as the distance increases, and the arrows may be arranged in series toward the controller object 12.

Next, the game apparatus body 1002 starts pseudo-surround output of signal sound through the headphones 1302 (step S68).
Specifically, the relative direction and the relative distance from the HMD 1310 to the game controller 1200 are calculated, and pseudo surround control is performed so that a signal sound whose pattern is changed according to the relative distance from the relative direction is heard by the headphones 1302.

  Next, the game apparatus body 1002 starts vibration control by the game controller 1200 (step S70). The vibration pattern and the vibration intensity may be set / changed according to the relative distance between the controller object 12 and the left hand object 14 or the right hand object 16.

Next, it is determined whether guidance display is necessary. Since the acceptance of the operation input for confirming the intention to play the game is started in step S14, it is determined here that the guidance display is necessary.
If guidance display is necessary (YES in step S90), the game apparatus main body 1002 has a virtual stereo camera 10 and a virtual space in front of the virtual stereo camera 10, specifically in the depth direction viewed from the virtual stereo camera 10. A guide character 20 and a guide display panel 22 are arranged between the coordinate origin (step S92: see FIG. 5). Incidentally, since the acceptance of the operation input for confirming the game play is started in step S14, the guidance display panel 22 displays a text prompting the operation input for confirming the will (see FIG. 6).

  Furthermore, the game apparatus body 1002 invalidates the determination of the gesture input to the guide character 20 (step S94). Thereby, the intention of the gesture input to the selected object 24 is not mistaken for the operation input to the unintended guide character 20.

Moving to the flowchart of FIG. 14, the game apparatus body 1002 next determines whether or not an input of a selection operation is necessary. Since acceptance of an operation input for confirming the intention to play the game is started in step S14, an affirmative determination is made here.
If the result is affirmative (YES in step S110), the game apparatus main body 1002 places the selected object 24 associated with the option in the virtual space (step S112). Specifically, since the selected object 24 of this embodiment has a “balloon” shape, the tip of the balloon drawer is set near the guide character 20, and the main body of the balloon is placed beside the guide character 20. (Refer to FIG. 6).

  Next, the game apparatus main body 1002 performs movement control of the selected object 24 according to changes in the position and posture of the virtual stereo camera 10 and deformation control of the front direction guide unit 25 (step S114). That is, movement control is performed so that the selected object 24 is fixedly displayed at a predetermined position of the stereoscopic image (game screen), and control is performed so that the front surface of the selected object 24 always faces the virtual stereo camera 10. Further, deformation control is performed so that the front direction guide portion 25 extends from the side of the guide character 20 and is connected to the selection target 24.

  Note that step S114 may be skipped when a change in the position or orientation of the virtual stereo camera 10 satisfies a given condition. For example, when the front of the virtual stereo camera 10 (the front direction of the HMD 1310) is 90 ° or more left and right from the front-rear direction of the real space (the front direction of the distance measuring sensor unit 1100), it is skipped or the control is performed immediately before It may be held by.

Then, for each selected object 24, automatic adjustment of the display form according to the relative distance from the nearest left hand object 14 or right hand object 16 is started (step S116).
Specifically, if the relative distance between the selected object 24 and the left-hand object 14 or the right-hand object 16 does not satisfy a predetermined proximity condition that the selected object 24 is selected (reached), the standard display Form, and a special display form for notifying selection when the proximity condition is satisfied.

Next, the game apparatus main body 1002 detects another object that hides a part or the whole of the controller object 12 from the virtual stereo camera 10, and if there is a corresponding other object, the controller device 12 is hidden behind the controller object 12. Image processing (display processing) to be visualized is started (step S118).
Specifically, a part or the whole of the other object is displayed in a semi-transparent manner, or a part of the model of the other object is hidden or deformed to display the controller object 12 hidden behind.

In addition, the game apparatus main body 1002 detects another object that hides a part or the whole of the selected object 24 when viewed from the virtual stereo camera 10, and the selected object 24 that is hidden behind when there is a corresponding other object. The image processing (expressing process) for displaying and visualizing is started (step S120).
In the same manner, the left hand object 14 and the right hand object 16 are also hidden behind the other object that is detected when other objects that are partially or entirely hidden from the virtual stereo camera 10 are detected. Image processing (display processing) for displaying the left hand object 14 and the right hand object 16 for visualization is started (step S122).

Here, the game images as shown in FIGS. 6A to 8A can be seen in the field of view of the player 2.
Next, selection input by gesture input is determined (step S170). In the present embodiment, when the left hand object 14 or the right hand object 16 reaches one of the selected bodies 24, it is determined that the input is made by a gesture. If the selection input is two choices such as YES / NO, YES and NO (or options) are assigned to the left and right directions of the field of view of the HMD 1310, respectively, and the left and right fields of view are changed by the left hand object 14 or the right hand object 16. It may be determined in which area a predetermined motion (for example, a hand is pushed forward, a hand is shaken), or a hand is in a predetermined form like a bark.

  If it is determined that the selection has been input (YES in step S170), the game apparatus body 1002 determines the input of the option corresponding to the corresponding selection target 24 (step S172). Further, at this time, the display form of the corresponding selected object 24 is changed to a predetermined form, and the player 2 is notified of the choice input.

  And the display of the to-be-selected body 24 is cancelled | released, and the determination of the gesture input to the guidance character 20 is validated (step S174). If there are a plurality of selected objects 24, the selected object 24 determined to be reached in step S170 is erased from the screen at the end, and the effect of clearly indicating which selected object 24 the player has selected is obtained. Born and suitable. Further, even when there is one selected object 24 (for example, the selected object 24 as a confirmation button), steps S172 to S174 can be executed similarly.

  Next, if it is detected that the game controller 1200 is gripped by the player 2 after starting the player assisting process (YES in step S180), the game apparatus body 1002 generates vibrations in the game controller 1200, The pseudo surround output from the headphones 1302 is terminated (step S182), and the player auxiliary process is terminated.

Returning to FIG. 12, in a state where step S16 is completed, the player 2 wears the headset 1300 on the head and holds the game controller 1200 in the hand. In other words, it is ready for game play.
Therefore, the game apparatus main body 1002 starts control of game play next, and starts control for displaying the game screen on the HMD 1310 and control for emitting the game sound with the headphones 1302 (step S200).

  The game apparatus body 1002 periodically determines whether or not the player auxiliary activation condition is satisfied even during game play. The player auxiliary activation condition can be set as appropriate according to the content of the game. However, in the present embodiment, when the game controller 1200 is not held, when an operation input is performed by a device other than the game controller 1200 such as a keyboard, When gesture input by the hand of the player 2 is required, it is set by a single or a combination.

  If the player assistance trigger condition is satisfied (YES in step S202), the player assistance process is executed again (step S204).

  If the player auxiliary activation condition is satisfied until the player auxiliary activation condition is satisfied, or if the player auxiliary activation condition is satisfied until the game progress condition satisfies a predetermined game completion condition after the player auxiliary processing is not satisfied, In step S210, YES), the game apparatus body 1002 ends the series of processes.

As described above, according to the present embodiment, first, inconvenience when using the HMD with a game device or the like can be improved, and usability can be improved.
That is, when it is detected that the player 2 is not holding the game controller 1200, a stereoscopic image of the virtual space viewed from the viewpoint position of the player that has been head-tracked is displayed on the HMD 1310 as a game image. In the stereoscopic image, the controller object 12 is displayed at the position of the game controller 1200 viewed from the user's viewpoint, and the left hand object 14 and the right hand object 16 of the player to be motion-traced are displayed. Therefore, even if the player wears the HMD and cannot see the appearance of the outside world, the virtual space of the stereoscopic image becomes a virtual reality space, so that the player can easily reach out to the place where the game controller 1200 is located. be able to. The inconvenience of searching for a controller by groping is eliminated.

  Further, when the game controller 1200 is outside the field of view of the HMD 1310, the direction guide object 13 indicates which direction the game controller 1200 can be seen by turning the neck. Can be easily picked up. Moreover, even when the controller object 12 is hidden behind another object, the other object is automatically rendered semi-transparent or cut out, and the controller object 12 becomes visible to the player. Can be found.

Moreover, according to this embodiment, the usability at the time of implement | achieving gesture input using 2nd HMD can be improved 2ndly.
That is, in the stereoscopic image displayed on the head-tracked HMD 1310, the selected object 24 to which the content of gesture input is assigned remains displayed at the same or substantially the same position even if the player changes the head direction. . If you are not familiar with gesture input, it is difficult to understand how you can make the desired input on the spot, but the user can use the selected object as a guide for gesture operation, so it will be requested smoothly at that time Gesture input will be possible.

  Further, when the left hand object 14 or the right hand object 16 reaches the selected body 24, the gesture input is confirmed. Therefore, from the positional relationship between the left hand object 14 or the right hand object 16 and the selected body 24, anyone can easily perform the gesture input required at that time.

In addition, even when the selected object 24 is hidden behind the other object, the other object is automatically rendered translucent or cut out so that the selected object 24 can be seen by the player. So you can find it right away.
In addition, when there is a gesture input target other than the selected object in the virtual space, the input to the selected object is given priority and unintended input by invalidating the gesture input to other than the selected object. Can be prevented.

[Modification]
Note that the embodiment of the present invention is not limited to the above, and it is possible to add, omit, and change components as appropriate.

[Part 1]
For example, in the above-described embodiment, the non-transparent HMD that cannot be seen from the inside when the HMD is mounted has been described. However, the non-transparent HMD is not necessarily required depending on the content of the game. For example, the present invention can be applied to a transflective HMD, and the same effects as those of the above embodiment can be obtained.

[Part 2]
Further, the headset 1300 itself may be, for example, a glasses-type or goggles-type wearable computer equipped with a sensor capable of detecting a posture, and may also serve as the game apparatus main body 1002. Furthermore, if the game system 1000 of the said embodiment is a computer provided with HMD, it will not be a game device assumed to be a home stationary type, but an attraction device using an HMD installed in a facility such as an amusement park or a museum. Other types of devices, such as aircraft and helicopter operation simulation devices that are equipped with HMDs, and sports simulation devices that are equipped with HMDs to swing sports with real golf clubs in the studio. Good.

[Part 3]
In the above embodiment, the stand-alone game system 1000 is exemplified, but as shown in FIG. 15, it may be realized by a client-server game system 1000. In this case, the game calculation unit 202 (see FIG. 9) in the above embodiment is realized by a server system 1900 connected via the communication line 9, and the game apparatus body 1002 in the configuration executes a predetermined client program. By doing so, a function of sequentially transmitting an input signal from the operation input unit 100 to the server system 1900, a function of displaying an image on the HMD 1310 based on data received from the server system 1900, and a function of outputting sound through the headphones 1302 And realize. That is, it comes to have a control function as a user terminal.

[Part 4]
In the above embodiment, the method of detecting the attitude of the HMD 1310 is configured to use an image captured by the distance measuring sensor unit 1100 and a detection result of the 6-axis sensor 1308 of the HMD 1310, but is not limited thereto.
For example, as shown in FIG. 15, an image sensor 1320 capable of infrared imaging is added to the HMD 1310 as an attitude detection sensor in order to capture the front outside world, and the ranging sensor unit 1100 has an infrared illuminator 1102 in a predetermined arrangement relationship. Place multiple. The controller position / orientation calculation unit 212 is used as a controller capture detection unit. Specifically, if the infrared illuminator 1102 is regarded as a fixed marker installed at a predetermined position in the real space, and the infrared illuminator 1102 is reflected in an image captured by the distance measuring sensor unit 1100, the game controller 1200 is displayed. It is determined that the image is captured, the imaging position (including the distance) of the infrared irradiator 1102 is obtained from the captured image, the positional relationship between each infrared irradiator 1102 is rotated or deformed, and the distance between the infrared irradiators 1102 is reflected. The position Ph and the posture of the HMD 1310 with respect to the position of the image sensor 1320 are calculated geometrically from the size of the infrared irradiator 1102 that is present. Then, the controller object control unit 222 of the player auxiliary control unit 220 may perform predetermined notification control by placing the controller object 12 in the virtual space according to the detection result of the controller capture detection unit 208. That is, the controller object control unit 222 may function as a capture notification control unit. The notification method can be realized by an appropriate method such as display / sound output / vibration.

[Part 5]
Moreover, in the said embodiment, although the position Pc of the game controller 1200 was described by the position coordinate in the coordinate system (refer FIG. 4) of a ranging space, it is not restricted to this.
For example, the position Pc of the game controller 1200 may be described as a relative position based on the position Ph of the HMD 1310 in the recognition space and the attitude of the HMD 1310 (in other words, the viewpoint of the player 2 wearing the HMD 1310 and the line-of-sight direction). It is possible, and the description of the above embodiment can be appropriately read as such.

[Part 6]
Moreover, in the said embodiment, the to-be-selected body 24 is arrange | positioned so that it may oppose the virtual stereo camera 10 in the predetermined area | region in the visual field of HMD1310, and we decided to perform movement control and attitude | position control according to the position and attitude | position of HMD1310. However, the movement control and the posture control may be omitted, and similarly to the guidance display panel 22, even if the position and posture of the HMD 1310 change, it may be arranged in a predetermined posture in the virtual space. In the above-described embodiment, the head tracking is intense, such as FPS (First Person Shooting), which is suitable for a game content that is likely to lose sight of the selected object 24, but conversely, for example, a secret room in an adventure game. In a game including contents such as inputting a number and an operation for unlocking the key, the selected object 24 is fixedly arranged without moving to a predetermined position in the virtual space, and the HMD 1310 faces the direction of the predetermined position. It is preferable that only appear in the field of view.

  Of course, it is not necessary to select either method, and both methods may be adopted. In that case, data defining a selection input event is stored in the storage unit 500 (see FIG. 9), and information indicating whether or not the arrangement position is fixed together with the selection input contents for each selected object 24 corresponds to the data. In addition, the selected object control unit 228 (see FIG. 9) may control the arrangement and posture (orientation) of the selected object 24 according to this.

[Part 7]
In the above embodiment, the player 2's hand is used as the operation instruction for the gesture input, but some pointing device may be used depending on the game content. As the pointing device, a known gun-type controller (so-called “Gankon”), a known stick-type motion controller, or the like may be used. In this case, a marker corresponding to the light emitting marker 1312 (see FIG. 3) is provided at the tip of the gun-type controller or stick-type motion controller, and the tip position is detected by photographing this with the distance measuring sensor unit 1100. It is preferable that the mounted 6-axis sensor detects the posture change. Of course, in that case, the pointing device may also serve as the game controller 1200.

[Part 8]
Moreover, although the image displayed on HMD1310 was demonstrated as a stereoscopic vision image in the said embodiment, it is good also as a planar vision image. In that case, one virtual camera may be arranged at an intermediate position between the left and right virtual cameras of the virtual stereo camera 10 and used instead of the virtual stereo camera 10, or one of the left and right virtual cameras is used. May be.

[Part 9]
In the above embodiment, the coordinate origin of the coordinate system (see FIG. 5) of the virtual space is the coordinate origin of the coordinate system (see FIG. 4) of the ranging space (real space), but the present invention is not limited to this.
For example, as shown in FIG. 16, the game calculation unit 202 may be provided with a coordinate system initial setting unit 204 that initially sets the coordinate system of the virtual space so as to correspond to the real space. The coordinate system initial setting unit 204 sets the position Ph of the HMD 1310 based on the distance measurement result by the distance measurement sensor unit 1100 at the tracking start time as the coordinate origin of the virtual space, aligns the coordinate axis with the attitude of the HMD 1310 at the tracking start time, The space coordinate system and the virtual space coordinate system are matched and initialized at the same timing.

Alternatively, as shown in FIG. 17, the intermediate point between the left hand position Pl and the right hand position Pr based on the distance measurement result by the distance measurement sensor unit 1100 at the time of tracking start is set as the coordinate origin VO of the virtual space, and the attitude of the HMD 1310 at the time of tracking start. The coordinate axis may be aligned with.
In the definition of the coordinate system, when the front and rear and the left and right are reversed in the real space and the virtual space, when the distance measurement result is reflected on the object in the virtual space (for example, based on the left hand position Pl, the left hand object 14 When performing a motion trace, etc.), coordinate conversion should be performed as appropriate.

[Part 10]
In the above embodiment, a function for correcting the scale of tracking or motion trace based on the relative positional relationship between the position Ph of the HMD 1310 in the real space and the operation instruction (left hand position Pl or right hand position Pr) is added. May be.
Specifically, as shown in FIG. 16, a virtual position correction unit 206 is provided. When the relative positional relationship based on the orientation of the HMD 1310 satisfies a predetermined correction condition, the virtual position correction unit 206 corrects the virtual position according to a correction calculation associated with the correction condition.
More specifically, as a correction condition, when the front surface of the HMD 1310 is 90 ° to the left or 90 ° to the right with respect to the front direction of the distance measuring sensor unit 1100 (the front-rear direction of the real space), the left 90 If it is more than 0 °, the scale of the motion trace at the left hand position Pl may be increased by a predetermined magnification, and if it is 90 ° or more at the right, the scale of the motion trace at the right hand position Pr may be increased by a predetermined magnification.

DESCRIPTION OF SYMBOLS 2 ... Player 10 ... Virtual stereo camera 12 ... Controller object 13 ... Direction guide object 14 ... Left hand object 16 ... Right hand object 20 ... Guide character 22 ... Guide display panel 24 ... Selected object 25 ... Front direction guide part 100 ... Operation input part DESCRIPTION OF SYMBOLS 102 ... Head posture change detection part 104 ... Distance measuring part 200 ... Processing part 202 ... Game operation part 210 ... HMD position / orientation calculation part 212 ... Controller position / orientation calculation part 214 ... Hand position calculation part 216 ... Controller gripping determination part 220 ... Player auxiliary control unit 221 ... Virtual stereo camera control unit 222 ... Controller object control unit 224 ... Direction guidance object control unit 226 ... Hand object control unit 228 ... Selected object control unit 230 ... Expression processing unit 232 ... Stereoscopic image generation control Part 234 Controller vibration control unit 236... Pseudo surround control unit 238 .. gesture input determination unit 240... Character gesture input invalidation control unit 396 .. controller oscillation unit 500 .. storage unit 502 .. game calculation program 510. 521 ... HMD position coordinates 522 ... HMD posture data 523 ... Controller position coordinates 525 ... Left hand position coordinates 526 ... Right hand position coordinates 602 ... Virtual stereo camera control data 604 ... Controller object control data 606 ... Direction guidance object control data 608 ... Left hand object control Data 610 ... Right hand object control data 612 ... Guidance character control data 614 ... Guidance display panel control data 616 ... Selected object control data 630 ... Virtual space control Data 1000 ... Game system 1002 ... Game device main body 1050 ... Control board 1100 ... Ranging sensor unit 1104 ... Color image sensor 1200 ... Game controller 1206 ... Vibrator 1208 ... 6-axis sensor 1250 ... Controller board 1300 ... Headset 1302 ... Headphone 1308 ... 6-axis sensor 1310 ... HMD
1312 ... Light emitting marker 1320 ... Image sensor Ph ... HMD position Pc ... Game controller position Pl ... Left hand position Pr ... Right hand position

Claims (16)

Photographing a real space including the HMD (Head Mounted Display) worn on the user's head, the user's hand or a pointing device (hereinafter collectively referred to as “operation instruction”) held by the user, and the HMD. A computer connected to the imaging means installed for
A calculation unit that calculates a virtual position corresponding to the position of the operation instruction in the real space in a virtual space that is a display image space of the HMD, using a captured image of the imaging unit;
An input determination object arrangement control means for arranging an input determination object at a given position in the virtual space;
An input determination means for determining an input to the input determination body based on the virtual position and an arrangement position of the input determination body;
Program to function as. HMD position and orientation determination means for determining the position and / or orientation of the HMD in the real space;
Virtual space image generation means for generating an image of the virtual space whose field of view changes according to the position and / or orientation of the HMD determined by the HMD position and orientation determination means as the image of the virtual space to be displayed on the HMD ,
Further functioning the computer as
The input determination object arrangement control means arranges the input determination object at a given position in the view following the change in the view.
The program according to claim 1. HMD position and orientation determination means for determining the position and / or orientation of the HMD in the real space;
Virtual space image generation means for generating an image of the virtual space whose field of view changes according to the position and / or orientation of the HMD determined by the HMD position and orientation determination means as the image of the virtual space to be displayed on the HMD ,
Further functioning the computer as
The input determination object arrangement control means sets the arrangement position of the input determination object as a predetermined position in the virtual space regardless of the change in the field of view.
The program according to claim 1. The virtual space image generation means generates an image in which characters are arranged in the virtual space,
The input determination body arrangement control means causes the input determination body to accompany the character.
The program according to claim 3. The input determination body arrangement control means, when a given condition is satisfied, interrupts the accompanying display of the input determination body and positions the input determination body at a predetermined position in the field of view.
The program according to claim 4. The input determination body arrangement control means, when the character is located within the field of view and then outside the field of view, when the given condition is satisfied,
The program according to claim 5. Pointing object display control means for causing the HMD to display a pointing object indicating that the virtual pointing position is a position corresponding to the position of the operation pointing object in the real space;
The program as described in any one of Claims 2-6 for functioning the said computer as. The input determination means determines an input to the input determination body based on a relative positional relationship between a display position of the input determination body and a display position of the pointing object;
The program according to claim 7.   The said computer is further functioned as a 1st image processing means to image-process so that the said pointing object may be exposed when the said virtual position turns into a hidden position in the said virtual space seeing from the said HMD. The program according to 7 or 8.   For further causing the computer to function as second image processing means for performing image processing so that the input determination object is exposed when the input determination object is in a hidden position in the virtual space when viewed from the HMD. The program as described in any one of Claims 2-9. The virtual space image generation means generates an image in which characters are arranged in the virtual space,
Character target operation input determining means for determining an operation input to the character based on the virtual position;
Means for invalidating the character target operation input determining means when the input determining body is displayed, and enabling the character target operation input determining means when not being displayed;
The program as described in any one of Claims 2-10 for functioning the said computer as. A coordinate system initial setting means for initial setting the coordinate system of the virtual space so as to correspond to the real space;
The program as described in any one of Claims 1-11 for making the said computer further function as. The coordinate system initial setting means refers to the position of the operation instruction object in the captured image of the imaging means, and executes the initial setting.
The program according to claim 12. The calculation means includes virtual position correction means for correcting the virtual position based on a relative positional relationship between the HMD and the operation instruction in the real space.
The program as described in any one of Claims 1-13. The virtual position correcting unit corrects the virtual position according to a correction calculation associated with the correction condition when the relative positional relationship based on the orientation of the HMD satisfies a predetermined correction condition.
The program according to claim 14. An image generation apparatus for generating an image to be displayed on the HMD, connected to an HMD mounted on a user's head and an imaging unit installed to capture an operation instruction object and a real space including the HMD Because
A calculation unit that calculates a virtual position corresponding to the position of the operation instruction in the real space in a virtual space that is a display image space of the HMD, using a captured image of the imaging unit;
An input determination object arrangement control means for arranging an input determination object at a given position in the virtual space;
Input determining means for determining an input to the input determining body based on the virtual position and an arrangement position of the input determining body;
An image generation apparatus comprising:

Images