JP2018126340A - Simulation system, program and controller - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a simulation system capable of improving a virtual reality of a user in a system using a head wearing type display device and a controller which is gripped by a user.SOLUTION: A simulation system comprises: a controller 10 comprising a grip part 20 which is gripped by a user on a bar-shaped part between a tip part and a rear end part, and comprising a generation mechanism 30 for generating at least one of reaction and vibration between one end part out of the tip part and the rear end part and the grip part 20; an information acquisition part for acquiring at least one of position information, posture information and motion information of the controller 10 in a real space; a game processing part for performing game processing; a display processing part for, based on a result of the game processing, as a display image of an HMD 200 which a user wears so as to cover a visual field, generating an image including an image of an object corresponding to the controller; and a control part for controlling the generation mechanism 30 for generating at least one of reaction and the vibration according to a state of a game which a user plays.SELECTED DRAWING: Figure 8

Description

  The present invention relates to a simulation system, a program, a controller, and the like.

  2. Description of the Related Art Conventionally, simulation systems that generate images that can be viewed from a virtual camera in a virtual space are known. For example, as a prior art of a simulation system that realizes virtual reality (VR) by displaying an image seen from a virtual camera on an HMD (head-mounted display device), there is a technique disclosed in Patent Document 1. Further, as a conventional technique of a system that provides an actuator in a game controller and performs tactile feedback by vibration, there is a technique disclosed in Patent Document 2.

JP-A-11-309269 Japanese Patent Laid-Open No. 2003-199974

  In the simulation system using the HMD, the user's field of view is covered with the HMD, and the field of view is blocked from the outside. On the other hand, an object such as an enemy character exists in the VR space that the user can see via the HMD, and the user can enjoy a battle game with the enemy character.

  However, objects such as enemy characters that exist in the virtual space that is the VR space do not exist in the real space. For this reason, for example, even if the user holds a weapon-type controller or the like and attacks an enemy character in the virtual space, the user can obtain a bodily sensation such as a hit feeling or a tactile sensation when the attack is hit. I can't. For this reason, there was a problem that the virtual reality of the user could not be improved one more time.

  According to some aspects of the present invention, it is possible to provide a simulation system, a program, a controller, and the like that can improve a user's virtual reality in a system using a head-mounted display device and a controller held by the user.

  One aspect of the present invention has a grip portion that a user grips in a rod-shaped portion between a front end portion and a rear end portion, and includes one end portion of the front end portion, the rear end portion, and the grip portion. A controller provided with a generating mechanism for generating at least one of reaction and vibration, an information acquisition unit for acquiring at least one of position information, posture information and motion information of the controller in real space, and game processing An image including an image of an object corresponding to the controller is generated as a display image of a head-mounted display device worn by the user so as to cover the field of view based on a game processing unit to be performed and a result of the game processing A simulation system comprising: a display processing unit; and a control unit that controls the generating mechanism that generates at least one of reaction and vibration according to a situation of a game played by the user. Related to.

  In one aspect of the present invention, the controller has a grip portion in a rod-shaped portion between the front end portion and the rear end portion, and the reaction and rebound between one end portion of the front end portion and the rear end portion and the grip portion. A generating mechanism for generating at least one of vibrations is provided. Then, at least one of position information, posture information, and motion information of the controller is acquired. In addition, a game process is performed, and an image including an image of an object corresponding to the controller is generated as a display image of the head-mounted display device based on the result of the game process. The generation mechanism is controlled in accordance with the state of the game played by the user, and at least one of reaction and vibration is generated by the generation mechanism. In this way, the field of view is covered by the head-mounted display device, so that the reaction or vibration corresponding to the game situation is generated by the generation mechanism of the controller for the user who does not see the real space situation, It is possible to let the user experience the reaction or vibration that has occurred, for example, via a gripping part or the like. This makes it possible to provide a simulation system or the like that can improve the virtual reality of the user in a system using a head-mounted display device and a controller held by the user.

  In the aspect of the invention, the control unit may control the generation mechanism so as to rotate the controller with the gripping unit as a rotation center.

  In this way, the controller is controlled so that the controller rotates about the gripping portion by the virtual reaction or vibration generated by the generation mechanism, so that the controller rotates as if by the actual reaction or vibration. It is possible to give a virtual reality to the user.

  In the aspect of the invention, the controller may be a sword-type controller, and the generation mechanism may be provided on a blade portion of the sword-type controller.

  In this way, even if the blade part of the sword hits another object in the virtual space, a reaction will occur in the part corresponding to the blade of the sword in the controller, as if using a real sword. It is possible to give the user a virtual reality feeling as if he / she is.

  In one aspect of the present invention, an information acquisition unit that acquires at least one of the position information, posture information, and motion information of the user in real space (a computer functions as the information acquisition unit), the control unit includes The game situation may be determined based on at least one of the position information, the posture information, and the movement information of the user, and the generation mechanism may be controlled.

  In this way, it is possible to determine the game situation based on the position information, posture information, or movement information of the user in the real space, and to generate a reaction or vibration according to the game situation by the generation mechanism.

  In the aspect of the invention, the control unit may control the generation mechanism according to a hit determination result between an object corresponding to the controller and another object.

  In this way, it is possible to let the user experience the reaction or vibration that should occur due to a hit between the object corresponding to the controller and another object, using the virtual reaction or vibration caused by the generation mechanism.

  In one aspect of the present invention, the control unit determines that the other object has been disconnected after the object corresponding to the controller hits the other object, and determines that the object has not been disconnected. The generation mode of reaction or vibration in the generation mechanism may be varied.

  In this way, after the hit occurs, depending on whether another object is cut or not, the generation mode of the reaction or vibration in the generation mechanism is different, which further enhances the virtual reality of the user. It becomes possible.

  In the aspect of the invention, the control unit may control the generation mechanism based on at least one of the controller type information, motion information, the other object type information, and motion information.

  In this way, it is possible to realize control of reaction or vibration reflecting the controller type information, motion information, other object type information, and motion information.

  In the aspect of the invention, the control unit may control at least one of reaction and vibration generated by the generation mechanism by controlling an output signal to the power source of the generation mechanism.

  In this way, it becomes possible to realize control of reaction or vibration generation by the generating mechanism by a simple control process of controlling the output signal to the power source of the generating mechanism.

  In the aspect of the invention, the generation mechanism may include a weight and a receiving portion of the weight, and may generate a reaction by causing the weight to collide with the receiving portion.

  In this way, it is possible to generate the reaction of the impact caused by the collision of the weight with the receiving part as a virtual reaction according to the game situation.

  In the aspect of the invention, the control unit may control at least one of the reaction and vibration generated by the generation mechanism by changing a stroke distance of the weight.

  In this way, by changing the stroke distance of the weight, the strength of the reaction or vibration can be adjusted.

  In the aspect of the invention, the control unit may perform control to change at least one of the type of the weight and the type of the receiving unit.

  In this way, by changing the type of the weight or the receiving portion, it is possible to control the strength of the reaction or vibration, the texture, and the like.

  In one aspect of the present invention, the controller includes a second generation mechanism that generates at least one of a reaction and a vibration, and the control unit generates the reaction or vibration in the generation mechanism, and the second The reaction mode or vibration generation mode of the generation mechanism may be different.

  By doing so, it becomes possible to realize the control of the reaction or vibration more precisely and with high virtual reality by making the generation mode in the generation mechanism different from the generation mode of the second generation mechanism.

  In another aspect of the present invention, an information acquisition unit that acquires at least one of position information, posture information, and motion information of a controller in real space, a game processing unit that performs game processing, and a result of the game processing Based on the display processing unit that generates an image including an image of an object corresponding to the controller as a display image of the head-mounted display device that the user wears so as to cover the field of view, and controls the controller The controller functions as a control unit, and the controller has a grip portion to be gripped by a user in a rod-shaped portion between the front end portion and the rear end portion, and one end portion of the front end portion and the rear end portion And a generating mechanism for generating at least one of reaction and vibration is provided between the grip and the gripping unit, and the control unit is configured to react according to the situation of the game played by the user. It is provided a program for controlling the generation mechanism that generates at least one of micro-vibration. The present invention also relates to a computer-readable information storage medium storing the above program.

  According to another aspect of the present invention, a grip portion provided at a rod-shaped portion between the front end portion and the rear end portion and gripped by a user, one end portion of the front end portion and the rear end portion, and the grip A generating mechanism that generates at least one of reaction and vibration, and the generation mechanism rotates at least one of the reaction and vibration so as to rotate the controller with the gripping part as a rotation center. Related to the controller that generates.

  In another aspect of the present invention, the user can grip the controller using a grip portion provided in a bar-shaped portion between the front end portion and the rear end portion. A reaction mechanism or vibration that rotates the controller about the gripping portion as a rotation center can be generated by a generating mechanism provided between the gripping portion gripped by the user and one end portion of the front end portion and the rear end portion. In this way, the virtual reaction or vibration generated by the generation mechanism causes the controller to rotate about the gripping portion as the center of rotation. Therefore, it is as if the controller has been rotated by the actual reaction or vibration. It is possible to give the user a sense of reality.

  In another aspect of the present invention, the generation mechanism may include a weight and a receiving portion of the weight, and may generate a reaction by causing the weight to collide with the receiving portion.

The block diagram which shows the structural example of the simulation system of this embodiment. 2A and 2B are explanatory diagrams of an example of tracking processing of the HMD. 3A and 3B are explanatory diagrams of another example of the HMD tracking process. The left view which shows the structural example of the controller of this embodiment. The right view which shows the structural example of the controller of this embodiment. The upper perspective view which shows the structural example of the controller of this embodiment. The lower perspective view which shows the structural example of the controller of this embodiment. The figure which shows the example of a user's operation | movement in the real space using a controller. The figure which shows the example of the game image produced | generated by this embodiment. The figure which shows the example of a user's operation | movement in the real space using a controller. The figure which shows the example of the game image produced | generated by this embodiment. The figure which shows the example of a user's operation | movement in the real space using a controller. The figure which shows the example of the game image produced | generated by this embodiment. The figure which shows the example of the game image produced | generated by this embodiment. The flowchart explaining the process example of this embodiment. The flowchart explaining the detailed process example of this embodiment. FIG. 17A and FIG. 17B are explanatory diagrams of a method for changing the stroke distance of the weight. 18A and 18B are explanatory diagrams of a method for changing the types of weights and receiving portions. Explanatory drawing of the method of making the generation | occurrence | production aspect of a reaction or a vibration differ by two generation | occurrence | production mechanisms.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Simulation System FIG. 1 is a block diagram illustrating a configuration example of a simulation system (a simulator, a game system, and an image generation system) according to the present embodiment. The simulation system of the present embodiment is a system that simulates virtual reality (VR), for example, a game system that provides game content, a real-time simulation system such as a sports competition simulator or a driving simulator, a system that provides an SNS service, an image The present invention can be applied to various systems such as a content providing system that provides content such as an operating system that implements remote work. Note that the simulation system of the present embodiment is not limited to the configuration shown in FIG. 1, and various modifications such as omitting some of the components (each unit) or adding other components are possible.

  The operation unit 160 is for a user (player) to input various operation information (input information). The operation unit 160 can be realized by various operation devices such as an operation button, a direction instruction key, a joystick, a handle, a pedal, a lever, or a voice input device.

  The operation unit 160 includes the controller 10 (user interface). Details of the controller 10 will be described later.

  The storage unit 170 stores various types of information. The storage unit 170 functions as a work area such as the processing unit 100 or the communication unit 196. The game program and game data necessary for executing the game program are held in the storage unit 170. The function of the storage unit 170 can be realized by a semiconductor memory (DRAM, VRAM), HDD (Hard Disk Drive), SSD, optical disk device, or the like. The storage unit 170 includes an object information storage unit 172 and a drawing buffer 178.

  The information storage medium 180 (a computer-readable medium) stores programs, data, and the like, and its function can be realized by an optical disk (DVD, BD, CD), HDD, semiconductor memory (ROM), or the like. . The processing unit 100 performs various processes of the present embodiment based on a program (data) stored in the information storage medium 180. That is, in the information storage medium 180, a program for causing a computer (an apparatus including an input device, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute processing of each unit). Is memorized.

  The HMD 200 (head-mounted display device) is a device that is mounted on the user's head and displays an image in front of the user's eyes. The HMD 200 is preferably a non-transmissive type, but may be a transmissive type. The HMD 200 may be a so-called glasses-type HMD.

  The HMD 200 includes a sensor unit 210, a display unit 220, and a processing unit 240. A modification in which a light emitting element is provided in the HMD 200 is also possible. The sensor unit 210 is for realizing tracking processing such as head tracking, for example. For example, the position and direction of the HMD 200 are specified by tracking processing using the sensor unit 210. By specifying the position and direction of the HMD 200, the user's viewpoint position and line-of-sight direction can be specified.

  Various methods can be adopted as the tracking method. In the first tracking method, which is an example of the tracking method, a plurality of light receiving elements (photodiodes and the like) are provided as the sensor unit 210, as will be described in detail with reference to FIGS. 2A and 2B described later. Then, by receiving light (laser or the like) from a light emitting element (LED or the like) provided outside by the plurality of light receiving elements, the position of the HMD 200 (user's head) in the real world three-dimensional space, Identify the direction. In the second tracking method, a plurality of light emitting elements (LEDs) are provided in the HMD 200 as will be described in detail with reference to FIGS. 3A and 3B described later. And the position and direction of HMD200 are pinpointed by imaging the light from these light emitting elements with the imaging part provided outside. In the third tracking method, a motion sensor is provided as the sensor unit 210, and the position and direction of the HMD 200 are specified using this motion sensor. The motion sensor can be realized by, for example, an acceleration sensor or a gyro sensor. For example, by using a 6-axis motion sensor using a 3-axis acceleration sensor and a 3-axis gyro sensor, the position and direction of the HMD 200 in a three-dimensional space in the real world can be specified. Note that the position and direction of the HMD 200 may be specified by a combination of the first tracking method and the second tracking method, or a combination of the first tracking method and the third tracking method. Further, instead of specifying the user's viewpoint position and line-of-sight direction by specifying the position and direction of the HMD 200, tracking processing that directly specifies the user's viewpoint position and line-of-sight direction may be employed.

  The display unit 220 of the HMD 200 can be realized by, for example, an organic EL display (OEL) or a liquid crystal display (LCD). For example, the display unit 220 of the HMD 200 includes a first display or first display area set in front of the user's left eye and a second display or second display area set in front of the right eye. It is provided and stereoscopic display is possible. When performing stereoscopic display, for example, a left-eye image and a right-eye image with different parallax are generated, a left-eye image is displayed on the first display, and a right-eye image is displayed on the second display. To do. Alternatively, the left-eye image is displayed in the first display area of one display, and the right-eye image is displayed in the second display area. The HMD 200 is provided with two eyepiece lenses (fisheye lenses) for the left eye and the right eye, thereby expressing a VR space that extends over the entire perimeter of the user's field of view. Then, correction processing for correcting distortion generated in an optical system such as an eyepiece is performed on the left-eye image and the right-eye image. This correction process is performed by the display processing unit 120.

  The processing unit 240 of the HMD 200 performs various processes necessary for the HMD 200. For example, the processing unit 240 performs control processing of the sensor unit 210, display control processing of the display unit 220, and the like. Further, the processing unit 240 may perform a three-dimensional sound (stereoscopic sound) process to realize reproduction of a three-dimensional sound direction, distance, and spread.

  The sound output unit 192 outputs the sound generated by the present embodiment, and can be realized by, for example, a speaker or headphones.

  The I / F (interface) unit 194 performs interface processing with the portable information storage medium 195, and the function can be realized by an ASIC for I / F processing or the like. The portable information storage medium 195 is for a user to save various types of information, and is a storage device that retains storage of such information even when power is not supplied. The portable information storage medium 195 can be realized by an IC card (memory card), a USB memory, a magnetic card, or the like.

  The communication unit 196 communicates with the outside (another apparatus) via a wired or wireless network, and functions thereof are hardware such as a communication ASIC or communication processor, or communication firmware. Can be realized.

  A program (data) for causing a computer to function as each unit of this embodiment is distributed from the information storage medium of the server (host device) to the information storage medium 180 (or storage unit 170) via the network and communication unit 196. May be. Use of an information storage medium by such a server (host device) can also be included in the scope of the present invention.

  The processing unit 100 (processor) is used for operation information from the operation unit 160, tracking information in the HMD 200 (information on at least one of the position and direction of the HMD, information on at least one of the viewpoint position and the line-of-sight direction), a program, and the like. Based on this, game processing (simulation processing), virtual space setting processing, moving body processing, virtual camera control processing, display processing, sound processing, or the like is performed.

  Each processing (each function) of this embodiment performed by each unit of the processing unit 100 can be realized by a processor (a processor including hardware). For example, each process of the present embodiment can be realized by a processor that operates based on information such as a program and a memory that stores information such as a program. In the processor, for example, the function of each unit may be realized by individual hardware, or the function of each unit may be realized by integrated hardware. For example, the processor may include hardware, and the hardware may include at least one of a circuit that processes a digital signal and a circuit that processes an analog signal. For example, the processor can be configured by one or a plurality of circuit devices (for example, ICs) mounted on a circuit board or one or a plurality of circuit elements (for example, resistors, capacitors, etc.). The processor may be, for example, a CPU (Central Processing Unit). However, the processor is not limited to the CPU, and various processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor) can be used. The processor may be an ASIC hardware circuit. The processor may include an amplifier circuit, a filter circuit, and the like that process an analog signal. The memory (storage unit 170) may be a semiconductor memory such as SRAM or DRAM, or may be a register. Alternatively, it may be a magnetic storage device such as a hard disk device (HDD) or an optical storage device such as an optical disk device. For example, the memory stores instructions that can be read by a computer, and the processing (function) of each unit of the processing unit 100 is realized by executing the instructions by the processor. The instruction here may be an instruction set constituting a program, or an instruction for instructing an operation to the hardware circuit of the processor.

  The processing unit 100 includes an input processing unit 102, an arithmetic processing unit 110, and an output processing unit 140. The arithmetic processing unit 110 includes an information acquisition unit 111, a virtual space setting unit 112, a moving body processing unit 113, a virtual camera control unit 114, a game processing unit 115, a hit determination unit 116, a control unit 117, a display processing unit 120, and sound processing. Part 130. As described above, each process of the present embodiment executed by these units can be realized by a processor (or a processor and a memory). Various modifications such as omitting some of these components (each unit) or adding other components are possible.

  The input processing unit 102 performs processing for receiving operation information and tracking information, processing for reading information from the storage unit 170, and processing for receiving information via the communication unit 196 as input processing. For example, the input processing unit 102 stores, in the storage unit 170, processing for acquiring operation information input by the user using the operation unit 160, tracking information detected by the sensor unit 210 of the HMD 200, and the information specified by the read command. A process for reading data from a computer or a process for receiving information from an external device (such as a server) via a network is performed as an input process. Here, the reception process includes a process of instructing the communication unit 196 to receive information, a process of acquiring information received by the communication unit 196, and writing the information in the storage unit 170, and the like.

  The arithmetic processing unit 110 performs various arithmetic processes. For example, arithmetic processing such as information acquisition processing, virtual space setting processing, moving body processing, virtual camera control processing, game processing (simulation processing), display processing, or sound processing is performed.

  The information acquisition unit 111 (program module for information acquisition processing) performs various information acquisition processing. For example, the information acquisition unit 111 acquires position information of the user wearing the HMD 200 and the controller 10. The information acquisition unit 111 may acquire posture information (direction information), movement information, and the like of the user and the controller 10.

  The virtual space setting unit 112 (program module for virtual space setting processing) performs setting processing for a virtual space (object space) in which an object is placed. For example, various objects representing display objects such as moving objects (people, robots, cars, trains, airplanes, ships, monsters, animals, etc.), maps (terrain), buildings, auditoriums, courses (roads), trees, walls, water surfaces, etc. A process of placing and setting an object (an object composed of a primitive surface such as a polygon, a free-form surface, or a subdivision surface) in a virtual space is performed. In other words, the position and rotation angle of the object in the world coordinate system (synonymous with direction and direction) are determined, and the rotation angle (rotation angle around the X, Y, and Z axes) is determined at that position (X, Y, Z). Arrange objects. Specifically, in the object information storage unit 172 of the storage unit 170, object information that is information such as the position, rotation angle, moving speed, and moving direction of an object (part object) in the virtual space is associated with the object number. Is remembered. The virtual space setting unit 112 performs a process of updating the object information for each frame, for example.

  The moving body processing unit 113 (moving body processing program module) performs various processes on a moving body that moves in the virtual space. For example, processing for moving the moving body in virtual space (object space, game space) and processing for moving the moving body are performed. For example, the mobile object processing unit 113 is based on operation information input by the user through the operation unit 160, acquired tracking information, a program (movement / motion algorithm), various data (motion data), and the like. Control processing for moving the model object) in the virtual space or moving the moving object (motion, animation) is performed. Specifically, a simulation for sequentially obtaining movement information (position, rotation angle, speed, or acceleration) and motion information (part object position or rotation angle) of a moving body for each frame (for example, 1/60 second). Process. A frame is a unit of time for performing a moving / movement process (simulation process) and an image generation process of a moving object. The moving body is, for example, a user moving body corresponding to a user (player) in real space. The user moving body is a virtual user (virtual player, avatar) in the virtual space, or a boarding moving body (operation moving body) on which the virtual user is boarded (operated).

  The virtual camera control unit 114 (virtual camera control processing program module) controls the virtual camera. For example, a process for controlling the virtual camera is performed based on user operation information, tracking information, and the like input by the operation unit 160.

  For example, the virtual camera control unit 114 controls the virtual camera set as the first-person viewpoint or the third-person viewpoint of the user. For example, by setting a virtual camera at a position corresponding to the viewpoint (first-person viewpoint) of a user moving body moving in the virtual space, and setting the viewpoint position and line-of-sight direction of the virtual camera, the position (positional coordinates) of the virtual camera And control the attitude (rotation angle around the rotation axis). Alternatively, the position and orientation of the virtual camera are controlled by setting the virtual camera at the position of the viewpoint (third-person viewpoint) following the user moving body and setting the viewpoint position and line-of-sight direction of the virtual camera.

  For example, the virtual camera control unit 114 controls the virtual camera so as to follow the change of the user's viewpoint based on the tracking information of the user's viewpoint information acquired by the viewpoint tracking. For example, in the present embodiment, tracking information (viewpoint tracking information) of viewpoint information that is at least one of the user's viewpoint position and line-of-sight direction is acquired. This tracking information can be acquired by performing tracking processing of the HMD 200, for example. Then, the virtual camera control unit 114 changes the viewpoint position and the line-of-sight direction of the virtual camera based on the acquired tracking information (information on at least one of the user's viewpoint position and the line-of-sight direction). For example, the virtual camera control unit 114 is configured so that the viewpoint position and the line-of-sight direction (position and posture) of the virtual camera change in the virtual space according to changes in the viewpoint position and the line-of-sight direction of the user in the real space. Set up the camera. By doing in this way, a virtual camera can be controlled to follow a user's viewpoint change based on tracking information of a user's viewpoint information.

  The game processing unit 115 (game processing program module) performs various game processes for the user to play a game. In other words, the game processing unit 115 (simulation processing unit) executes various simulation processes for the user to experience virtual reality (virtual reality). The game process is, for example, a process for starting a game when a game start condition is satisfied, a process for advancing the started game, a process for ending a game when a game end condition is satisfied, or calculating a game result. Processing.

  The hit determination unit 116 (a program module for hit determination processing) performs hit determination processing. The control unit 117 (control processing program module) performs various control processes for the controller 10. Details of the hit determination unit 116 and the control unit 117 will be described later.

  The display processing unit 120 (display processing program module) performs display processing of a game image (simulation image). For example, a drawing process is performed based on the results of various processes (game process, simulation process) performed by the processing unit 100, thereby generating an image and displaying it on the display unit 220. Specifically, geometric processing such as coordinate transformation (world coordinate transformation, camera coordinate transformation), clipping processing, perspective transformation, or light source processing is performed. Based on the processing result, drawing data (the position of the vertex of the primitive surface) Coordinates, texture coordinates, color data, normal vector, α value, etc.) are created. Based on the drawing data (primitive surface data), the object (one or a plurality of primitive surfaces) after perspective transformation (after geometry processing) is converted into image information in units of pixels such as a drawing buffer 178 (frame buffer, work buffer, etc.). Draw in a buffer that can be stored. As a result, an image that can be seen from the virtual camera (given viewpoints, the first and second viewpoints for the left eye and the right eye) in the virtual space is generated. Note that the drawing processing performed by the display processing unit 120 can be realized by vertex shader processing, pixel shader processing, or the like.

  The sound processing unit 130 (sound processing program module) performs sound processing based on the results of various processes performed by the processing unit 100. Specifically, game sounds such as music (music, BGM), sound effects, or sounds are generated, and the game sounds are output to the sound output unit 192. Note that part of the sound processing of the sound processing unit 130 (for example, three-dimensional sound processing) may be realized by the processing unit 240 of the HMD 200.

  The output processing unit 140 performs various types of information output processing. For example, the output processing unit 140 performs processing for writing information in the storage unit 170 and processing for transmitting information via the communication unit 196 as output processing. For example, the output processing unit 140 performs a process of writing information specified by a write command in the storage unit 170 or a process of transmitting information to an external apparatus (server or the like) via a network. The transmission process is a process of instructing the communication unit 196 to transmit information, or instructing the communication unit 196 to transmit information.

  As shown in FIG. 1, the simulation system of the present embodiment includes an information acquisition unit 111 that acquires user position information in real space. For example, the information acquisition unit 111 acquires user position information through user viewpoint tracking or the like. Then, the moving body processing unit 113 performs processing for moving the user moving body (user character, avatar) based on the acquired position information, and the display processing unit 120 generates a display image of the HMD 200 worn by the user. . For example, the user moving body in the virtual space is moved so as to follow the movement of the user in the real space. Then, an image that can be seen from the virtual camera corresponding to the user moving object is generated as a display image of the HMD 200.

  For example, the information acquisition unit 111 acquires position information of a user who wears the HMD 200 so as to cover the field of view. For example, the information acquisition unit 111 acquires the position information of the user in real space based on the tracking information of the HMD 200 and the like. For example, the position information of the HMD 200 is acquired as the position information of the user wearing the HMD 200. Specifically, when the user is located in a play field (simulation field, play area) in real space (real world), position information in the play field is acquired. Note that the position information of the user may be acquired by a method of directly tracking a part such as the user or the user's head instead of the tracking process of the HMD 200.

  Further, the virtual camera control unit 114 controls the virtual camera so as to follow the change in the user's viewpoint based on the tracking information of the user's viewpoint information.

  For example, the input processing unit 102 (input reception unit) acquires tracking information of viewpoint information of the user wearing the HMD 200. For example, tracking information (viewpoint tracking information) of viewpoint information that is at least one of the user's viewpoint position and line-of-sight direction is acquired. This tracking information can be acquired by performing tracking processing of the HMD 200, for example. Note that the user's viewpoint position and line-of-sight direction may be directly acquired by tracking processing. As an example, the tracking information includes the change information of the viewpoint position from the initial viewpoint position of the user (change value of the coordinate of the viewpoint position), and the change information of the gaze direction from the initial gaze direction of the user (the rotation axis in the gaze direction). At least one of the rotation angle change values around the rotation angle). Based on the change information of the viewpoint information included in such tracking information, the user's viewpoint position and line-of-sight direction (information on the user's head position and posture) can be specified.

  In the present embodiment, a virtual reality simulation process is performed as the game process of the game played by the user. The virtual reality simulation process is a simulation process for simulating an event in the real space in the virtual space, and is a process for causing the user to experience the event virtually. For example, a virtual user corresponding to a user in real space or a moving body such as a boarding moving body is moved in the virtual space, or processing for causing the user to experience changes in the environment and surroundings associated with the movement is performed.

  The processing of the simulation system of this embodiment in FIG. 1 includes processing devices such as home game devices and arcade game devices, processing devices such as PCs installed in facilities, and processing devices (back Pack PC) or distributed processing of these processing devices. Or you may implement | achieve the process of the simulation system of this embodiment with a server system and a terminal device. For example, it may be realized by distributed processing of a server system and a terminal device.

2. Tracking Processing Next, an example of tracking processing will be described. FIG. 2A shows an example of the HMD 200 used in the simulation system of this embodiment. As shown in FIG. 2A, the HMD 200 is provided with a plurality of light receiving elements 201, 202, and 203 (photodiodes). The light receiving elements 201 and 202 are provided on the front side of the HMD 200, and the light receiving element 203 is provided on the right side of the HMD 200. In addition, a light receiving element (not shown) is also provided on the left side, upper surface, and the like of the HMD.

  The user US has the controller 10 with one hand (for example, the right hand) and the controller 80 with the other hand (for example, the left hand). The controller 10 is a sword-type controller, for example. The controller 80 is a controller corresponding to a shield described later. The controllers 10 and 80 are provided with light receiving elements in the same manner as the HMD 200. By using the light receiving elements, the tracking processing of the controllers 10 and 80 can be realized as in the case of the HMD 200. For example, the position information of the controllers 10 and 80 can be detected. Further, by providing the controllers 10 and 80 with a plurality of light receiving elements, the posture information (direction information) of the controllers 10 and 80 can be detected.

  As shown in FIG. 2B, base stations 280 and 284 are installed around the user US. The base station 280 is provided with light emitting elements 281 and 282, and the base station 284 is provided with light emitting elements 285 and 286. The light emitting elements 281, 282, 285, and 286 are realized by LEDs that emit laser (infrared laser or the like), for example. The base stations 280 and 284 use these light emitting elements 281, 282, 285, and 286 to emit, for example, a laser beam radially. Then, the light receiving elements 201 to 203 and the like provided in the HMD 200 in FIG. 2A receive the lasers from the base stations 280 and 284, thereby realizing the tracking process of the HMD 200 and facing the head position of the user US. The direction (viewpoint position, line-of-sight direction) can be detected. For example, position information and posture information (direction information) of the user US can be detected. Further, by providing the controllers 10 and 80 with light receiving elements, the position information and orientation information (direction information) of the controllers 10 and 80 can be detected.

  FIG. 3A shows another example of the HMD 200. In FIG. 3A, a plurality of light emitting elements 231 to 236 are provided for the HMD 200. These light emitting elements 231 to 236 are realized by, for example, LEDs. The light emitting elements 231 to 234 are provided on the front side of the HMD 200, and the light emitting element 235 and the light emitting element 236 (not shown) are provided on the back side. These light emitting elements 231 to 236 emit (emit) light in a visible light band, for example. Specifically, the light emitting elements 231 to 236 emit light of different colors.

  The imaging unit 150 shown in FIG. 3B is installed in at least one place around the user US (for example, the front side, the front side, the rear side, or the like). ˜236 images of light. That is, the spot light of these light emitting elements 231 to 236 is reflected in the captured image of the imaging unit 150. And the tracking of the user's US head (HMD) is implement | achieved by performing the image process of this captured image. That is, the three-dimensional position of the head of the user US and the facing direction (viewpoint position, line of sight direction) are detected.

  For example, as illustrated in FIG. 3B, the imaging unit 150 includes first and second cameras 151 and 152, and the first and second cameras 151 and 152 of the first and second cameras 151 and 152 are provided. By using the captured image, the position of the head of the user US in the depth direction can be detected. Further, based on the motion detection information of the motion sensor provided in the HMD 200, the rotation angle (line of sight) of the head of the user US can also be detected. Therefore, by using such an HMD 200, when the user US faces in any direction of all 360 degrees around the image, the image (the user's virtual space in the virtual space (virtual three-dimensional space)) Image viewed from a virtual camera corresponding to the viewpoint) can be displayed on the display unit 220 of the HMD 200.

  Note that instead of visible light, infrared LEDs may be used as the light emitting elements 231 to 236. Further, the position or movement of the user's head may be detected by another method such as using a depth camera.

  Further, by providing the light emitting elements in the controllers 10 and 80, the tracking processing of the controllers 10 and 80 can be realized as in the case of the HMD200. For example, the position information of the controllers 10 and 80 can be detected. In addition, by providing the controllers 10 and 80 with a plurality of light emitting elements, posture information (direction information) of the controllers 10 and 80 can be detected.

  Further, tracking processing methods for detecting the position information and posture information (viewpoint position, line-of-sight direction) of the user US and detecting the position information and posture information of the controllers 10 and 80 are shown in FIGS. The method described in B) is not limited. For example, the tracking information of the HMD 200 and the controllers 10 and 80 may be performed using a motion sensor provided in the HMD 200 and the controllers 10 and 80 to detect position information and posture information. That is, tracking processing is realized without providing external devices such as the base stations 280 and 284 in FIG. 2B and the imaging unit 150 in FIG. Alternatively, various viewpoint tracking methods such as known eye tracking, face tracking, or head tracking may be used.

3. Controller Next, a configuration example of the controller 10 according to the present embodiment will be described with reference to FIGS. 4 is a left side view of the controller 10, and FIG. 5 is a right side view. 6 is a perspective view of the controller 10 as viewed from above, and FIG. 7 is a perspective view of the controller 10 as viewed from below.

  In addition, the shape and structure of the controller 10 are not limited to FIGS. For example, FIG. 4 to FIG. 7 are examples of a sword-type controller imitating a sword. As the controller 10, a weapon-type controller imitating a weapon other than a sword, or a sports instrument imitating an instrument used in sports. It is possible to envisage a type controller or a musical instrument type controller imitating musical instruments such as musical instruments and musical instrument tools. As a weapon-type controller other than a sword, a controller imitating an axe, spear, wand or whip can be assumed. As a sports equipment type controller, a bat type controller imitating a baseball bat, a racket type controller imitating a tennis or table tennis racket, a club type controller imitating a golf club, and the like can be assumed. As a musical instrument type controller, for example, a stick type controller imitating a drum or drumstick can be assumed. Further, the controller 10 may be a controller of a type other than a weapon type, a sports equipment type, or a musical instrument type.

  As shown in FIGS. 4 to 7, the controller 10 (user interface) of the present embodiment has a grip portion 20 (handle) that the user grips at a bar-shaped portion between the front end portion and the rear end portion. For example, the controller 10 has at least a grip portion 20 in a bar shape so that the user can grip it with one hand, for example. For example, when the longitudinal direction of the controller 10 is DR1, the grip portion 20 has a bar shape in which the direction DR1 is the longitudinal direction. The direction DR1 is also a direction connecting the front end portion and the rear end portion of the controller 10, for example. 4 to 7, the direction orthogonal to the direction DR1 is DR2, and the direction orthogonal to the direction DR1 and the direction DR2 is DR3.

  It suffices that the controller 10 has a bar shape at least in the grip portion 20, and other parts do not necessarily have a bar shape. For example, in the case of a racket-type controller used in games such as tennis, the grip portion gripped by the user has a rod shape, while the frame portion such as a racket face does not have to have a rod shape. In the case of a club-type controller used in a golf game, the grip portion has a rod shape, while the head portion does not have to have a rod shape. Similarly, in the case of a weapon-type controller imitating an axe, the grip portion has a rod shape, while the ax blade portion does not have to have a rod shape. Further, like the baseball bud, the stick-shaped gripping portion 20 may be one in which the thickness of the grip slightly changes. As described above, the controller 10 may have a rod shape at least in the grip portion 20. The rod-shaped cross section is not limited to a circle or an ellipse, and may be a polygonal shape such as a triangle or a quadrangle. Also good.

  In the controller 10 of this embodiment, a generating mechanism that generates at least one of reaction and vibration is provided between one end of the front end and the rear end and the grip 20. For example, in the controller 10 of FIGS. 4 to 7, a generating mechanism 30 that generates, for example, a reaction is provided between the distal end portion and the grip portion 20. Further, a generating mechanism 60 (second generating mechanism) that generates vibration or the like is provided between the rear end portion of the controller 10 and the grip portion 20. For example, a generation mechanism that generates vibration or a generation mechanism that generates both reaction and vibration may be provided between the tip of the controller 10 and the grip 20. Alternatively, for example, a generation mechanism that generates a reaction or a generation mechanism that generates both the reaction and vibration may be provided between the rear end of the controller 10 and the grip 20. Or you may provide the generating mechanism which generate | occur | produces at least one of a reaction and a vibration in the location of the holding part 20, for example.

  Specifically, the generating mechanism 30 includes a weight 32 and a receiving part 34. The generating mechanism 30 generates a reaction (reaction force) by causing the weight 32 to collide with the receiving portion 34. In addition, vibration can also be generated along with the reaction. The reaction is, for example, being pushed back by the reaction when another force or action is applied. For example, as will be described later with reference to FIGS. 8 to 14, in the present embodiment, the user holds the sword-shaped controller 10 in real space and performs an operation of swinging with a sword. When a sword object corresponding to the sword hits another object such as an enemy character in the virtual space, a virtual reaction to the hit is realized by the generation mechanism 30. For example, the generating mechanism 30 generates a reaction that can be felt by the user as if a reaction force in a direction opposite to the direction in which the sword hits. Specifically, the weight 32 swings and collides with the receiving portion 34 as indicated by A1 in FIG. As a result, the user can feel as if the reaction force shown in A2 has occurred, and a virtual reaction caused by a sword hit can be realized.

  For example, the weight 32 of the generating mechanism 30 is attached to the arm 40. As shown in FIG. 7, the arm 40 swings along an opening provided in the guide portion 41. For example, the controller 10 has a base 12, and a plate-like member 47 is attached to the base 12. A weight 32 is attached to one end of the arm 40, and the other end of the arm 40 is rotatably attached to a plate-like member 47 by a fixture 42 (shaft member). Further, a solenoid 50 (power source or actuator in a broad sense) is attached to the base portion 12, and the solenoid 50 moves the shaft member 52 along the direction DR1. The other end side of the arm 40 is attached to the head portion 54 of the shaft member 52 by a fixture 44 (shaft member). The other end of the elastic member 46 (spring, wire, etc.) having one end attached to the base portion 12 is also attached to the head portion 54 of the shaft member 52 by the attachment tool 44. Then, when the shaft member 52 is moved in the direction DR1 as indicated by A3 in FIG. 4 by the solenoid 50, the arm 40 rotates as indicated by A4 with the position of the fixture 42 as the rotation center. As a result, the weight 32 swings as shown by A1, collides with the receiving portion 34, and a virtual reaction as shown by A2 occurs. After the collision, for example, the weight 32 and the arm 40 are returned to their original positions by the elastic force of the elastic member 46.

  A generation mechanism 60 that is provided between the rear end of the controller 10 and the grip 20 and generates vibration includes a transducer 62 (vibration device). Specifically, a plate member 64 is attached to the base portion 12, and a transducer 62 is attached to the plate member 64. The transducer 62 converts a sound signal and corresponds to a high-power subwoofer. For example, when the processing unit 100 (control unit 117) in FIG. 1 performs a sound file reproduction process, a sound signal is input to the transducer 62. For example, vibration is generated based on a low frequency component of the sound signal. A vibration motor may be provided as a vibration device instead of the transducer 62. For example, the vibration motor generates vibration by rotating an eccentric weight. Specifically, eccentric weights are attached to both ends of the drive shaft (rotor shaft) so that the motor itself swings.

  A controller 70 for tracking processing is attached to the controller 10. The controller 70 is a controller of the same type as the controller 80 held by the user US in FIGS. 2 (A) and 3 (A). In the tracking method of FIGS. 2A and 2A, a light receiving element is provided for the controller 70, and the position information of the controller 10 (70) is obtained by tracking processing using the light receiving element. And posture information are detected. In the tracking method of FIGS. 3A and 3B, a light emitting element is provided for the controller 70, and the position information of the controller 10 (70) is obtained by tracking processing using the light emitting element. And posture information are detected.

4). Next, the method of this embodiment will be described. Hereinafter, a case where the method of the present embodiment is applied to a battle game using a sword (hereinafter referred to as a sword game) will be mainly described. That is, an application example to a sword game using the sword-type controller 10 described in FIGS. 4 to 7 will be described. However, the game to which the method of the present embodiment is applied is not limited to such a sword game. For example, the method of the present embodiment is a variety of games such as a battle game other than a sword game (virtual experience game, RPG, action game, competition game, sports game, horror experience game, simulation game of vehicles such as trains and airplanes, puzzles, etc. Game, communication game, music game, etc.) and can be applied to other than games.

4.1 Description of Game First, a sword game realized by the method of this embodiment will be described with reference to FIGS. As shown in FIG. 8, the user US wears the HMD 200 so as to cover the field of view. For example, the sword-shaped controller 10 is held with the right hand and the controller 80 is held with the left hand. Specifically, the user US has the controller 10 like a sword by holding the grip 20 (hand) with the right hand. The direction opposite to the direction (A1) in which the weight 32 of the generating mechanism 30 described in FIG. 4 moves is the direction in which the blade of the sword-shaped controller 10 faces. The position information and orientation information of the controllers 10 and 80 are acquired by the tracking processing described with reference to FIGS.

  FIG. 9 shows an example of a game image displayed on the HMD 200. The game image displays objects such as an enemy character OBE, a sword SWE possessed by the enemy character OBE, a shield SLE, and a background. Also displayed are objects such as the right hand HR and left hand HL of the user character (avatar) corresponding to the user US, the sword SW held by the right hand HR, and the shield SL held by the left hand HL.

  For example, when the user US moves the controller 10 held by the right hand in the real space of FIG. 8, the right hand HR and the sword SW in the virtual space of FIG. 9 also move in conjunction with the movement. When the user US moves the controller 80 held by the left hand in the real space of FIG. 8, the left hand HL and shield SL in the virtual space of FIG. 9 also move in conjunction with the movement. That is, the real space controller 10 corresponds to the virtual space sword SW, and the real space controller 80 corresponds to the virtual space shield SL.

  FIG. 9 is a first-person game image, in which an image from the viewpoint of a user character (user moving body in a broad sense) in a virtual space corresponding to the user US in the real space is displayed. In practice, the HMD 200 displays an image of the VR space that extends over the entire periphery of the field of view to the user US.

  In FIG. 10, the user US swings down the sword-shaped controller 10 from the top to the bottom like an actual sword as an operation of cutting the enemy character OBE of FIG. 9 with the sword SW. As a result, as shown in FIG. 11, the sword SW in the virtual space hits the shield SLE of the enemy character OBE, and an effect such as a spark is displayed.

  When it is determined that the sword SW hits the shield SLE or the like as described above, the weight 32 of the generating mechanism 30 swings and collides with the receiving portion 34 as shown by B1 in FIG. For example, when the solenoid 50 (power source, actuator) of FIG. 4 moves the shaft member 52 as indicated by A3, the weight 32 swings and collides with the receiving portion 34. As a result, a virtual reaction (reaction force) as shown in B2 of FIG. 10 occurs, and the user US can experience a hit feeling and a tactile sensation as if his / her own sword actually hit the enemy character OBE. .

  In this case, in the present embodiment, the generation mechanism 60 vibrates as shown in B3 with a time delay from the occurrence of the reaction caused by the hit shown in B2. That is, vibration is generated by the transducer 62. As a result, the user US can feel a slight vibration following a large reaction at the time of a sword hit, and the virtual reality can be greatly improved.

  In FIG. 12, the user US further swings the controller 10 and continues the attack with the sword. As shown in FIG. 13, the sword SW in the virtual space hits the body of the enemy character OBE, and an effect such as a spark is generated. doing. Also in this case, as shown by C1 in FIG. 12, when the weight 32 of the generating mechanism 30 swings and collides with the receiving portion 34, a virtual reaction (reaction force) as shown by C2 occurs. Then, with a delay in time from the occurrence of the reaction, the transducer 62 of the generating mechanism 60 vibrates as indicated by C3. Then, when such an attack on the enemy character OBE is performed several times and the hit point of the enemy character OBE becomes 0, the enemy character OBE can be defeated as shown in FIG.

  In addition, when the sword SW hits the shield SLE as shown in FIG. 11 and when the sword SW hits the enemy character OBE as shown in FIG. It is desirable to vary the generation mode. That is, depending on the type of the hit object, the generation mode of the reaction or vibration by the generation mechanisms 30 and 60 is varied. By doing so, the virtual reality of the user US can be further improved.

4.2 Generation of Reaction and Vibration In order to realize the virtual reality in the sword game as described above, the simulation system of the present embodiment includes a controller 10, an information acquisition unit 111, a game processing unit as shown in FIG. 115, a display processing unit 120, and a control unit 117. The simulation system can also include an HMD 200. Various modifications such as omitting some of these components (each unit) or adding other components are possible.

  And as demonstrated in FIG. 4, the controller 10 has the holding part 20 which a user hold | grips in the rod-shaped part between a front-end | tip part and a rear-end part. A generating mechanism that generates at least one of reaction and vibration is provided between one end of the front end and the rear end and the grip 20. In FIG. 4, a generating mechanism 30 for generating a reaction is provided.

  The information acquisition unit 111 acquires at least one of position information, posture information, and motion information of the controller 10 in real space. For example, at least one of position information, posture information, and motion information of the controller 10 is acquired by the tracking process described with reference to FIGS. Alternatively, at least one of position information, posture information, and motion information of the controller 10 may be acquired by a motion sensor (an acceleration sensor, a gyro sensor, or the like) provided in the controller 10. Alternatively, at least one of position information, posture information, and motion information of the controller 10 may be acquired by photographing the controller 10 with an external camera. By acquiring the position information, posture information, or motion information of the controller 10, an image of an object corresponding to the controller 10 can be displayed. For example, as shown in FIG. 9, an image of a sword SW that is an object corresponding to the controller 10 can be displayed. In addition, by acquiring the position information, posture information, or movement information of the controller 10, hit determination between an object (such as a sword SW) corresponding to the controller 10 and another object (such as an enemy character OBE, shield SLE, or sword SWE) can be performed. It becomes possible. The information acquisition unit 111 acquires at least one of position information, posture information, and motion information of the controller 80. Thereby, an image of the object corresponding to the controller 80 (shield SL in FIG. 9) is displayed, or hit determination between the object corresponding to the controller 80 and another object (enemy character OBE, shield SLE, sword SWE, etc.) is performed. It becomes possible.

  The game processing unit 115 performs game processing. For example, game processing such as game start processing, game progress processing, game end processing, or game score calculation processing is performed. Based on the result of the game process, the display processing unit 120 displays an image of an object (such as a sword SW) corresponding to the controller 10 as a display image of the HMD 200 (head-mounted display device) worn by the user so as to cover the field of view. An image including is generated. For example, game images as shown in FIGS. 9, 11, 13, and 14 are generated as display images of the HMD 200. For example, a stereoscopic left eye image and right eye image displayed on the display unit 220 of the HMD 200 are generated as display images.

  The control unit 117 controls the generation mechanisms 30 and 60 that generate at least one of reaction and vibration according to the situation of the game played by the user. For example, a reaction or vibration is generated in the generation mechanisms 30 and 60 according to the game situation, or a reaction or vibration is generated in the generation mechanisms 30 and 60 in a generation manner corresponding to the game situation. The game situation includes, for example, a user game play situation, a game progress situation, a game battle situation, a game battle situation, a game hit judgment situation, a character (moving body) situation in the game, a character and a user Status parameter status or game performance status, for example, various statuses represented by game parameters. The game situation is specified by, for example, game processing in the game processing unit 115. For example, the user's game situation is specified based on information such as various game parameters used in the game process.

  For example, as shown in FIGS. 11 and 13, when a game situation occurs in which the user's sword SW hits the enemy shield SLE, the sword SWE, or the enemy character OBE, a reaction is generated by the generating mechanisms 30 and 60. Or generate vibration. In addition, depending on the game situation, the generation mode of reaction or vibration in the generation mechanisms 30 and 60 is varied. For example, the occurrence mode such as the strength of reaction or vibration, the generation timing or the generation period is varied depending on the game situation. For example, in the game situation where the sword SW hits the enemy shield SLE as shown in FIG. 11 and the game situation where the sword SW hits the enemy character OBE as shown in FIG. The generation mode (strength, timing, period, etc.) is varied.

  For example, in this embodiment, the user wears the HMD 200 so as to cover the field of view, and the user's field of view in real space is blocked. Because the HMD 200 is in a state where the real space situation is not visible in this way, the user can feel the feeling of cutting the object due to the reaction and vibration when cutting with the sword by the reaction and vibration generated by the controller 10. become.

  Here, the reaction or vibration of the controller 10 depending on the game situation means that the object (enemy etc.) in the virtual space and the object such as the sword SW corresponding to the controller 10 are in the virtual space. For example, reaction or vibration is generated in accordance with the result of the hit determination process determined based on the result. In this case, as described with reference to FIGS. 10 and 12, the generation mechanism 30 generates a reaction or the like in the reaction direction with respect to the operation direction of the controller 10 (sword). For example, in FIGS. 10 and 12, the user swings down the controller 10 in the downward direction, and in this case, as shown in B2 and C2, a reaction or the like is generated in the upward direction that is the direction of the reaction at the time of hit. . By doing so, it becomes possible for the user to experience the feeling of cutting the object due to the reaction and vibration when cutting with the sword.

  Further, the control unit 117 in FIG. 1 controls the generating mechanism 30 so as to rotate the controller 10 with the gripping unit 20 as a rotation center. For example, in FIGS. 10 and 12, the control unit 117 controls the generating mechanism 30 so that the controller 10 is rotated about the gripping unit 20 as the rotation center RC as a reaction when the sword SW hits another object in the virtual space. To do. For example, the control unit 117 controls the solenoid 50 (power source) in FIG. 4 to generate a reaction (reaction force) that rotates the controller 10 in the directions B2 and C2 with the gripping unit 20 as the rotation center RC. By doing so, it becomes possible to give the user a virtual reality as if the sword SW hit another object and the sword SW was pushed back in the opposite direction by the reaction of the hit.

  10 and 12, when the same rotation as described above is caused by using the generating mechanism 60 provided between the rear end portion of the controller 10 and the gripping portion 20, B4 in FIGS. The generating mechanism 60 may generate a reaction (reaction force) in the direction indicated by C4. As a result, the controller 10 can be rotated in the directions of B4 and C4 with the grip 20 as the rotation center RC. By doing so, it is possible to give the user a virtual reality as if the sword SW hit another object and the sword SW is pushed back in the reverse direction by the reaction of the hit. That is, when the reaction generating mechanism 60 is located on the rear end side of the grip portion 20 of the controller 10, the direction of action of the reaction such as B4 and C4 is opposite to B2 and C2. Set to. Then, the user is given the illusion that a reaction (impact) occurs in the reaction direction of the hit in the blade portion on the tip side.

  In the present embodiment, the controller 10 is a sword-type controller, and the generation mechanism 30 is provided on the blade portion of the sword-type controller 10. For example, in FIG. 4, the generation mechanism 30 is provided on the blade portion of the sword imitated by the controller 10 (the portion corresponding to the blade of the sword), and is provided at a location closer to the tip than the grip portion 20. Yes. In this way, even when the blade portion of the sword SW hits another object such as the enemy character OBE, shield SLE or the like in the virtual space, the controller 10 causes a reaction at the portion corresponding to the blade of the sword. become. Therefore, it is possible to give the user a virtual reality feeling as if the object was cut with a real sword.

  Also, the information acquisition unit 111 in FIG. 1 acquires at least one of user position information, posture information, and motion information in real space. For example, the user's position information, posture information, or motion information in real space is acquired using a light receiving element, a light emitting element, or the like as described with reference to FIGS. Alternatively, the position information, posture information, or motion information of the user in real space may be acquired using a motion sensor (acceleration sensor, gyro sensor) provided in the controller 10 or attached to the user. Then, the control unit 117 determines the game situation based on at least one of the user position information, posture information, and motion information, and controls the generation mechanisms 30 and 60. Here, generating reaction or vibration according to the game situation means that reaction or vibration occurs according to the game situation even if the sword SW in the virtual space corresponding to the controller 10 does not hit another object. Such cases are also included.

  For example, a sensor function (including a position detection function and a motion detection function such as acceleration and angular velocity) is mounted on the controller 10 (sword interface) so that position information, posture information, or motion information of the controller 10 can be detected. In addition, the user's position information, posture information, or motion information can be detected by the user's monitor process using a motion sensor or an external device. Then, a reaction or vibration is generated according to the detected position information, posture information, or motion information and the game situation. For example, even when the user's sword SW and other objects are not hit in the virtual space, a reaction or vibration is generated according to the motion of shaking the controller 10 (sword). By doing so, in the simulation system using the HMD 200, it is possible to realize an effective user interface for improving the virtual reality than ever before.

  Further, the control unit 117 controls the generation mechanisms 30 and 60 according to the hit determination result between the object corresponding to the controller 10 and another object. The object corresponding to the controller 10 is, for example, an object (display object) that moves in the virtual space according to the movement of the controller 10 in the real space. For example, an object in a virtual space controlled based on position information, posture information, or motion information of the controller 10. For example, objects corresponding to the controller 10 are objects such as the sword SW and the shield SL in FIG. Alternatively, the object corresponding to the controller 10 may be a user character (user moving body) in a virtual space corresponding to the user or a part such as a hand, foot, or head of the user character. The other object is an object to be hit with an object corresponding to the controller 10. For example, the other objects are objects such as the enemy character OBE, shield SLE, and sword SWE in FIG. Alternatively, the other object may be a wall, a door, a building, a road, or the like constituting the background, or various items.

  The hit determination process of the object corresponding to the controller 10 and other objects is performed by the hit determination unit 116 of FIG. For example, the hit determination unit 116 performs the hit determination process by, for example, an intersection determination process between an object corresponding to the controller 10 or a hit volume that contains the object and a hit volume that internally includes another object or the other object. Realize. The hit determination process is not limited to such an intersection determination process, and various known determination processes can be employed.

  The control unit 117 controls the generation mechanisms 30 and 60 based on the hit determination result in the hit determination unit 116 to generate a reaction or vibration in the controller 10. For example, as shown in FIGS. 11 and 13, when it is determined that the user character's sword SW hits the enemy shield SLE, the enemy character OBE, etc., B2 and B3 in FIG. 10 and C2 and C3 in FIG. As a result, reaction or vibration is generated.

  According to the simulation system using the HMD 200, since the VR space that extends over the entire circumference of the user's field of view is expressed, the virtual reality of the user is greatly improved. However, the user wearing the HMD 200 cannot visually recognize the situation in the real space because the field of view is covered by the HMD 200. Under such circumstances, the user plays the game by moving the controller 10 by shaking the controller 10 while viewing the image in the virtual space as shown in FIG. At this time, as shown in FIG. 11 and FIG. 13, the virtual reality realized with the HMD 200 if the user's sword SW hits the shield SLE or the enemy character OBE but the user has no response to the controller 10 in his / her hand. A situation occurs in which the feeling is reduced.

  In this regard, according to the present embodiment, when the sword SW of the user character hits the shield SLE or the enemy character OBE as shown in FIGS. 11 and 13, as described in FIGS. 10 and 12, The generating mechanisms 30 and 60 generate a reaction or vibration. As a result, the user can experience a response such as a hit feeling and a tactile sensation as if the enemy had been cut with an actual sword. Therefore, a virtual experience that is actually fighting with a sword is possible, so that the virtual reality of the user can be greatly improved.

  In addition, the control unit 117 generates the generation mechanisms 30 and 60 when it is determined that the other object is disconnected after the object corresponding to the controller 10 hits another object and when the other object is not disconnected. You may vary the generation | occurrence | production aspect of the reaction or vibration in.

  For example, in FIG. 13, it is assumed that the sword SW that is an object corresponding to the controller 10 hits an enemy character OBE that is another object. In this case, it is determined whether or not the enemy character OBE has been cut by the sword SW. For example, when the speed at which the user swings the controller 10 is high or the strength of the swing is strong, it is determined that the enemy character OBE has been cut. For example, it is determined that the sword SW has penetrated the armor or the like of the enemy character OBE and at least a part of the body (body, muscle) of the enemy character OBE has been cut. On the other hand, if the speed at which the user swings the controller 10 is slow or the swing strength is weak, it is determined that the enemy character OBE has not been cut. For example, the sword SW does not penetrate the armor or the like of the enemy character OBE, and determines that the body of the enemy character OBE has not been cut. Then, the occurrence of reaction or vibration in the generation mechanisms 30 and 60 is made different depending on whether it is determined that the enemy character OBE has been cut or not. For example, when it is determined that it has been cut, the reaction mechanism 30 generates a stronger reaction or the generation mechanism 60 generates a stronger vibration than when it is determined that it has not been cut. Alternatively, the generation mode of the vibration generated by the generation mechanism 60 (or the vibration generated by the generation mechanism 30) is made different between the case where it is determined that it has been cut and the case where it is determined that it has not been cut. For example, a sound file for cutting determination and a sound file for non-cutting determination are prepared. At the time of cutting determination, an audio signal of the audio file for cutting determination is input to the transducer 62, and vibration suitable for cutting is generated. On the other hand, at the time of non-cutting determination, the sound signal of the sound file for non-cutting determination is input to the transducer 62 to generate vibration suitable for non-cutting. For example, at the time of cutting, a vibration that can feel as if the enemy's body was cut is generated. For example, the vibration is strengthened or the vibration period is extended so that the reverberation of the cutting lasts longer.

  The control unit 117 controls the generation mechanisms 30 and 60 based on at least one of the type information, motion information, other object type information, and motion information of the controller 10. The type information of the controller 10 is type information about an object represented by the controller 10, for example. Taking the sword-shaped controller 10 as an example, whether the sword corresponding to the controller 10 is a large sword, a small sword, a heavy sword, a light sword, a thick sword, a thin sword, etc. Information. Alternatively, the type information of the controller 10 may be information for classifying the attack power of the sword. For example, if the sword is heavy, the reaction or vibration generated by the generation mechanisms 30 and 60 is increased. If the sword is light, the reaction or vibration generated by the generation mechanisms 30 and 60 is decreased. If the sword has high attack power, the reaction or vibration generated by the generation mechanisms 30 and 60 is increased. If the sword has low attack power, the reaction or vibration generated by the generation mechanisms 30 and 60 is reduced. If the controller 10 is a weapon-type controller other than a sword (such as an axe, spear, cane, or whip), or a sports equipment-type controller such as a bat, a racket, or a club, or a drum or drumstick The same applies to an imitated musical instrument type controller, and the type information of the controller 10 is information (size, weight, attack power, performance, etc.) on a weapon, sports equipment, or musical instrument.

  Further, the movement information of the controller 10 is information about the operation speed, the operation acceleration, or the operation direction of the controller 10, for example. For example, when the operation speed of the controller 10 is high, the reaction or vibration generated by the generation mechanisms 30 and 60 is increased, and when the operation speed is low, the reaction or vibration generated by the generation mechanisms 30 and 60 is decreased. Further, the direction or intensity of reaction or vibration is changed according to the operation direction of the controller 10. Further, the type information of other objects is information representing the attributes of the object represented by the object. For example, the other object type information is information for specifying the generation mode such as the strength, reaction period, or generation timing of the reaction or vibration generated by the generation mechanisms 30 and 60. The type information of other objects is information such as whether the object represents a hard object or an object representing a soft object. As an example, in the case of an object representing a hard object, the reaction or vibration generated by the generation mechanisms 30 and 60 is increased or the generation period is shortened. On the other hand, in the case of an object representing a soft object, the reaction or vibration generated by the generation mechanisms 30 and 60 is reduced or the generation period is lengthened. However, the present embodiment is not limited to such a generation mode. Further, the movement information of other objects is information on the movement speed, movement acceleration, or movement direction of the other objects. For example, in FIG. 11 and FIG. 13, the generation mode of the reaction or vibration generated by the generation mechanisms 30 and 60 in consideration of the movement speed, the movement acceleration, or the movement direction of the other objects such as the shield SLE and the enemy character OBE. Control. For example, the reaction or vibration is controlled in accordance with the relative speed or relative direction relationship between the sword SW and another object.

  The control unit 117 controls at least one of reaction and vibration generated by the generation mechanism by controlling an output signal to the power source of the generation mechanisms 30 and 60. In the example of FIG. 4, by controlling an output signal (control signal or power supply voltage) to the solenoid 50 that is a power source of the generation mechanism 30, the magnitude, generation period, or occurrence of the reaction generated by the generation mechanism 30. Controls the generation mode such as timing. For example, the strength of the reaction is controlled based on the power supply voltage supplied to the solenoid 50, or the reaction occurrence period or generation timing is controlled based on the control signal supplied to the solenoid 50. In addition, by controlling the output signal (sound signal or power supply voltage, etc.) to the transducer 62 that is the power source of the generation mechanism 60, the magnitude, generation period, or generation timing of vibration generated by the generation mechanism 60 is generated. Control aspects. For example, the intensity of vibration is controlled based on the power supply voltage supplied to the transducer 62, and the generation period or generation timing of vibration is controlled based on the control signal supplied to the transducer 62. The power source may be a motor such as a vibration motor. For example, instead of a method of causing the weight to collide with the receiving portion, a reaction or vibration may be generated by an inertial force or the like when the weight is moved or rotated.

  FIG. 15 is a flowchart for explaining a processing example of this embodiment. As shown in FIG. 15, first, position information, posture information, motion information, etc. of the user are acquired (step S1). For example, the user's position information, posture information, motion information, or the like is specified and acquired by the tracking process described with reference to FIGS. 2A to 3B or a process using a sensor such as a motion sensor. Next, the game situation is determined based on the user's position information, posture information, movement information, or the like (step S2). For example, the user's game play status, game progress status, game battle status, game battle status, game hit determination status, character status appearing in the game, character or user status parameter status, or game Judging the status of the grades. Then, the generation mechanisms 30 and 60 are controlled according to the game situation (step S3). For example, the generation mode such as the strength of the reaction or vibration generated by the generation mechanisms 30 and 60, the generation period, or the generation timing is controlled according to the game situation.

  FIG. 16 is a flowchart for explaining a detailed processing example of this embodiment. As shown in FIG. 16, first, position information, posture information, motion information, and the like of the user are acquired (step S11). Next, based on the user's position information, posture information, movement information, or the like, hit determination processing is performed between the user character's object such as a sword or part and another object such as an enemy character or shield (step S12). That is, as shown in FIGS. 11 and 13, it is determined whether or not a hit (collision) between the sword and another object has occurred. When a hit occurs, the generation mechanism is controlled based on controller type information, motion information, other object type information, motion information, and the like (steps S13 and S14). For example, by controlling the output signal to the power source (solenoid, transducer, motor, etc.) of the generating mechanism, the reaction mechanism of the generating mechanism or the mode of occurrence of vibration is controlled.

  In this embodiment, as shown in FIG. 17A, the generating mechanism 30 includes a weight 32 and a receiving portion 34 of the weight 32, and generates a reaction by causing the weight 32 to collide with the receiving portion 34. ing. That is, the generating mechanism 30 generates a reaction (or vibration) by moving the weight 32 by the solenoid 50 that is a power source and causing the weight 32 to collide with the receiving portion 34. In this way, it is possible to realize the occurrence of a virtual reaction that can make the user feel a hit feeling due to a hit with another object with a simple mechanism. Note that the configuration of the generating mechanism 30 is not limited to the configuration of FIG. For example, the reaction may be generated by driving a weight or the like into a receiving portion by a rotational movement using a motor.

  The control unit 117 may control at least one of the reaction and vibration generated by the generation mechanism 30 by changing the stroke distance of the weight 32. For example, in FIG. 17A, the stroke distance LST (swing movement distance) of the weight 32 is short, but in FIG. 17B, the stroke distance LST is long. The longer the stroke distance LST is, the stronger the reaction or vibration generated by the generating mechanism 30 is. That is, the strength of reaction or vibration can be controlled by the stroke distance LST. For example, the control unit 117 controls the stroke distance LST according to the result of the hit determination process (game situation in a broad sense) to control the strength of the reaction or vibration. The stroke distance LST of the weight 32 can be set by controlling the moving distance of the shaft member 52 by the solenoid 50. For example, the stroke distance LST can be set by the position where the arm 40 is locked by a stopper (such as the elastic member 46) of the shaft member 52. Alternatively, the stroke distance LST may be set by setting the length of the arm 40. For example, a plurality of generation mechanisms having different lengths of the arm 40 or different positions where the arm 40 is locked by a stopper are prepared, and the control unit 117 controls which of the plurality of generation mechanisms is used. The distance may be changed.

  The control unit 117 may perform control to change at least one of the type of the weight and the type of the receiving unit. For example, the control unit 117 changes the type (mass, material, etc.) of the weight or the receiving part in accordance with the result of the hit determination process, and adjusts the strength of reaction or vibration, the texture, or the like.

  For example, in FIG. 18A, a plurality of types of weights 32A, 32B, and 32C are prepared. The weights 32A, 32B, and 32C have, for example, different masses. Alternatively, the weights 32A, 32B, and 32C may be made of different materials in order to adjust the strength or texture of the reaction or vibration. Then, for example, according to the result of the hit determination process (game situation), which of the weights 32A, 32B, and 32C is used is controlled. By doing so, it becomes possible to adjust the strength of the reaction or vibration, the texture, or the like according to the result of the hit determination process.

  In FIG. 18B, a plurality of types of receiving portions 34A, 34B, and 34C are prepared. The receiving portions 34A, 34B, 34C are made of, for example, different materials. Alternatively, the masses and the like of the receiving portions 34A, 34B, 34C may be varied. Then, for example, depending on the result (game situation) of the hit determination process, which of the receiving units 34A, 34B, and 34C is used is controlled. By doing so, it becomes possible to adjust the strength of the reaction or vibration, the texture, or the like according to the result of the hit determination process. Specifically, in FIG. 18B, the first, second, and third regions of the disk-shaped member are set to the receiving portions 34A, 34B, and 34C. The disc-shaped member is rotated as indicated by E1 based on a control signal from the control unit 117, thereby controlling which of the receiving units 34A, 34B, and 34C is used.

  In the present embodiment, the controller 10 includes a generation mechanism 60 (second generation mechanism) that generates at least one of reaction and vibration. Then, the control unit 117 makes the reaction or vibration generation mode in the generation mechanism 30 (first generation mechanism) different from the reaction or vibration generation mode in the generation mechanism 60 (second generation mechanism). For example, the generation mode such as the strength of the reaction or vibration in the generation mechanism 30, the generation period, the generation timing or the presence / absence of occurrence, the strength of the reaction or vibration in the generation mechanism 60, the generation period, the generation timing or the presence / absence of the generation, etc. The generation mode of is different. For example, the generation mode of the generation mechanism 30 and the generation mode of the generation mechanism 60 are set according to the result of the hit determination process (game situation). For example, how to properly use the generation mechanism 30 and the generation mechanism 60 is determined according to the result (game situation) of the hit determination process. For example, it is determined which of the generation mechanisms 30 and 60 is used according to the type of another object to be hit (properties such as the hardness or material of the object represented by the object).

  For example, in FIG. 19, for example, after a hit with another object (such as an enemy character) occurs, a strong reaction is generated by the generation mechanism 30 in the period T1. In a period T2 that is later in time than the period T1, a weak vibration is generated by the generation mechanism 60. In the period T1, the strength of the generated recoil is strong, but the length of the generation period is short. In the period T2, the intensity of the generated vibration is weak, but the length of the generation period is long. As shown in FIG. 19, by making the generation mode in the generation mechanism 30 different from the generation mode in the generation mechanism 60, it is possible to realize the reaction control or vibration generation control that allows the user to feel more realistic. For example, when an enemy character or the like is cut with a sword, the user can first feel a response due to a strong reaction by the generation mechanism 30. Thereafter, a weak vibration such as a reverberation of the reaction is generated by the generating mechanism 60 and is transmitted to the user's hand after cutting. Therefore, it is possible to give the user a virtual reality as if the enemy character was cut with a real sword.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, in the specification or the drawings, terms (hit determination process result, solenoid transducer motor, user, etc.) described together with different terms (game situation, power source, user moving body, etc.) having a broader meaning or the same meaning at least once. Character, etc.) can be replaced by the different terms anywhere in the specification or drawings. In addition, the controller configuration, structure, generation mechanism configuration, structure, position information acquisition processing, virtual space setting processing, moving body movement processing, display processing, game processing, generation mechanism control processing, etc. in this embodiment It is not limited to what was demonstrated, The method, process, and structure equivalent to these are also contained in the scope of the present invention. The present invention can be applied to various games. Further, the present invention can be applied to various simulation systems such as a business game device, a home game device, or a large attraction system in which a large number of users participate.

US user, HR, HL hand, SW, SWE sword, SL, SLE shield,
OBE enemy character, LST stroke distance, DR1, DR2, DR3 direction,
10 controller, 12 base, 20 gripping part,
30 generating mechanism, 32, 32A-32C weight, 34, 34A-34C receiving part,
40 arms, 41 guide portions, 42, 44 fittings, 46 elastic members,
47 plate member, 50 solenoid (power source), 52 shaft member, 54 head part,
60 generating mechanism (second generating mechanism), 62 transducer,
64 plate member, 70, 80 controller,
100 processing units, 102 input processing units, 110 arithmetic processing units, 111 information acquisition units,
112 virtual space setting unit, 113 moving body processing unit, 114 virtual camera control unit,
115 game processing unit, 116 hit determination unit, 117 control unit,
120 display processing unit, 130 sound processing unit,
140 output processing unit, 150 imaging unit, 151, 152 camera, 160 operation unit,
170 storage unit, 172 object information storage unit, 178 drawing buffer,
180 Information storage medium, 192 sound output unit, 194 I / F unit,
195 portable information storage medium, 196 communication unit,
200 HMD (head-mounted display device), 201-203 light receiving element, 210 sensor unit,
220 display unit, 231 to 236 light emitting element, 240 processing unit,
280, 284 Base station, 281, 282, 285, 286 Light emitting element

Claims (15)

A rod-shaped portion between the front end portion and the rear end portion has a grip portion to be gripped by a user. Between the front end portion and one end portion of the rear end portion and the grip portion, reaction and vibration are generated. A controller provided with a generating mechanism for generating at least one of them;
An information acquisition unit that acquires at least one of position information, posture information, and motion information of the controller in real space;
A game processing unit for performing game processing;
A display processing unit that generates an image including an image of an object corresponding to the controller as a display image of a head-mounted display device worn by the user so as to cover the field of view based on the result of the game process;
A control unit that controls the generating mechanism that generates at least one of reaction and vibration according to the situation of the game played by the user;
The simulation system characterized by including. In claim 1,
The controller is
A simulation system characterized in that the generation mechanism is controlled so that the controller is rotated with the gripper as a rotation center. In claim 1 or 2,
The simulation system according to claim 1, wherein the controller is a sword-type controller, and the generation mechanism is provided on a blade portion of the sword-type controller. In any one of Claims 1 thru | or 3,
The information acquisition unit
Obtaining at least one of position information, posture information and motion information of the user in real space;
The controller is
The simulation system characterized by determining the game situation based on at least one of the position information, the posture information, and the movement information of the user, and controlling the generation mechanism. In any one of Claims 1 thru | or 4,
The controller is
A simulation system, wherein the generation mechanism is controlled according to a hit determination result between an object corresponding to the controller and another object. In claim 5,
The controller is
After the object corresponding to the controller hits the other object, when it is determined that the other object has been disconnected, and when it is determined that the object has not been disconnected, the reaction mechanism or vibration of the generation mechanism A simulation system characterized by different generation modes. In claim 5 or 6,
The controller is
A simulation system, wherein the generation mechanism is controlled based on at least one of the controller type information, motion information, the other object type information, and motion information. In any one of Claims 1 thru | or 7,
The controller is
A simulation system characterized by controlling at least one of reaction and vibration generated by the generating mechanism by controlling an output signal to a power source of the generating mechanism. In any one of Claims 1 thru | or 8.
The generating system includes a weight and a receiving portion of the weight, and generates a reaction by causing the weight to collide with the receiving portion. In claim 9,
The controller is
A simulation system characterized by controlling at least one of reaction and vibration generated by the generating mechanism by changing a stroke distance of the weight. In claim 9 or 10,
The controller is
The simulation system characterized by performing control which changes at least one of the kind of said weight, and the kind of said receiving part. In any one of Claims 1 thru | or 11,
The controller includes a second generation mechanism that generates at least one of reaction and vibration,
The controller is
A simulation system, wherein a generation mode of reaction or vibration in the generation mechanism is different from a generation mode of reaction or vibration in the second generation mechanism. An information acquisition unit that acquires at least one of position information, posture information, and motion information of the controller in real space;
A game processing unit for performing game processing;
A display processing unit that generates an image including an image of an object corresponding to the controller as a display image of a head-mounted display device worn by the user so as to cover the field of view based on the result of the game process;
As a control unit for controlling the controller,
Make the computer work,
The controller has a grip portion gripped by a user in a rod-shaped portion between the front end portion and the rear end portion, and between the one end portion of the front end portion and the rear end portion and the grip portion, A generating mechanism for generating at least one of reaction and vibration is provided,
The controller is
A program for controlling the generating mechanism that generates at least one of reaction and vibration according to a situation of a game played by the user. A grip portion provided in a rod-shaped portion between the front end portion and the rear end portion and gripped by a user;
A generating mechanism that is provided between one end of the front end and the rear end and the grip, and generates at least one of reaction and vibration;
Including
The generating mechanism generates at least one of reaction and vibration so as to rotate the controller around the grip portion as a rotation center. In claim 14,
The generation mechanism is
A controller comprising a weight and a receiving portion of the weight, wherein the reaction is caused by causing the weight to collide with the receiving portion.

Images