JP6883394B2 - Programs, computers and methods - Google Patents

Description

The present invention relates to a system, a method, and a program for interlocking a head-mounted display and a controller to operate a screen on a display on the head-mounted display, and more particularly, having a communication function between the head-mounted display and an external network. It relates to a system that operates the screen in conjunction with a controller.

There is known a head-mounted display (HMD) that is attached to the user's head and can present an image to the user by a display or the like placed in front of the eyes. Since the conventional HMD controller is connected to an external network such as the Internet via the main unit, the controller itself does not communicate with the external network. In addition, the conventional controller requires a dedicated controller for connecting to the HMD.

Further, conventionally, the controller used for various operations of the display image in the HMD is mainly operated by buttons and keys, and does not have a sensor display. Recently, such a controller has a built-in sensor capable of detecting motion (for example, an acceleration sensor), and a device for detecting the motion of the controller has been developed.

In the system related to the above background technology, since the controller itself does not have a function of connecting to an external network, the controller itself is always connected to the main unit when connecting to the Internet.

In addition, the controller connected to the conventional HMD can be operated by input from a sensor that can detect buttons, keys, and movements, but since it does not have a sensor display, it is possible to perform more intuitive and detailed operations. could not. Especially in the case of button and key operation, if the user is completely immersed in the virtual space with the HMD attached, the current state that the user is trying to operate (which key or button the user's finger is placed on) ) Can not be recognized by the user. Therefore, when the user's finger is removed from the desired key or button, an operation undesired by the user is performed.

Further, when one operation screen is shared and operated by a plurality of users located in the same space, the load on the main body device is high because a plurality of controllers are connected to one main body device in the conventional system. .. Further, in the conventional system, when one operation screen is shared and operated by a plurality of users located in different spaces, the plurality of controllers are connected to different main units for each space and via a network. Each main unit was connected to the server for communication. In this case, a separate main unit is required for a plurality of controllers.

Further, when a dedicated controller is required for the operation of the HMD, the user needs to carry a dedicated controller for each HMD operation.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a system for linking an HMD and a controller having a communication function with the outside. Further, in the present invention, since the controller has a communication function with the outside, when connecting to the main body device, software necessary for the connection can be downloaded from the external server, so that the controller is dedicated to the main body device. It is not necessary to have a terminal, and a generally distributed terminal such as a mobile terminal or a smartphone can be used.

(1) The present invention
A system that outputs the generated image to a head-mounted display.
An operation judgment unit that determines the user operation required according to the content of the running application,
An input detection unit that detects output data corresponding to the required user operation from the output data from the controller, and
An operation determination unit that determines the operation of the display object based on the detected output data,
A display calculation unit that calculates the display parameters of the display object based on the determined operation of the display object and the detected output data to generate an image of the display object displayed in the virtual space.
An image generation output unit that generates an image in which an image of a display object generated in an image in virtual space is arranged and outputs it to a head-mounted display connected to the system, and an image generation output unit.
Is equipped with
The present invention relates to a system, wherein the controller and the system are connected by a controller-side interface program received from a server by a communication unit of the controller and a system-side interface program of the system.

According to the present invention, by using a controller having a communication function with an external network, various information, moving images, images, and the like can be obtained from the external server even if the controller itself is not connected to the main unit. Voices and other information (for example, map information, weather information, stock price information, applications) can be obtained and reflected in the virtual space presented to the HMD. Further, in the present invention, since the controller has a communication function with the outside, when connecting to the main body device, software necessary for connecting to the main body device can be downloaded from the external server, so that the main body It does not have to be dedicated to the device. Therefore, according to the present invention, a mobile terminal such as a smartphone, which has become popular in recent years, can be used as an HMD controller, and the user can operate the HMD without requiring a dedicated controller. ..

(2) Further, in the present invention, the output data from the controller can be output data generated based on the operation input by the user from the sensor display of the controller. Further, in the present invention, the operation input can be a swipe operation, a tap operation, or a pinch-in / pinch-out operation.

According to the present invention, by using the controller provided with the sensor display, the user can perform various intuitive and detailed operations (swipe operation, tap operation, pinch operation) on the HMD. When the user wears the HMD and is completely immersed in the virtual space, the user recognizes the current state (which key or button the user's finger is placed on) that the user is trying to operate. I can't. The controller according to the present invention includes a sensor display, and the user can perform various intuitive operations.

(3) Further, in the present invention, the output data from the controller can be output data generated based on the voice input by the user from the microphone of the controller. Further, according to the present invention, the image of the display object displayed in the virtual space can be changed based on the voice input by the user.

According to the present invention, by using a controller provided with a microphone, the user can change the image to be displayed on the HMD by performing an intuitive operation (voice input) on the HMD.

(4) Further, in the present invention, the output data from the controller can be output data generated based on the operation input by the user from the acceleration sensor or the gyro sensor of the controller.

According to the present invention, by using a controller provided with an acceleration sensor or a gyro sensor, various intuitive operations by the user can be performed by detecting the swing width, swing cycle, speed, and strength of the controller. ..

(5) Further, in the present invention, the output data from the controller includes an image stored in the memory of the controller and output data generated based on an operation input by the user from the sensor display of the controller, and operates. The determination unit can determine the display object based on the image and determine the operation of the display object based on the detected output data. Further, in the present invention, the controller can be a smartphone.

According to the present invention, when the controller is a mobile terminal such as a smartphone that is always carried by the user, images, videos, and sounds taken by the user in the past are stored, and the user himself / herself stored in the smartphone Images, videos, and sounds taken in the past can be reflected in the HMD. Further, when the controller is a mobile terminal such as a smartphone that the user always carries, the user does not carry a dedicated controller each time the HMD operation is performed, and the smartphone that the user always carries is used as the controller for the HMD operation. It can be used.

(6) Further, in the present invention, the output data from the controller includes the position information of the controller from the GPS of the controller and the output data generated based on the operation input by the user from the sensor display of the controller, and the display calculation is performed. The unit can be characterized in that the display parameters of the display object are calculated based on the position information and the determined operation of the display object to generate an image of the display object displayed in the virtual space. ..

According to the present invention, it is possible to change the image of the display object displayed on the virtual space presented to the HMD according to the current position information of the controller.

(7) Further, in the present invention, the operation determination unit determines a plurality of user operations required according to the content of the application being executed.
The input detection unit detects output data corresponding to the plurality of required user operations from the output data from the controller, respectively.
The operation determination unit determines the operation of the display object based on the first output data of the detected output data, and converts the operation of the display object based on the second output data of the detected output data. It is a feature.

According to the present invention, by providing an acceleration sensor and a gyro sensor, the operation of a display object (display object jumps) corresponding to one operation input (for example, tap operation) to the controller can be performed by a simple switching operation such as shaking the terminal. It can be instantly converted to the behavior of another display object (display object is sliding). When the user wears the HMD and is immersed in the virtual space, the user cannot directly see the controller being operated. The accelerometer and gyro sensor allow the user to reliably and instantly convert the movement of the displayed object with a simple and intuitive operation such as shaking the terminal.

(8) Further, the present invention is a program for causing a processor to execute a method of outputting a generated image to a head-mounted display.
An operation judgment step that determines the user operation required according to the content of the running application, and
An input detection step that detects output data corresponding to the required user operation from the output data from the controller, and
An operation determination step that determines the operation of the display object based on the detected output data, and
A display calculation step that calculates the display parameters of the display object based on the determined operation of the display object and the detected output data to generate an image of the display object displayed in the virtual space.
An image generation output step of generating an image in which an image of a display object generated in an image in virtual space is arranged and outputting it to a head-mounted display connected to the system, and an image generation output step.
Is equipped with
It is related to a program, characterized in that the controller and the system are connected by a controller-side interface program received from a server via the Internet by the communication unit of the controller and a system-side interface program of the system. ..

(9) Further, the present invention is a control method for a system that outputs a generated image to a head-mounted display.
An operation judgment step that determines the user operation required according to the content of the running application, and
An input detection step that detects output data corresponding to the required user operation from the output data from the controller, and
An operation determination step that determines the operation of the display object based on the detected output data, and
A display calculation step that calculates the display parameters of the display object based on the determined operation of the display object and the detected output data to generate an image of the display object displayed in the virtual space.
An image generation output step of generating an image in which an image of a display object generated in an image in virtual space is arranged and outputting it to a head-mounted display connected to the system, and
Is equipped with
The present invention relates to a method characterized in that the controller and the system are connected by a controller-side interface program received from a server via the Internet by the communication unit of the controller and a system-side interface program of the system. ..

The above and other features and advantages of the present invention will become apparent from the following more specific description of the embodiments of the present invention, the accompanying drawings and the claims.

A system configuration diagram of a system according to an embodiment of the present invention is shown. A functional configuration of the controller 100 according to an embodiment of the present invention is shown. The functional configuration of the main body apparatus 120 according to one Example of this invention is shown. The screen displayed on the HMD generated based on the swipe operation of the user according to one embodiment of the present invention is shown. The screen displayed on the HMD generated based on the tap operation of the user according to one embodiment of the present invention is shown. The screen displayed on the HMD generated based on the tap operation of the user according to one embodiment of the present invention is shown. The screen displayed on the HMD generated based on the tap operation of the user according to one embodiment of the present invention is shown. The screen displayed on the HMD generated based on the pinch-in / pinch-out operation of the user according to the embodiment of the present invention is shown. The screen displayed on the HMD generated based on the pinch-in / pinch-out operation of the user according to the embodiment of the present invention is shown. The screen displayed on the HMD generated based on the pinch-in / pinch-out operation of the user according to the embodiment of the present invention is shown. The screen displayed on the HMD generated based on the operation input from the acceleration sensor / gyro sensor according to the embodiment of the present invention is shown. The screen displayed on the HMD generated based on the operation input from the acceleration sensor / gyro sensor according to the embodiment of the present invention is shown. The screen displayed on the HMD generated based on the voice input from the accelerometer / gyro sensor according to the embodiment of the present invention is shown. The screen displayed on the HMD generated based on the operation input from the acceleration sensor / gyro sensor according to the embodiment of the present invention is shown. The screen displayed on the HMD generated based on the operation input from the acceleration sensor / gyro sensor according to the embodiment of the present invention is shown. The screen displayed on the HMD generated based on the operation input from the acceleration sensor / gyro sensor according to the embodiment of the present invention is shown. A screen displayed on an HMD generated based on a voice input from a microphone sensor according to an embodiment of the present invention is shown. A screen displayed on an HMD generated based on an input from a microphone sensor according to an embodiment of the present invention is shown. A flowchart showing a processing flow of a controller according to an embodiment of the present invention is shown. The flowchart which shows the process flow of the main body apparatus according to one Example of this invention is shown. A flowchart showing a flow of processing for connecting a controller and a main unit device on an application according to an embodiment of the present invention is shown. A system configuration diagram of a system composed of a plurality of controllers according to an embodiment of the present invention is shown. An exemplary user display screen in which one operation screen is operated by a plurality of users according to an embodiment of the present invention is shown.

Hereinafter, this embodiment will be described. FIG. 1 shows a system configuration diagram of a system according to an embodiment of the present invention.
In this system, the controller 100 and the main unit 120 are connected via a wireless network such as Wi-Fi or Bluetooth, and the main unit 120 and the head-mounted display 140 are connected to a wireless or wired network such as Wi-Fi or Bluetooth. Connected via. The controller 100 further includes an antenna 110, and can connect to a network such as the Internet and communicate with the server 160 without going through the main body device. The server 160 is realized by a known server computer configured around a processor, various storage devices, and a data communication device, and transmits / receives data to / from the controller 100. Further, when there are a plurality of controllers, the server 160 manages the communication performed between the controllers.

Controller Configuration Controller 100 may include at least one processor 102, at least one interface 104, at least one memory 106, at least one sensor display 108, and antenna 110. The controller 100 can be realized by, for example, a known smartphone or mobile terminal. The controller 100 is held and worn by the user himself / herself, and can detect various operation inputs according to the user's operation, and the user's operation input and voice input according to the user's movement.

The controller-side processor 102 is configured to perform various processes such as execution of the controller-side interface program 106A and instruction of instructions to each functional unit. The controller-side interface program 106A is for making the transmitting unit and the receiving unit (FIG. 2) of the user controller, which will be described later, function.

The controller-side interface program 106A transmits output data from the controller 100 to the main unit 120 via the network 180 by making the transmission unit 204, which will be described later, function. Further, by making the receiving unit 206, which will be described later, function, it is possible to receive various data including experience information such as an image that the user experiences in the virtual space from the main unit 120 to the controller 100.

The controller-side interface 104 is an interface that transmits output data to the main unit 120 via the network 180 and receives data from the main unit 120, and is realized by implementing the controller-side interface program 106A. More specifically, the controller-side interface 104 transmits data obtained from various sensors built in the controller, output data such as an image stored in the memory of the controller to the main unit, and / or the user head mounts the data. It is an interface that receives experience information such as images and moving images that the user has experienced in the virtual space from the main unit when the device is attached and immersed in the virtual space.

The controller-side memory 106 stores the controller-side interface program 106A. Further, the controller-side memory 106 collects various data such as images, moving images, and memo information stored in the past by the user, and data obtained from various sensors generated in response to the user's operation and operation. Can be stored.

The sensor display 108 is a touch sensor, and detects a swipe operation, a pinch-in / pinch-out operation, and a tap operation from the user. By providing a sensor display, it becomes possible to detect swipe operations and pinch-in / pinch-out operations from the user, which was not possible with a controller equipped with only an acceleration sensor, a gyro sensor, keys, and buttons. The user can perform intuitive detailed operations by swiping and pinching in / pinching out. Furthermore, when the user is immersed in the virtual space, the finger may come off from the key or button of the controller during operation, and in such a case, the head mount device must be removed once to check the position of the outside world. It was. Swipe operation, tap operation, or pinch operation using the sensor display can be intuitively operated without checking the positions of the keys and buttons.

The controller 100 includes an antenna 110, and the controller 100 itself has a communication function with an external network such as the Internet. The controller 100 acquires various information, weather information, map information, applications, etc. that can be obtained via the external network in real time without imposing a load on the main unit during application execution, and executes the application on the main unit. It can be reflected on the display screen of the application and HMD. Further, the controller can download the controller-side interface program 106A from the server 160 to the user controller in advance by the antenna 110 via a network such as the Internet, and install the controller on the user controller. By installing the controller-side interface program 106A on the controller 100, the user can connect the controller to the main unit using an arbitrary controller, for example, a smartphone, without using a dedicated controller for the head mount device. The output data from can be reflected on the display display of the head mount device.

The controller 100 can further include a microphone for detecting a voice input from the user, an acceleration sensor for detecting an operation input by the user, a gyro sensor, and a button, a key, a camera, a GPS, and a vibration device (not shown).

The microphone detects the user's utterance and the like by the user holding and wearing the controller. By providing a microphone, the words spoken by the user can be reflected in the display object displayed on the HMD. In addition, the voice recognition application stored in the memory is used to voice-recognize the words spoken by the user, and for example, an operation such as opening the door when a spell is cast on a game application running on the main unit. Can be done by the display object.

The acceleration sensor and the gyro sensor can detect the movement of the user holding and wearing the controller, for example, the movement of the user swinging the controller and the movement of throwing an object. Further, the accelerometer can detect the number of steps of the user. In addition, by providing an acceleration sensor, a gyro sensor, and a sensor display, the operation of the display object (display object jumps) corresponding to one operation input (for example, tap operation) to the controller can be performed by a simple switching operation such as shaking the terminal. It can be instantly converted to the behavior of another display object (display object is sliding). In particular, since the tap operation is a simple operation for the user wearing the HMD, the movements of various display objects on the application, such as shooting, slashing, and jumping, are assigned to the tap operation. When the user wears the HMD and is immersed in the virtual space, the user cannot directly see the controller being operated. The accelerometer and gyro sensor allow the user to reliably and instantly convert the movement of the displayed object with a simple and intuitive operation such as shaking the terminal.

Buttons and keys accept operation input by the user holding and wearing the user controller.
The camera allows the user to take a picture or video, and the picture or video taken is stored in the memory of the user controller. Images and moving images captured in the past by the user can be reflected in the virtual space displayed on the head-mounted display.

GPS detects the position of the user holding and wearing the controller. Therefore, for example, the screen display of the head-mounted display can be generated according to the detected position of the user. For example, when the user is located in Hokkaido, the character of the display object operated by the user in the virtual space displayed on the HMD can be the character of Hokkaido.

The vibration device is, for example, equipped with a head-mounted device, and when a user playing a game is attacked in the virtual space displayed on the head-mounted display, the controller is linked to the game. You can vibrate. The information experienced by the user during the execution of the application in the virtual space is transmitted to the interface program 106A via the network 180 by the interface program 126A on the main unit side. Then, the vibration device vibrates the controller based on the information experienced by the user received by the controller-side interface program 106A. By vibrating the controller, the user can be more immersed in the virtual space.

Configuration of main unit The main unit 120 transmits / receives data to / from the user controller 100 and the head-mounted display 140, and includes at least one processor 122, at least one interface 124, and at least one memory 126. Can be done.

The main unit processor 122 instructs each function unit to execute various applications such as games, executes the main unit interface program 126A, executes the image generation program 126B for performing image generation output processing, and performs sound generation processing. It is configured to perform various processes such as.

The interface 124 on the main unit side receives output data transmitted from the controller 100 via the network, transmits predetermined data from the main unit 120 via the network, and outputs a generated image in which display objects are arranged to the HMD 140. It is an output interface.

The main body device side memory 126 stores the main body device side interface program 126A, the image generation program 126B, and programs and data for realizing various functions executed by the main body device processor.

The interface program 126A on the main unit side is for operating the input detection unit 304 (FIG. 3) described later. By executing the interface program 126A on the main unit side, the main unit 120 detects and receives the output data transmitted from the controller 100, and transmits various data from the main unit 120 to the controller 100. The main unit device side interface program 126A is pre-installed in the main unit device. For example, the interface program 126A on the main unit side can be downloaded to the main unit in advance or installed using a recording medium via a network (not shown) such as the Internet or an intranet connected to the main unit. By installing the interface program 126A on the main unit side and installing the interface program 106A on the controller side described later on the controller, the user can use any controller, for example, a mobile terminal or a smartphone, without using a dedicated controller for the head mount device. , User input from the user controller 100 can be reflected in the HMD, display on the HMD, and application information being operated can be transmitted to the user controller. Further, the interface program 126A on the main body device side can acquire the output data from the plurality of controllers and make it identifiable, so that the plurality of controllers can be connected to the main body device 120.

The image generation program 126B is for making the operation determination unit 302, the operation determination unit 306, the display calculation unit 308, and the image generation output unit 310, which will be described later, function.
The HMD 140 is connected to the main body device 120 and displays a generated image output from the main body device 120. The HMD 140 may virtually display the sensor display (602 in FIG. 6), buttons, keys, etc. of the controller held by the user.

Functional configuration of the controller FIG. 2 is a diagram showing a functional configuration of the controller 100. As shown in the figure, the controller 100 can include an operation unit 202, a transmission unit 204, a reception unit 206, a communication unit 208, an image pickup unit 210, or a part of these functions. These functions are realized by the processor 102 executing a predetermined program stored in the memory 106 of the user controller 100. The transmitter and receiver of the user controller can be realized by executing the controller-side interface program 106A by the controller-side processor. The interface program 106A is pre-downloaded from the server 160 via the Internet using the controller antenna 130 and installed in the controller 100.

The operation unit 202 acquires inputs related to predetermined operations / operations performed on the controller by the user from various sensors provided in the controller. The various sensors are a display sensor, an acceleration sensor, a gyro sensor, a microphone sensor, and a GPS sensor. In addition, various sensors can be combined with inputs from other operation units such as buttons and keys (for example, operation inputs for swinging the controller while pressing the buttons on the controller). When the user moves the controller, the operation unit acquires the acceleration and the angular velocity for the operation from the acceleration sensor and the gyro sensor. When the user performs a swipe operation, a pinch-in / pinch-out operation, or a tap operation on the sensor display of the controller, the operation unit contacts the sensor display with various data related to the type and / or the operation, for example, the display ( Tap), contact time (long press / tap), number of contacts within a predetermined time (continuous hit, single press), displacement amount of continuous contact to the display (swipe distance, pinch distance), contact displacement direction (swipe direction, (Pinch direction) and contact displacement speed (swipe speed, pinch speed) are acquired. When the user speaks into the microphone of the controller, the operation unit 202 acquires the user voice input from the microphone. Next, the operation unit 202 sequentially stores these information (acceleration / angular velocity, operation input information, voice input information, position information) in the memory.

Here, in this embodiment, the acceleration and the angular velocity obtained from the acceleration sensor and the gyro sensor are referred to as motion input information, and are obtained from the sensor display when the user performs a swipe operation, a tap operation, or a pinch operation on the sensor display. The displacement amount of the contact with the sensor display, the displacement amount of the continuous contact with the display, the contact displacement direction, and the contact displacement speed are referred to as operation input information.

The transmission unit 204 transmits these information stored in the memory as output data to the input detection unit 304 via the network 180 in response to a request from the main unit. The output data includes operation input information, operation input information, voice input information, position information, and an image stored in the memory 106 of the controller 100 regarding a predetermined operation / operation performed on the controller 100 from the operation unit 202. Corresponds to video or a combination of these.

The controller 100 can include a receiving unit 206 that receives information transmitted from the main unit 120. The receiving unit 206 receives the information experienced by the user in the virtual space when the user wears the HMD and is immersed in the virtual space. For example, when a user playing a game is attacked in the virtual space displayed on the HMD 140, the attacked information is received. Then, the controller-side processor can execute the vibration function based on the received attacked information to vibrate the controller. In addition, the receiving unit 206 can receive the video during game play from the main unit. The controller-side processor 102 can store the received video in the memory of the controller and display it on the controller-side display 108.

The communication unit 208 is for downloading various data, for example, a program from the server 160, uploading a video during user operation to the server 160, and directly communicating with another controller when there are a plurality of controllers. Yes, usually communicate via the Internet. By providing the communication unit 208, the controller 100 can be connected to the external server 160 even if it is not connected to the main unit. Therefore, even if the controller is not connected to the main unit 120, the controller-side interface program 106A can be installed and updated at any location as long as there is an environment in which the controller can be connected to the Internet. In addition, the controller 100 can acquire various information (weather information, stock price information, map information) that can be acquired from the Internet and reflect it on the screen during the user's operation or the game content during the user's operation. Further, when there are a plurality of controllers, the controller can communicate with other controllers via the communication unit without going through the main body device.

The image pickup unit 210 can take an image or a moving image in a real space. By providing the image pickup unit, images and moving images captured in advance by the user can be reflected on the screen during operation of the user and the game screen during play.

The user controller 100 can further be equipped with GPS (not shown). By providing GPS, the position information of the controller acquired by GPS can be reflected on the screen during user operation. For example, when the controller is located in Hokkaido, the character of the display object operated by the user in the virtual space displayed on the HMD may be the character of Hokkaido, or the user operation screen of the virtual space displayed on the HMD may be the background of Hokkaido. Can be.

Functional configuration of the main unit FIG. 3 is a diagram showing a functional configuration of the main unit 120. As shown in the figure, the main body device 120 includes an operation determination unit 302, an input detection unit 304, an operation determination unit 306, a display calculation unit 308, and an image generation output unit 310. These functions are realized by the processor executing a predetermined program stored in the memory of the main unit.

The operation determination unit 302 first determines what the user operation is required to be input by the application based on the content of the application being executed by the main unit. The required user operation is a user operation on the controller that is determined based on the content of the running application, the situation in which the user is currently operating in the running application, etc., and the running application allows input. Is. For example, in an operating state in which a user plays a battle game in a virtual space and is in battle with an enemy, the required user operations corresponding to the operating state are, for example, "beating", "shooting", and "dodging an attack". ..

Next, the input detection unit 304 acquires the output data corresponding to the required user operation among the output data stored in the memory of the controller 100 via the network 180. For example, when the required user operation is "hit", the output data corresponding to the required user operation is operation input information (contact with the display by tap operation). The output data can also include the number of contacts within a predetermined time. Then, the input detection unit 304 outputs the detected output data to the operation determination unit 306. In this way, when the user operation required in the running application is "hit", the input detection unit 304 detects only the contact with the display corresponding to the tap operation among the operation input information of the output data. And get it. The operation determination unit 306 can ignore the voice input when the output data does not correspond to the required user operation, for example, when the required user operation is "beating".

In this way, the input detection unit 304 includes various information transmitted from the transmission unit 204 of the controller via the network 180, operation input information by the user (acceleration sensor, acceleration obtained from the gyro sensor, angular velocity), and operation input. Information (swipe operation, pinch-in / pinch-out operation type, and / or contact with sensor display, displacement amount of continuous contact with display, contact displacement direction, contact displacement speed), voice input information (obtained from microphone sensor) Of the information such as voice, volume), image, moving image, position information, or a combination thereof, the data corresponding to the required user operation determined by the operation determination unit 302 is detected as the output data from the controller.

Further, the input detection unit 304 may detect the output data at a predetermined timing for a predetermined time. Further, the input detection unit 304 can change the timing interval for receiving the output data. For example, when the display object is to perform a rough operation such as turning the knob of the door in the virtual space, the detection timing interval is lengthened to detect the input 15 times per second, and the display object is driven by the automobile in the virtual space. When performing an operation that requires the fine movement of the steering wheel to be sequentially reflected on the screen, such as driving the vehicle, the detection timing interval can be shortened to perform input detection 60 times per second.

Table 1 exemplifies the relationship between the content of the running application, the required user operation corresponding to the content of the running application, and the output data corresponding to the required user operation. The contents of the application, the required user operations, and the output data shown in Table 1 are merely examples, and those skilled in the art will easily understand that other user operation states and other user operations exist. ..

Next, the operation determination unit 306 determines an operation to be performed by the display object displayed on the head-mounted display based on the output data detected by the input detection unit 304 according to the required user operation. For example, when the required user operation is "hit", the action (beating, hitting frequency) to be performed by the display object in the virtual space is determined according to the contact detection to the display included in the output data and the number of touches. To do. The display object is an object to be operated in the virtual space by the user. For example, when the user is playing a game, it is a game character or a game item, and when an image or a moving image captured by the user is displayed. Are those images and videos.

For example, the motion determination unit 306 acquires the direction, displacement amount, and displacement speed of the swipe operation in the operation input information of the output data when the required user operation is “slide of image”. Then, the motion determination unit determines the motion (image slide) to be performed by the display object according to the direction of the swipe operation to the display, the displacement amount, and the displacement speed.

For example, when the required user operation is "to attach two display objects", the operation determination unit 306 indicates the direction and displacement amount of the pinch-in operation (or swipe operation) to the display among the operation input information of the output data. , Get the displacement velocity. Then, the motion determination unit determines the motion to be performed by the display object (combining the two display objects into one) according to the direction of the pinch-in operation to the display, the amount of displacement, and the displacement speed. The motion determination unit can determine the direction in which the display object is moved and the speed at which the display object is moved according to the direction of the pinch-in operation to the display by the user, the amount of displacement, and the displacement speed.

For example, the motion determination unit 306 acquires the acceleration and the angular velocity in the motion input information of the output data when the required user operation is the “handle operation”. Since the "handle operation" requires a detailed operation, the motion determination unit can increase the number of times to acquire the acceleration and the angular velocity in a fixed time. Then, the motion determination unit determines the motion (changing the direction of the display object) to be performed by the display object according to the acceleration and the acceleration.

For example, the motion determination unit 306 acquires the acceleration and the angular velocity in the motion input information of the output data when the required user operation is “swinging”. If coarse detection is sufficient for the "swinging" operation, the motion determination unit can reduce the number of times the acceleration and angular velocity are acquired in a certain period of time. Then, the motion determination unit determines the motion (moving / swinging the display object) to be performed by the display object according to the acceleration and the acceleration. The motion determination unit can determine the strength and direction of swinging the display object according to the acceleration and the angular velocity.

For example, the motion determination unit 306 acquires the acceleration and the angular velocity in the motion input information of the output data when the required user operation is “throw”. Then, the motion determination unit determines the motion (throwing the display object) to be performed by the display object according to the acceleration and the angular velocity. Further, the motion half-price portion can determine the strength and direction of throwing the display object according to the acceleration and the angular velocity.

For example, the motion determination unit 306 acquires voice input information from the output data when the required user operation is "loud". Then, the operation determination unit determines the operation (enlarges the display object) to be performed by the display object according to the volume. Further, the motion determination unit can determine the size of the display object to be enlarged according to the volume.

Next, the display calculation unit 308 performs a display calculation of the display object based on the operation determined by the operation determination unit 306 and the operation data for the display object associated with the operation, thereby displaying the image of the display object. Generate. The operation data includes acceleration, angular velocity, displacement amount, displacement velocity, direction, etc. when moving the display object. When generating a so-called three-dimensional image, display parameters including data of each vertex of the display object (position coordinates of the vertex in the world coordinate system, color data) are calculated, and based on the calculated display parameter, the display parameters are calculated. Images of display objects are sequentially generated frame by frame.

The image generation output unit 310 generates an image in which the calculated display object is arranged on the screen displayed on the display of the activated head mount device for each frame, and outputs the image to the screen. The HMD displays the image output from the image generation output unit.

FIG. 4 is an example of the screen 400 displayed on the HMD according to the present embodiment generated based on the swipe operation of the user. When the output data acquired from the input detection unit is a swipe operation, the display object (six images in FIG. 5A) to be operated is slid in the swipe direction. As shown in FIG. 4, by virtually displaying the display (402, 404) corresponding to the sensor display operated by the user on the HMD, the user wearing the HMD is completely immersed in the virtual space. Even if it is, the user can surely grasp the operation that the user is performing or is about to perform.

5A and 5B are examples of screens (500A, 500B) displayed on the HMD according to the present embodiment generated based on the tap operation of the user. When the output data received from the input detection unit corresponds to the tap operation, the display object (hand 502 in FIG. 5A, gun 504 in FIG. 5B) to be operated performs a shooting / hitting operation. For example, each time the user taps the sensor display of the controller 100, the display object performs a shooting / hitting action. Furthermore, if the output data includes the number of times the object has been tapped within a certain period of time, the speed at which the object shoots or hits can be changed according to the number of taps.

FIG. 6 is an example of the screen 600 displayed on the HMD according to the present embodiment generated based on the tap operation of the user. The display screen 600 of FIG. 6 includes a sensor display 602 that is virtually displayed in the virtual space. The virtually displayed sensor display 602 can virtually display the user's finger 604 (operation finger). A user who is immersed in the virtual space cannot recognize the controller in the real space in detail because the HMD is worn and the outside world information is blocked. By virtually displaying the sensor display of the controller in the real space on the virtual space, the user can grasp various operations on the controller without removing the head-mounted display.

7A, B, and C are examples of screens displayed on the HMD according to the present embodiment generated based on the pinch-in / pinch-out operation of the user. When the output data received from the input detection unit corresponds to the pinch-out operation, the display object (the image in the black circle in FIG. 7A) to be operated is zoomed in on the display object associated with the operation input. Do the action. Further, when the output data received from the input detection unit includes the displacement amount of the operation, the screen operation amount of the zoom-in operation can be determined based on the displacement amount. Further, when the type of output data received from the input detection unit is a pinch-out operation, the pinch-out is performed on the display object (the center circle drawn as 4 in FIG. 7B) which is the operation target selected by the user. When the operation is performed, the display object is separated into two objects (water droplets in FIG. 7B) (the objects are torn into two). Object separation can also be achieved by swiping. When the output data received from the input detector is a pinch-in operation, the display objects to be operated (two circles drawn as 2 in FIG. 7C) are combined into one large circle drawn as 4. Can be (two objects are stuck together).

8A, B, and C are examples of screens displayed on the HMD according to this embodiment. When the output data received from the input detector corresponds to the user's motion input information from the accelerometer / gyro sensor, the display object (sword in FIG. 8A, fishing rod in FIG. 8B, darts in FIG. 8C) to be operated is displayed. On the other hand, the display object is made to swing the sword in FIG. 8A, or to throw a fishing rod or darts in FIGS. 8B and 8C. Further, based on the acceleration included in the output data received from the input detection unit, the display object can be moved at a predetermined speed and acceleration, or the display object can be shaken / thrown at a predetermined strength. The motion determination unit can further move the display object in a predetermined direction when the output data received from the input detection unit includes the angular velocity.

9A, B, and C are examples of screens displayed on the HMD based on the input from the accelerometer / gyro sensor according to this embodiment. When the output data received from the input detection unit corresponds to the user's motion input information from the acceleration sensor / gyro sensor, the display object to be operated (the entire race screen background in FIG. 9A, the person in FIG. 9B, or the entire screen). , In FIG. 9C, the door knob) is made to perform an operation of rotating the display object associated with the operation input information. Further, the display object can be rotated at a predetermined angular velocity based on the angular velocity included in the output data received from the input detection unit.

10A and 10B are examples of screens displayed on the HMD based on the input from the microphone sensor according to this embodiment. When the type of output data received from the input detector is the user's voice input from the microphone, for the display object (cat in FIG. 10A, the character "wa" in FIG. 10B) associated with the voice input. , A predetermined motion (in FIG. 10A, the cat is surprised, and in FIG. 10B, the character "wa" is enlarged). Furthermore, the size and speed of the enlarged display object can be changed based on the volume and direction from the microphone sensor.

FIG. 11 is a flowchart showing a processing flow of the controller 100.
First, the interface program 106A is downloaded to the controller 100 by the antenna provided in the controller 100 via the Internet and installed (step 1100). By installing the interface program 106A, the controller can operate the screen of the HMD 140 by, for example, a known mobile terminal (smartphone), even if the controller is not a dedicated controller for the main unit.

Next, the operation unit 202 acquires input information regarding a predetermined operation / operation performed by the user on the controller and stores it in the memory (step 1102).
Next, the transmission unit 204 uses various inputs related to a predetermined operation, various input information related to a predetermined operation, and / or images, moving images, position information stored in the memory of the controller, or a combination thereof as output data. It is transmitted to the main body device 120 (step 1104). The transmission unit 204 can transmit the output data corresponding to the required user operation according to the content of the application being executed among the output data in accordance with the request from the input detection unit 304 of the main unit.

FIG. 12 is a flowchart showing a processing flow of the main body device 120.
The interface program 126A on the main unit side is installed in the main unit. The main unit may be connected to an external network to acquire the interface program 126A, or may acquire the interface program 126A using a storage medium (step 1200).

The operation determination unit 302 determines a required user operation that allows the application to input to the operating user according to the content of the application being executed in the main unit (step 1202).

Next, the input detection unit 304 transmits the output data corresponding to the user operation required by the running application from the output data transmitted from the transmission unit 204 of the controller 100 to the network 180 from the transmission unit 204 of the controller 100. Receive and detect via. (Step 1204).

Next, the operation determination unit 306 determines an operation to be performed by the display object in the virtual space of the running application according to the output data corresponding to the required user operation (step 1206).

Next, the display calculation unit 308 performs a display calculation of the display object based on the operation determined by the operation determination unit 306 and the operation data for the display object associated with the operation, thereby displaying the image of the display object. Generate (step 1208).

Next, the image generation output unit 310 generates an image in which the generated display object is arranged on the screen displayed on the display of the activated head-mounted device for each frame, and outputs the image to the HMD (step 1210). ).

FIG. 13 is a flowchart showing a flow of processing for connecting the controller 100 and the main unit 120 on the application. The controller 100 and the main unit 120 are connected to each other so as to be able to communicate with each other via the network 180.

First, in the main unit 120, an application for connecting to the HMD and operating the HMD is started (step 1300).
Next, the interface program 106A is executed on the controller 100 side, and the controller 100 searches whether an application including the interface program 126A is stored in the main unit (step 1302).

Next, when an application including the interface program 126A is searched in the main unit 120, the controller 100 transmits the recognition information for the search to the interface program 126A of the main unit 100 (step 1304).

Next, the controller 100 displays the application including the recognized interface program 126A on the display 108. Then, when the displayed recognized application is tapped by the user, the controller 100 transmits a pairing request to the main unit device side interface program 126A (step 1306).

Next, the main unit device side interface program 126A transmits the pairing response to the interface program 106A of the controller 100 in response to the pairing request transmitted from the controller (step 1308).

Next, the interface program 106A of the controller 100 transmits information indicating that the controller 100 and the main body device 120 have been paired to the main body device side interface program 126A (step 1310). As a result, the connection between the controller 100 and the main unit 120 on the application is completed.

FIG. 14 shows a system configuration diagram of a system composed of a plurality of controllers according to a second embodiment of the present invention. The plurality of controllers 100 are connected to each other via the Internet instead of the main unit 120. Hereinafter, the first controller 100A and the second controller 100B can each include a processor, a memory, an interface, and a sensor display as shown in FIG. Further, the first controller and the second controller can each include the functional configuration shown in FIG. 2 or a part of the functional configuration shown in FIG. Further, the controllers 100A and 100B can be configured by three or more controllers.

In a conventional system, when one operation screen is shared and operated by a plurality of users located in the same space, the load on the main body device is high because the plurality of controllers are connected to one main body device. It was. Further, in the conventional system, when one operation screen is shared and operated by a plurality of users located in different spaces, the plurality of controllers are connected to the main body devices existing in the separate spaces. Then, each main unit connects to the server via the network. In this case, a main unit capable of connecting to an external network is required for each of the plurality of controllers.

In this embodiment, only one controller is connected to one main unit, and the controllers are connected to the server via the Internet. As a result, various operations can be performed on one operation screen among a plurality of users without increasing the load on one main unit. Further, a plurality of users do not have to be located in the same space, but can be located in different spaces.

FIG. 15 is an exemplary user display screen in which a plurality of users perform operations on one operation screen. In the embodiment according to FIG. 15, user A is equipped with a head mount device, and user B is not equipped with a head mount device. User B, who is not equipped with the head mount device, can also see the operation screen of user A through the display of the second controller, or swipe the screen during operation of user A by the sensor display of the second controller. , Tap or pinch operation input, operation input by the acceleration sensor of the second controller, angular velocity sensor can be performed. For example, as shown in the figure, while user A is playing a racing game, another user B uses the sensor display of user B's controller to display an obstacle on the user A's operating screen (in the figure, in the figure). Bananas) can be placed.

Although the embodiment of the present invention has been described above, the present invention is not limited to the above embodiment. Those skilled in the art will appreciate that various embodiments may be modified without departing from the spirit and scope of the invention described in the preceding claims.

Claims (9)

A program that is run on a computer to provide virtual space via an immersive head-mounted display, said program to the computer.
The step of acquiring the output data output from the operating device held by the user, and
A step of controlling the virtual space based on the acquired output data,
The step of outputting the image of the virtual space to the immersive head-mounted display, and
A step of virtually displaying the sensor display of the operating device in the real space on the virtual space,
To run,
When the output data is data output from a motion sensor that detects the motion of the operating device, the control step is the first object displayed in the virtual space. When the operation is performed and the output data is data output from the touch sensor by the user's touch operation on the touch sensor of the operating device, the first object is different from the first operation. A program that causes the second operation to be performed. The program according to claim 1, wherein the operating device and the computer are connected by the operating device-side interface program acquired by the operating device via a network and the computer-side interface program of the computer. The program according to claim 1 or 2, wherein the output data includes data output from the operator by the user's operation on the operator of the operating device. The program according to any one of claims 1 to 3, wherein the output data includes image data output from a memory included in the operating device. The output data further includes data for manipulating an object based on the image data.
The program according to claim 4, wherein in the control step, the operation of an object based on the image data is controlled in the virtual space based on the data for the operation. The program according to any one of claims 1 to 5, wherein the output data includes data output from the voice input device by voice input of the user to the voice input device of the operation device. The program according to any one of claims 1 to 6, wherein the output data includes data output from a sensor of the operating device that detects geographical information about the operating device. A computer for providing virtual space through an immersive head-mounted display.
The computer is a processor
Equipped with memory to store programs
The processor executes the program.
Acquires the output data output from the operating device held by the user,
The virtual space is controlled based on the acquired output data, and the virtual space is controlled.
The image of the virtual space is output to the immersive head-mounted display, and the image is output to the immersive head-mounted display.
The sensor display of the operating device in the real space is virtually displayed on the virtual space.
In the control of the virtual space, when the output data is the data output from the motion sensor that detects the movement of the operating device, the first object displayed in the virtual space is the first. When the output data is data output from the touch sensor by the user's touch operation on the touch sensor of the operating device, the first object is referred to as the first operation. A computer that causes a different second action. A computer-run method for providing virtual space through an immersive head-mounted display.
The step of acquiring the output data output from the operating device held by the user, and
A step of controlling the virtual space based on the acquired output data,
A step of virtually displaying the sensor display of the operating device in the real space on the virtual space,
The step of outputting the image of the virtual space to the immersive head-mounted display, and
Including
When the output data is data output from a motion sensor that detects the motion of the operating device, the control step is the first object displayed in the virtual space. When the operation is performed and the output data is data output from the touch sensor by the user's touch operation on the touch sensor of the operating device, the first object is different from the first operation. A method of performing a second operation.

Images