JP2016081408A - Screen operation system by means of head-mounted display and controller cooperating with each other, program, and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a system for operating a screen by a head-mounted display and a controller cooperating with each other, a method, and a program.SOLUTION: A screen operation system includes: an operation determination section 302 which determines a required user operation according to the contents of a running application; an input detection section 304 which detects output data corresponding to the required user operation, from data output from a controller; an operation determination section 306 which determines motion of a display object, on the basis of the detected output data; a display calculation section 308 which calculates a display parameter of the display object, on the basis of the determined motion of the display object and the detected output data, to generate an image of the display object; and an image generation output section 310 which generates an image by arranging the generated image of the display object in an image on a virtual space, to be output to the head-mounted display. The controller and the system 120 are connected to each other.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a system, a method, and a program for operating a display on a head mounted display by linking a head mounted display and a controller, and more specifically, has a communication function between the head mounted display and an external network. The present invention relates to a screen operation system linked with a controller.

  2. Description of the Related Art A head mounted display (HMD) that is mounted on a user's head and can present an image to the user with a display or the like disposed in front of the eyes is known. Since the conventional HMD controller is connected to an external network such as the Internet via the main unit, the controller itself does not communicate with the external network. Moreover, the conventional controller requires a dedicated controller for connecting to the HMD.

  Furthermore, conventionally, a controller used for various operations of a display image in the HMD is mainly operated by buttons and keys, and does not include a sensor display. Such a controller has recently been developed with a built-in sensor (for example, an acceleration sensor) capable of detecting motion and detecting the motion of the controller.

  In the system according to the background art, since the controller itself does not have a function of connecting to an external network, the controller itself is always connected to the main unit when connecting to the Internet.

  In addition, the controller connected to the conventional HMD can be operated by input from buttons, keys, and sensors that can detect movement, but since it does not have a sensor display, it can perform intuitive and finer operations. could not. In particular, in the case of operation with buttons and keys, if the user is completely immersed in the virtual space with the HMD attached, the current state that the user is trying to operate (which key or button the user's finger is placed on) ) Cannot be recognized by the user, and if the user's finger is removed from the desired key or button, an operation that the user does not desire has been performed.

  In addition, when a single operation screen is shared and operated by a plurality of users located in the same space, a load on the main unit is high because a plurality of controllers are connected to one main unit in the conventional system. . In a conventional system, when a single operation screen is shared and operated by a plurality of users located in separate spaces, the plurality of controllers are connected to separate main units for each space, and are connected via a network. Each main unit was connected to a server for communication. In this case, separate main body devices are required for a plurality of controllers.

Further, when a dedicated controller is required for the operation of the HMD, the user has to carry the dedicated controller for each HMD operation.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a system in which an HMD and a controller having a communication function with the outside are linked. Furthermore, according to the present invention, since the controller has a communication function with the outside, when connecting to the main unit, software necessary for connection can be downloaded from the external server. There is no need, and generally distributed terminals such as mobile terminals and smartphones can be used.

(1) The present invention
A system for outputting a generated image to a head mounted display, the system comprising:
An operation determination unit that determines a user operation required according to the contents of the application being executed;
An input detection unit for detecting output data corresponding to the requested user operation from output data from the controller;
A motion determination unit that determines the motion of the display object based on the detected output data;
A display calculation unit that calculates a display parameter of the display object based on the determined movement of the display object and the detected output data, and generates an image of the display object displayed on the virtual space;
An image generation output unit that generates an image in which an image of a display object generated in an image in a virtual space is arranged and outputs the generated image to a head mounted display connected to the system;
With
The controller and the system are connected by a controller-side interface program received from a server via the Internet by the communication unit of the controller and the system-side interface program of the system.

  According to the present invention, by using a controller having a communication function with an external network, even if the controller is not connected to the main unit, the controller itself can obtain various information, videos, images, Voice and other information (for example, map information, weather information, stock price information, application) can be obtained and reflected in the virtual space presented on the HMD. Furthermore, since the controller has a communication function with the outside, the software can download software necessary for connection with the main unit from an external server when connecting to the main unit. It need not be dedicated to the device. Therefore, according to the present invention, it is possible to use a portable terminal such as a smartphone, which has been widely used in recent years, as an HMD controller, and the user can operate the HMD without requiring a dedicated controller. .

(2) Further, according to the present invention, the output data from the controller can be output data generated based on an operation input by a user from the sensor display of the controller. Further, according to the present invention, the operation input can be a swipe operation, a tap operation, or a pinch-in / pinch-out operation.

  According to the present invention, by using a controller including a sensor display, a user can perform various detailed operations (swipe operation, tap operation, pinch operation) on the HMD in an intuitive manner. When the user wears the HMD and is completely immersed in the virtual space, the user recognizes the current state that the user is trying to operate (which key or button the user's finger is placed on). I can't. The controller according to the present invention includes a sensor display, and the user can perform various intuitive operations.

(3) Further, according to the present invention, the output data from the controller can be output data generated based on a voice input by a user from the microphone of the controller. Further, according to the present invention, the image of the display object displayed on the virtual space can be changed based on the voice input by the user.

  According to the present invention, by using a controller including a microphone, a user can change an image displayed on the HMD by performing an intuitive operation (voice input) on the HMD.

(4) Further, according to the present invention, the output data from the controller can be output data generated based on an operation input by a user from an acceleration sensor or a gyro sensor of the controller.

  According to the present invention, by using a controller having an acceleration sensor or a gyro sensor, various intuitive operations by the user can be performed by detecting the swing width, swing period, speed, and strength of the controller. .

(5) Further, according to the present invention, the output data from the controller includes an image stored in a memory of the controller and output data generated based on an operation input by a user from the sensor display of the controller, The determination unit may determine a display object based on an image and determine an operation of the display object based on the detected output data. Furthermore, in the present invention, the controller can be a smartphone.

  According to the present invention, in the case where the controller is a portable terminal that is always carried by a user such as a smartphone, images, videos, and sounds taken by the user in the past are stored, and the user himself / herself stored in the smartphone It is possible to reflect on the HMD using images, videos, and sounds taken in the past. Furthermore, when the controller is a mobile terminal that is always carried by a user such as a smartphone, the user always uses the smartphone that the user always carries as a controller for HMD operation without carrying a dedicated controller for each HMD operation. Can be used.

(6) Further, according to the present invention, the output data from the controller includes output information generated based on controller position information from the controller GPS and operation input by the user from the sensor display of the controller, The unit may calculate display parameters of the display object based on the position information and the determined operation of the display object, and generate an image of the display object displayed in the virtual space. .

  According to the present invention, it is possible to change the image of the display object displayed on the virtual space presented to the HMD corresponding to the position information of the controller currently placed.

(7) Further, according to the present invention, the operation determination unit determines a plurality of user operations required according to the contents of the application being executed,
The input detection unit detects output data corresponding to the plurality of user operations obtained from the output data from the controller,
The motion determination unit determines the motion of the display object based on the first output data among the detected output data, and converts the motion of the display object based on the second output data among the detected output data. Features.

  According to the present invention, by providing the acceleration sensor and the gyro sensor, the operation of the display object (the display object jumps) corresponding to one operation input (for example, a tap operation) to the controller can be performed by a simple switching operation such as a shake of the terminal. Instantaneous conversion to another display object action (display object sliding) is possible. When the user wears the HMD and is immersed in the virtual space, the user cannot directly see the controller being operated. With the acceleration sensor and the gyro sensor, the user can surely and instantaneously convert the operation of the display object by a simple and intuitive operation such as shaking the terminal.

(8) Further, the present invention is a program for causing a processor to execute a method of outputting a generated image to a head mounted display, and the method includes:
An operation determination step for determining a user operation required according to the contents of the application being executed;
An input detection step of detecting output data corresponding to the required user operation from output data from the controller;
An operation determination step for determining the operation of the display object based on the detected output data;
A display calculation step of calculating a display parameter of the display object based on the determined operation of the display object and the detected output data, and generating an image of the display object displayed on the virtual space;
An image generation output step of generating an image in which an image of a display object generated in an image on a virtual space is arranged and outputting the generated image to a head mounted display connected to the system;
With
The controller and the system are connected to each other by a controller-side interface program received from a server via the Internet by the communication unit of the controller and a system-side interface program of the system. .

(9) Further, the present invention is a system control method for outputting a generated image to a head mounted display,
An operation determination step for determining a user operation required according to the contents of the application being executed;
An input detection step of detecting output data corresponding to the required user operation from output data from the controller;
An operation determination step for determining the operation of the display object based on the detected output data;
A display calculation step of calculating a display parameter of the display object based on the determined operation of the display object and the detected output data, and generating an image of the display object displayed on the virtual space;
An image generation output step of generating an image in which an image of a display object generated in an image on a virtual space is arranged and outputting the generated image to a head mounted display connected to the system;
With
The method is characterized in that the controller and the system are connected by a controller-side interface program received from a server via the Internet by the communication unit of the controller and a system-side interface program of the system. .

  The above and other features and advantages of the present invention will become apparent from the following more specific description of the embodiments of the invention, the accompanying drawings, and the appended claims.

1 shows a system configuration diagram of a system according to an embodiment of the present invention. FIG. 2 shows a functional configuration of the controller 100 according to one embodiment of the present invention. 2 shows a functional configuration of main device 120 according to an embodiment of the present invention. 4 shows a screen displayed on an HMD generated based on a user's swipe operation according to an embodiment of the present invention. The screen displayed on HMD generated based on a user's tap operation according to one example of the present invention is shown. The screen displayed on HMD generated based on a user's tap operation according to one example of the present invention is shown. The screen displayed on HMD generated based on a user's tap operation according to one example of the present invention is shown. 4 shows a screen displayed on an HMD generated based on a user pinch-in / pinch-out operation according to an embodiment of the present invention. 4 shows a screen displayed on an HMD generated based on a user pinch-in / pinch-out operation according to an embodiment of the present invention. 4 shows a screen displayed on an HMD generated based on a user pinch-in / pinch-out operation according to an embodiment of the present invention. 4 shows a screen displayed on an HMD generated based on an operation input from an acceleration sensor / gyro sensor according to an embodiment of the present invention. 4 shows a screen displayed on an HMD generated based on an operation input from an acceleration sensor / gyro sensor according to an embodiment of the present invention. 4 shows a screen displayed on the HMD generated based on the voice input from the acceleration sensor / gyro sensor according to one embodiment of the present invention. 4 shows a screen displayed on an HMD generated based on an operation input from an acceleration sensor / gyro sensor according to an embodiment of the present invention. 4 shows a screen displayed on an HMD generated based on an operation input from an acceleration sensor / gyro sensor according to an embodiment of the present invention. 4 shows a screen displayed on an HMD generated based on an operation input from an acceleration sensor / gyro sensor according to an embodiment of the present invention. 6 shows a screen displayed on an HMD generated based on voice input from a microphone sensor according to one embodiment of the present invention. 4 shows a screen displayed on an HMD generated based on an input from a microphone sensor according to an embodiment of the present invention. FIG. 3 shows a flowchart diagram illustrating a processing flow of a controller according to an embodiment of the present invention. The flowchart figure which shows the flow of a process of the main body apparatus according to one Example of this invention is shown. The flowchart figure which shows the flow of the process which connects a controller and a main body apparatus on an application according to one Example of this invention is shown. 1 shows a system configuration diagram of a system including a plurality of controllers according to an embodiment of the present invention. FIG. 4 illustrates an exemplary user display screen in which a single operation screen is operated among a plurality of users according to an embodiment of the present invention.

Hereinafter, this embodiment will be described. FIG. 1 shows a system configuration diagram of a system according to an embodiment of the present invention.
In this system, the controller 100 and the main device 120 are connected via a wireless network such as Wi-Fi or Bluetooth (registered trademark), and the main device 120 and the head mounted display 140 are connected to the Wi-Fi or Bluetooth (registered trademark). Connected via a wireless or wired network. The controller 100 further includes an antenna 110, and can communicate with the server 160 by connecting to a network such as the Internet without going through the main unit. The server 160 is realized by a well-known server computer mainly configured with a processor, various storage devices, and a data communication device, and transmits and receives data to and from the controller 100. In addition, when there are a plurality of controllers, the server 160 manages communication performed between the controllers.

Controller Configuration The controller 100 can include at least one processor 102, at least one interface 104, at least one memory 106, at least one sensor display 108, and an antenna 110. The controller 100 can be realized by, for example, a known smartphone or mobile terminal. The controller 100 is held and worn by the user himself / herself, and can detect various operation inputs according to the user's operation and user's operation input and voice input according to the user's movement.

  The controller-side processor 102 is configured to perform various processes such as execution of the controller-side interface program 106A and instruction instruction to each functional unit. The controller-side interface program 106A is for causing a transmission unit and a reception unit (FIG. 2) of the user controller described later to function.

  The controller-side interface program 106 </ b> A transmits output data from the controller 100 to the main device 120 via the network 180 by causing a transmission unit 204 described later to function. In addition, by causing the receiving unit 206 described later to function, various types of data including experience information such as images experienced by the user in the virtual space can be received from the main body device 120 to the controller 100.

  The controller-side interface 104 is an interface that transmits output data to the main device 120 and receives data from the main device 120 via the network 180, and is realized by installing the controller-side interface program 106A. More specifically, the controller-side interface 104 transmits output data such as data obtained from various sensors built in the controller, images stored in the memory of the controller, and / or the head mounted by the user. This is an interface for receiving experience information such as images and moving images experienced by the user in the virtual space from the main unit when the device is attached and immersed in the virtual space.

  The controller side memory 106 stores a controller side interface program 106A. In addition, the controller-side memory 106 stores various data such as images and moving images captured in the past by the user, memo information stored in the past, and data obtained from various sensors generated according to user operations and operations. Can be stored.

  The sensor display 108 is a touch sensor, and detects a swipe operation, a pinch-in / pinch-out operation, and a tap operation from the user. By providing the sensor display, it is possible to detect a swipe operation and a pinch-in / pinch-out operation from a user that cannot be performed by a controller including only an acceleration sensor, a gyro sensor, a key, and a button. The user can perform intuitive detailed operations by swiping, pinching in and pinching out. Furthermore, when the user is immersed in the virtual space, the finger may come off from the keys and buttons of the controller being operated. In such a case, the head mount device must be removed once to check the position of the outside world. It was. The swipe operation, tap operation, or pinch operation using the sensor display can be performed intuitively without confirming the positions of the keys and buttons.

  The controller 100 includes an antenna 110, and the controller 100 itself has a communication function with an external network such as the Internet. The controller 100 acquires various information, weather information, map information, applications, and the like that can be obtained via an external network in real time without applying a load to the main unit during application execution, and executes them on the main unit. It becomes possible to reflect on the display screen to the application and HMD. In addition, the controller can download the controller-side interface program 106A from the server 160 to the user controller in advance via the network such as the Internet using the antenna 110 and install it in the user controller. By installing the controller-side interface program 106A in the controller 100, the user can connect the controller and the main device using any controller, for example, a smartphone, without using a dedicated controller for the head mount device. Can be reflected on the display of the head mounted device.

  The controller 100 can further include a microphone that detects voice input from the user, an acceleration sensor that detects motion input by the user, a gyro sensor, and buttons, keys, a camera, GPS, and a vibration device (not shown).

  The microphone detects the utterance of the user by the user holding and wearing the controller. By providing the microphone, the words spoken by the user can be reflected in the display object displayed on the HMD. In addition, the voice recognition application stored in the memory is used for voice recognition of the words spoken by the user. For example, when a spell is cast on a game application running on the main unit, the door opens. Can be made to display objects.

  The acceleration sensor and the gyro sensor can detect the movement of the user holding and wearing the controller, for example, the movement of the user swinging the controller and the movement of throwing an object. Further, the number of steps of the user can be detected by the acceleration sensor. In addition, by providing an acceleration sensor, a gyro sensor, and a sensor display, the operation of the display object (the display object jumps) corresponding to one operation input (for example, a tap operation) to the controller can be performed with a simple switching operation such as a terminal shake. Instantaneous conversion to another display object action (display object sliding) is possible. In particular, since the tap operation is a simple operation for the user wearing the HMD, various display object operations such as shooting, cutting, and jumping operations are assigned to the tap operation on the application. When the user wears the HMD and is immersed in the virtual space, the user cannot directly see the controller being operated. With the acceleration sensor and the gyro sensor, the user can surely and instantaneously convert the operation of the display object by a simple and intuitive operation such as shaking the terminal.

The buttons and keys accept an operation input by a user who holds and wears the user controller.
The camera allows the user to take a picture or a moving picture, and the taken picture or moving picture is stored in the memory of the user controller. Images and videos captured in the past by the user can be reflected in the virtual space displayed on the head mounted display.

  The GPS detects the position of the user holding and wearing the controller. Therefore, for example, the screen display of the head mounted display can be generated according to the detected position of the user. For example, when the user is located in Hokkaido, the character of the display object that the user operates in the virtual space displayed on the HMD can be the Hokkaido character.

  For example, when a user who is wearing a head mounted device and is playing a game receives an attack or the like in a virtual space displayed on the head mounted display, the vibration device is linked with the game. Can vibrate. Information experienced by the user during application execution in the virtual space is transmitted to the interface program 106A via the network 180 by the main body side interface program 126A. Then, based on the information experienced by the user received by the controller-side interface program 106A, the vibration device vibrates the controller. As the controller vibrates, the user can further immerse in the virtual space.

Configuration of Main Unit The main unit 120 transmits and receives data to and from the user controller 100 and the head mounted display 140, and includes at least one processor 122, at least one interface 124, and at least one memory 126. Can do.

  The main device side processor 122 instructs each function unit, executes various applications such as a game, executes the main device side interface program 126A, executes an image generation program 126B for performing image generation output processing, and sound generation processing It is configured to perform various processes such as.

  The main apparatus side interface 124 receives output data transmitted from the controller 100 via the network, transmits predetermined data from the main apparatus 120 via the network, and outputs a generated image in which display objects are arranged to the HMD 140. Output interface.

  The main device side memory 126 stores a main device side interface program 126A, an image generation program 126B, and programs and data for realizing various functions executed by the processor of the main device.

  The main unit side interface program 126A is for causing an input detection unit 304 (FIG. 3) described later to function. By executing the main body side interface program 126A, the main body 120 detects and receives output data transmitted from the controller 100, and transmits various data from the main body 120 to the controller 100. The main unit side interface program 126A is preinstalled in the main unit. For example, the main unit side interface program 126A can be downloaded to the main unit in advance via a network (not shown) such as the Internet or an intranet connected to the main unit, or can be installed using a recording medium. By installing the main body apparatus-side interface program 126A and installing a controller-side interface program 106A, which will be described later, in the controller, the user can use an arbitrary controller such as a portable terminal or a smartphone without using a dedicated controller for the head mount apparatus. The user input or the like from the user controller 100 can be reflected in the HMD, or display on the HMD and application information during operation can be transmitted to the user controller. Furthermore, the main body apparatus side interface program 126A can acquire the output data from the plurality of controllers, respectively, so that the plurality of controllers can be connected to one main body apparatus 120.

The image generation program 126B is for causing an operation determination unit 302, an operation determination unit 306, a display calculation unit 308, and an image generation output unit 310 to be described later to function.
The HMD 140 is connected to the main device 120 and displays a generated image output from the main device 120. The HMD 140 may virtually display a sensor display (602 in FIG. 6), buttons, keys, and the like of the controller held by the user.

Functional Configuration of Controller FIG. 2 is a diagram showing a functional configuration of the controller 100. As shown in the figure, the controller 100 can include an operation unit 202, a transmission unit 204, a reception unit 206, a communication unit 208, an imaging unit 210, or a part of these functions. These functions are realized by the processor 102 executing a predetermined program stored in the memory 106 of the user controller 100. The transmission unit and the reception unit of the user controller can be realized by executing the controller side interface program 106A by the controller side processor. The interface program 106A is downloaded in advance from the server 160 via the Internet using the antenna 130 of the controller and installed in the controller 100.

  The operation unit 202 acquires an input related to a predetermined operation / operation performed on the controller by the user from various sensors provided in the controller. Various sensors are a display sensor, an acceleration sensor, a gyro sensor, a microphone sensor, and a GPS sensor. Various sensors can be combined with inputs from other operation units such as buttons and keys (for example, operation inputs for performing an operation of shaking the controller while pressing a button of the controller). When the user moves the controller, the operation unit acquires the acceleration and angular velocity for the operation from the acceleration sensor and the gyro sensor. When the user performs a swipe operation, a pinch-in / pinch-out operation, or a tap operation on the sensor display of the controller, the operation unit may contact the type of the operation and / or various data relating to the operation, for example, contact with the display ( Tap), contact time (long press / tap), number of contacts within a predetermined time (continuous hit, single press), displacement amount of continuous contact to the display (swipe distance, pinch distance), contact displacement direction (swipe direction, Pinch direction) and contact displacement speed (swipe speed, pinch speed). When the user speaks to the microphone of the controller, the operation unit 202 acquires user voice input from the microphone. Next, the operation unit 202 sequentially stores these pieces of information (acceleration / angular velocity, operation input information, voice input information, position information) in the memory.

  In this embodiment, the acceleration and angular velocity obtained from the acceleration sensor and gyro sensor are referred to as operation input information, and obtained from the sensor display when the user performs a swipe operation, a tap operation, or a pinch operation on the sensor display. The contact to the sensor display, the displacement amount of the continuous contact to the display, the contact displacement direction, and the contact displacement speed are referred to as operation input information.

  The transmission unit 204 transmits these pieces of information stored in the memory as output data to the input detection unit 304 via the network 180 in response to a request from the main device. The output data includes operation input information, operation input information, voice input information, position information, and images stored in the memory 106 of the controller 100 regarding predetermined operations / operations performed on the controller 100 from the operation unit 202. It corresponds to a moving image or a combination thereof.

  The controller 100 can include a receiving unit 206 that receives information transmitted from the main device 120. The receiving unit 206 receives information experienced by the user in the virtual space when the user wears the HMD and is immersed in the virtual space. For example, when a user who is playing a game receives an attack or the like in a virtual space displayed on the HMD 140, the attacked information is received. Then, the controller-side processor can vibrate the controller by executing a vibration function based on the received attacked information. Further, the receiving unit 206 can receive an image during game play from the main device. The controller side processor 102 can store the received video in the memory of the controller or display it on the controller side display 108.

  The communication unit 208 is used for downloading various data, for example, programs from the server 160, uploading a user's operating video to the server 160, or directly communicating with other controllers when there are a plurality of controllers. Yes, usually communicate over the Internet. By including the communication unit 208, the controller 100 can be connected to the external server 160 even if it is not connected to the main device. Therefore, even if the controller is not connected to the main unit 120, the controller-side interface program 106A can be installed and updated at any location as long as there is an environment that can be connected to the Internet. Further, the controller 100 can acquire various information (meteorological information, stock price information, map information) that can be acquired from the Internet, and can reflect the information on the user operating screen and the game content being operated by the user. In addition, when there are a plurality of controllers, the controller can communicate with other controllers via the communication unit without going through the main unit.

  The imaging unit 210 can take images and moving images in real space. By providing the imaging unit, it is possible to reflect an image or a moving image captured in advance by the user on the user operating screen or the game screen being played.

  The user controller 100 can further include a GPS (not shown). By providing the GPS, the position information of the controller acquired by the GPS can be reflected on the screen during the user operation. For example, when the controller is located in Hokkaido, the character of the display object operated by the user in the virtual space displayed on the HMD is a Hokkaido character, or the user operation screen of the virtual space displayed on the HMD is the background of Hokkaido. It can be done.

Functional Configuration of Main Device FIG. 3 is a diagram illustrating a functional configuration of the main device 120. As shown in the figure, the main device 120 includes an operation determination unit 302, an input detection unit 304, an operation determination unit 306, a display calculation unit 308, and an image generation output unit 310. These functions are realized by a processor executing a predetermined program stored in a memory in the main device.

  The operation determination unit 302 first determines what user operation is requested to be input by the application based on the contents of the application being executed in the main device. The required user operation is a user operation on the controller that is determined based on the contents of the application being executed, the situation in which the user is currently operating on the application being executed, and the application being executed permits input. It is. For example, in an operation state in which a user plays a battle game in a virtual space and is in battle with an enemy, the required user operations corresponding to the operation state are, for example, “drill”, “shoot”, and “dodge the attack” .

  Next, the input detection unit 304 acquires output data corresponding to the requested user operation from the output data stored in the memory of the controller 100 via the network 180. For example, when the requested user operation is “talk”, the output data corresponding to the requested user operation is operation input information (contact on the display by a tap operation). The output data can also include the number of contacts within a predetermined time. Then, the input detection unit 304 outputs the detected output data to the operation determination unit 306. In this manner, the input detection unit 304 detects only the touch on the display corresponding to the tap operation from the operation input information of the output data when the user operation required in the application being executed is “speak”. And get. The operation determination unit 306 can ignore the voice input when the output data does not correspond to the requested user operation, for example, when the requested user operation is “talk”.

  As described above, the input detection unit 304 includes various information transmitted from the transmission unit 204 of the controller via the network 180, operation input information (acceleration and angular velocity obtained from the acceleration sensor and gyro sensor) by the user, and operation input. Information (type of swipe operation, pinch-in / pinch-out operation and / or contact with sensor display, displacement amount of continuous contact with display, contact displacement direction, contact displacement speed), voice input information (obtained from microphone sensor) Among the information that is sound, volume), image, moving image, position information, or a combination thereof, data corresponding to the requested user operation determined by the operation determination unit 302 is detected as output data from the controller.

  Further, the input detection unit 304 may detect output data at a predetermined timing for a predetermined time. Further, the input detection unit 304 can change the timing interval for receiving the output data. For example, if the display object is to be subjected to a rough operation such as turning a door knob in the virtual space, the detection timing interval is increased to detect the input 15 times per second, and the display object is moved to the vehicle in the virtual space. In the case of performing an operation necessary to reflect the detailed movement of the steering wheel sequentially on the screen, such as driving the vehicle, it is possible to detect input 60 times per second by reducing the detection timing interval.

  Table 1 exemplarily shows the relationship between the contents of the application being executed, the required user operation corresponding to the contents of the application being executed, and the output data corresponding to the required user operation. The contents of the application, required user operations, and output data shown in Table 1 are merely examples, and those skilled in the art will easily understand that other user operation states and other user operations exist. .

  Next, the action determination unit 306 determines an action to be performed on the display object displayed on the head mounted display, based on the output data detected by the input detection unit 304, in accordance with the requested user operation. For example, when the requested user operation is “speaking”, the action (speaking and scolding frequency) to be performed by the display object in the virtual space is determined according to the touch detection to the display included in the output data and the number of touches. To do. A display object is an object to be operated in a virtual space by the user. For example, when the user is playing a game, the display object is a game character or a game item, and when an image or video captured by the user is displayed. Are those images and videos.

  For example, when the requested user operation is “slide image”, the motion determination unit 306 acquires the swipe operation direction, displacement amount, and displacement speed from the operation input information of the output data. Then, the action determination unit determines the action (slide of the image) to be performed by the display object according to the direction of the swipe operation to the display, the displacement amount, and the displacement speed.

  For example, when the requested user operation is “attach two display objects”, the operation determination unit 306 includes the direction and displacement amount of the pinch-in operation (or swipe operation) to the display in the operation input information of the output data. Get the displacement speed. Then, the motion determination unit determines a motion to be performed by the display object (two display objects are combined into one) according to the direction of the pinch-in operation on the display, the amount of displacement, and the displacement speed. The motion determination unit can determine the direction in which the display object is moved and the speed at which the display object is moved according to the direction of the pinch-in operation on the display by the user, the displacement amount, and the displacement speed.

  For example, when the requested user operation is “handle operation”, the motion determination unit 306 acquires acceleration and angular velocity in the motion input information of the output data. Since the “handle operation” requires a fine operation, the motion determination unit can increase the number of times of acquiring acceleration and angular velocity in a certain time. Then, the motion determination unit determines a motion (changes the direction of the display object) to be performed by the display object according to the acceleration and the acceleration.

  For example, the motion determination unit 306 acquires acceleration and angular velocity in the motion input information of the output data when the requested user operation is “swing”. Note that when the “swing” operation is sufficient for rough detection, the motion determination unit can reduce the number of times of acquiring acceleration and angular velocity in a certain time. Then, the motion determination unit determines a motion (moving or swinging the display object) to be performed by the display object according to the acceleration and the acceleration. The motion determination unit can determine the strength and direction of swinging the display object according to the acceleration and the angular velocity.

  For example, when the requested user operation is “throw”, the motion determination unit 306 acquires acceleration and angular velocity in the motion input information of the output data. The motion determination unit determines a motion (throws the display object) to be performed by the display object according to the acceleration and the angular velocity. Further, the motion half-value portion can determine the strength and direction of throwing the display object according to the acceleration and angular velocity.

  For example, when the requested user operation is “loud”, the motion determination unit 306 acquires voice input information from the output data. Then, the motion determination unit determines a motion (enlarged display object) to be performed by the display object according to the volume. Furthermore, the motion determination unit can determine the size for enlarging the display object according to the volume.

  Next, the display calculation unit 308 performs display calculation of the display object based on the motion determined by the motion determination unit 306 and the operation data for the display object associated with the motion, thereby obtaining an image of the display object. Generate. The operation data includes acceleration, angular velocity, displacement amount, displacement velocity, direction, and the like when moving the display object. When generating a so-called three-dimensional image, a display parameter including data of each vertex of the display object (position coordinates of the vertex in the world coordinate system, color data) is calculated, and based on the calculated display parameter, The image of the display object is sequentially generated for each frame.

  The image generation / output unit 310 generates, for each frame, an image in which the calculated display objects are arranged on the screen displayed on the display of the activated head mount device, and outputs the generated image to the screen. The HMD displays the image output from the image generation output unit.

  FIG. 4 is an example of a screen 400 displayed on the HMD according to the present embodiment generated based on the user's swipe operation. When the output data acquired from the input detection unit is a swipe operation, the display object (six images in FIG. 5A) is slid in the swipe direction. As shown in FIG. 4, the display (402, 404) corresponding to the sensor display operated by the user is virtually displayed on the HMD so that the user wearing the HMD is completely immersed in the virtual space. Even in such a case, the user can surely grasp the operation that the user is performing or is about to perform.

  5A and 5B are examples of screens (500A and 500B) displayed on the HMD according to the present embodiment generated based on the user's tap operation. When the output data received from the input detection unit corresponds to the tap operation, the display object (the hand 502 in FIG. 5A and the gun 504 in FIG. 5B) that is the operation target shoots and strikes. For example, each time the user taps the sensor display of the controller 100, the display object performs a shooting / stroke action. Furthermore, when the output data includes the number of times tapped within a certain time, the speed of the motion of the object to shoot and beat can be changed according to the number of taps.

  FIG. 6 is an example of a screen 600 displayed on the HMD according to the present embodiment generated based on the user's tap operation. The display screen 600 of FIG. 6 includes a sensor display 602 that is virtually displayed in a virtual space. The virtually displayed sensor display 602 can virtually display a user's finger 604 (operation finger). Since the user who is immersed in the virtual space is wearing the HMD and is blocked from outside information, the user cannot recognize the controller in the real space in detail. By virtually displaying the sensor display of the controller in the real space on the virtual space, the user can grasp various operations on the controller without removing the head mounted display.

  7A, 7B, and 7C are examples of a screen displayed on the HMD according to the present embodiment generated based on the user's pinch-in / pinch-out operation. When the output data received from the input detection unit corresponds to a pinch-out operation, the display object associated with the operation input is zoomed in on the display object that is the operation target (the image in the black circle in FIG. 7A). Perform the action. Furthermore, when the amount of operation displacement is included in the output data received from the input detection unit, the screen operation amount for the zoom-in operation can be determined based on the amount of displacement. Further, when the type of output data received from the input detection unit is a pinch-out operation, a pinch-out is performed with respect to a display object (center circle drawn as 4 in FIG. 7B) selected by the user. When an operation is performed, the display object is separated into two objects (water droplets in FIG. 7B) (the object is divided into two). The separation of objects can also be realized by a swipe operation. When the output data received from the input detection unit is a pinch-in operation, the display object that is the operation target (two circles drawn as 2 in FIG. 7C) is combined into one big circle drawn as 4 (Two objects can be attached).

  8A, 8B, and 8C are examples of screens displayed on the HMD according to the present embodiment. When the output data received from the input detection unit corresponds to the user's motion input information from the acceleration sensor / gyro sensor, the display object to be operated (sword in FIG. 8A, fishing rod in FIG. 8B, dart in FIG. 8C) is displayed. On the other hand, the display object is caused to perform an operation of swinging a sword in FIG. 8A or an operation of throwing a fishing rod or a dart in FIGS. 8B and 8C. Furthermore, based on the acceleration included in the output data received from the input detection unit, the display object can be moved at a predetermined speed and acceleration, or the display object can be shaken or thrown at a predetermined strength. The motion determination unit can further move the display object in a predetermined direction when the output data received from the input detection unit includes an angular velocity.

  9A, 9B, and 9C are examples of screens displayed on the HMD based on inputs from the acceleration sensor / gyro sensor according to the present embodiment. When the output data received from the input detection unit corresponds to the user's motion input information from the acceleration sensor / gyro sensor, the display object that is the operation target (the entire race screen background in FIG. 9A, the person in FIG. 9B, or the entire screen) In FIG. 9C, the door knob) is caused to rotate the display object associated with the motion input information. Furthermore, the display object can be rotated at a predetermined angular velocity based on the angular velocity included in the output data received from the input detection unit.

  10A and 10B are examples of a screen displayed on the HMD based on an input from the microphone sensor according to the present embodiment. When the type of the output data received from the input detection unit is the user's voice input from the microphone, the display object associated with the voice input (a cat in FIG. 10A and the word “War” in FIG. 10B) , A predetermined operation (an operation that surprises a cat in FIG. 10A and an operation that expands the word “War” in FIG. 10B) is performed. Furthermore, the size and speed of the display object to be enlarged can be changed based on the volume and direction from the microphone sensor.

FIG. 11 is a flowchart showing a process flow of the controller 100.
First, the interface program 106A is downloaded to the controller 100 via the Internet using the antenna provided in the controller 100 and installed (step 1100). By installing the interface program 106A, the controller can operate the screen of the HMD 140 using, for example, a known portable terminal (smartphone), instead of a dedicated controller for the main device.

Next, the operation unit 202 acquires input information related to a predetermined operation / operation performed on the controller by the user and stores it in the memory (step 1102).
Next, the transmission unit 204 outputs, as output data, various inputs related to a predetermined operation, various input information related to a predetermined operation, and / or images, moving images, position information stored in the memory of the controller, or a combination thereof. It transmits to the main unit 120 (step 1104). In accordance with a request from the input detection unit 304 of the main device, the transmission unit 204 can transmit output data corresponding to a requested user operation in accordance with the contents of the application being executed among the output data.

FIG. 12 is a flowchart showing a process flow of the main device 120.
A main unit side interface program 126A is installed in the main unit. The main device may acquire the interface program 126A by connecting to an external network, or may acquire the interface program 126A using a storage medium (step 1200).

  The operation determination unit 302 determines a requested user operation that permits input to the user who is operating the application according to the contents of the application being executed in the main device (step 1202).

  Next, the input detection unit 304 outputs, from the output data transmitted from the transmission unit 204 of the controller 100, output data corresponding to the user operation required by the application being executed via the network 180 from the transmission unit 204 of the controller 100. To receive and detect. (Step 1204).

  Next, the action determination unit 306 determines an action to be performed on the display object in the virtual space of the application being executed according to the output data corresponding to the user operation that is obtained (step 1206).

  Next, the display calculation unit 308 performs display calculation of the display object based on the motion determined by the motion determination unit 306 and the operation data for the display object associated with the motion, thereby obtaining an image of the display object. Generate (step 1208).

  Next, the image generation / output unit 310 generates an image in which the generated display object is arranged on the screen displayed on the display of the activated head mount device for each frame, and outputs the generated image to the HMD (step 1210). ).

  FIG. 13 is a flowchart showing a flow of processing for connecting the controller 100 and the main device 120 on an application. The controller 100 and the main device 120 are connected to each other via a network 180 so as to communicate with each other.

First, in main device 120, an application for connecting to HMD and operating HMD is activated (step 1300).
Next, the interface program 106A is executed on the controller 100 side, and the controller 100 searches whether an application including the interface program 126A is stored in the main unit (step 1302).

  Next, when an application including the interface program 126A is searched for in the main body device 120, the controller 100 transmits recognition information for the search to the interface program 126A of the main body device 100 (step 1304).

  Next, the controller 100 displays an application including the recognized interface program 126 </ b> A on the display 108. When the displayed recognized application is tapped by the user, the controller 100 transmits a pairing request to the main body apparatus side interface program 126A (step 1306).

  Next, the main unit side interface program 126A transmits a pairing response to the interface program 106A of the controller 100 in response to the pairing request transmitted from the controller (step 1308).

  Next, the interface program 106A of the controller 100 transmits information indicating that the controller 100 and the main device 120 have been paired to the main device side interface program 126A (step 1310). Thereby, the connection on the application between the controller 100 and the main device 120 is completed.

  FIG. 14 shows a system configuration diagram of a system including a plurality of controllers according to the second embodiment of the present invention. The plurality of controllers 100 are connected not via the main unit 120 but via the Internet. Hereinafter, each of the first controller 100A and the second controller 100B may include a processor, a memory, an interface, and a sensor display as shown in FIG. In addition, each of the first controller and the second controller can include the functional configuration illustrated in FIG. 2 or a part of the functional configuration illustrated in FIG. Further, the controllers 100A and 100B can be configured by three or more units.

  In a conventional system, when a single operation screen is shared and operated by a plurality of users located in the same space, a plurality of controllers are connected to a single main body device, so the load on the main body device is high. It was. Further, in the conventional system, when a single operation screen is shared and operated by a plurality of users located in separate spaces, the plurality of controllers are connected to the main apparatus existing in the separate spaces. Each main device is connected to a server via a network. In this case, a plurality of main units that can be connected to an external network are required for a plurality of controllers.

  In this embodiment, only one controller is connected to one main device, and the controllers are connected to a server via the Internet. As a result, various operations can be performed on a single operation screen among a plurality of users without increasing the load on one main device. The plurality of users do not need to be located in the same space, and can be located in different spaces.

  FIG. 15 is an exemplary user display screen in which a single operation screen is operated among a plurality of users. In the embodiment according to FIG. 15, user A is wearing a head mount device, and user B is not wearing a head mount device. The user B who is not wearing the head mount device also sees the operation screen of the user A via the display of the second controller, or swipes with the sensor display of the second controller with respect to the user A operating screen. , Tap or pinch operation input, operation input by the acceleration sensor or angular velocity sensor of the second controller can be performed. For example, as shown in the figure, while a user A is playing a racing game, another user B uses the sensor display of the user B's controller to display an obstacle ( Banana) can be placed.

  As described above, the embodiment of the present invention has been described, but the present invention is not limited to the above embodiment. Those skilled in the art will appreciate that various modifications of the embodiments can be made without departing from the spirit and scope of the invention as set forth in the appended claims.

Claims (19)

A system for outputting a generated image to a head mounted display, the system comprising:
An operation determination unit that determines a user operation required according to the contents of the application being executed;
An input detection unit that detects output data corresponding to the requested user operation from output data from a smartphone;
A motion determination unit that determines the motion of the display object based on the detected output data;
A display calculation unit that calculates a display parameter of the display object based on the determined movement of the display object and the detected output data, and generates an image of the display object displayed on the virtual space;
An image generation output unit that generates an image in which an image of a display object generated in an image in a virtual space is arranged and outputs the generated image to a head mounted display connected to the system;
With
The system, wherein the smartphone and the system are connected by a smartphone-side interface program received from a server by the communication unit of the smartphone via the Internet and a system-side interface program of the system.   The system according to claim 1, wherein the output data from the smartphone is output data generated based on an operation input by a user from a sensor display of the smartphone.   The system according to claim 2, wherein the operation input is a swipe operation, a tap operation, or a pinch-in / pinch-out operation.   The system according to claim 1, wherein the output data from the smartphone is output data generated based on voice input by a user from a microphone of the smartphone.   The system according to claim 4, wherein an image of a display object displayed on the virtual space changes based on voice input by the user.   The system according to claim 1, wherein the output data from the smartphone is output data generated based on an operation input by a user from an acceleration sensor or a gyro sensor of the smartphone. The output data from the smartphone includes an image stored in the memory of the smartphone and output data generated based on an operation input by a user from the sensor display of the smartphone,
The system according to claim 1, wherein the motion determination unit determines a display object based on the image and determines a motion of the display object based on the detected output data. The output data from the smartphone includes output data generated based on the position information of the smartphone from the GPS of the smartphone and the operation input by the user from the sensor display of the smartphone,
The display calculation unit calculates a display parameter of the display object based on the position information and the determined operation of the display object, and generates an image of the display object displayed in a virtual space. The system of claim 1. The operation determination unit determines a plurality of user operations required according to the contents of the application being executed,
The input detection unit detects output data corresponding to the plurality of requested user operations from output data from a smartphone,
The motion determination unit determines the motion of the display object based on the first output data among the detected output data, and converts the motion of the display object based on the second output data among the detected output data. The system according to claim 6. A program for causing a processor to execute a method for outputting a generated image to a head-mounted display, the method comprising:
An operation determination step for determining a user operation required according to the contents of the application being executed;
An input detection step of detecting output data corresponding to the requested user operation from output data from a smartphone;
An operation determination step for determining the operation of the display object based on the detected output data;
A display calculation step of calculating a display parameter of the display object based on the determined operation of the display object and the detected output data, and generating an image of the display object displayed on the virtual space;
An image generation output step of generating an image in which an image of a display object generated in an image on a virtual space is arranged and outputting the generated image to a head mounted display connected to the system;
With
A program in which the smartphone and the system are connected by a smartphone-side interface program received from a server via the Internet by a communication unit of the smartphone and a system-side interface program of the system.   The program according to claim 10, wherein the output data from the smartphone is output data generated based on an operation input by a user from a sensor display of the smartphone.   The program according to claim 11, wherein the operation input is a swipe operation, a tap operation, or a pinch-in / pinch-out operation.   The program according to claim 10, wherein the output data from the smartphone is output data generated based on a voice input by a user from a microphone of the smartphone.   The program according to claim 13, wherein an image of a display object displayed on the virtual space changes based on voice input by the user.   The program according to claim 10, wherein the output data from the smartphone is output data generated based on an operation input by a user from an acceleration sensor or a gyro sensor of the smartphone. The output data from the smartphone includes an image stored in the memory of the smartphone and output data generated based on an operation input by a user from the sensor display of the smartphone,
The system according to claim 10, wherein the motion determination step determines a display object based on the image and determines a motion of the display object based on the detected output data. The output data from the smartphone includes output data generated based on the position information of the smartphone from the GPS of the smartphone and the operation input by the user from the sensor display of the smartphone,
The display calculation step calculates a display parameter of the display object based on the position information and the determined operation of the display object, and generates an image of the display object displayed in the virtual space. The system according to claim 10. The operation determining step determines a plurality of user operations required according to the contents of the application being executed,
The input detection step detects output data corresponding to the plurality of requested user operations from output data from a smartphone,
The motion determining step determines the motion of the display object based on the first output data among the detected output data, and converts the motion of the display object based on the second output data among the detected output data. The system of claim 15, wherein: A system control method for outputting a generated image to a head mounted display,
An operation determination step for determining a user operation required according to the contents of the application being executed;
An input detection step of detecting output data corresponding to the requested user operation from output data from a smartphone;
An operation determination step for determining the operation of the display object based on the detected output data;
A display calculation step of calculating a display parameter of the display object based on the determined operation of the display object and the detected output data, and generating an image of the display object displayed on the virtual space;
An image generation output step of generating an image in which an image of a display object generated in an image on a virtual space is arranged and outputting the generated image to a head mounted display connected to the system;
With
The method, wherein the smartphone and the system are connected by a smartphone-side interface program received from a server via the Internet by a communication unit of the smartphone and a system-side interface program of the system.