JP2017134601A - Information processing device, setting screen display method, and setting screen display program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To display a desired setting screen without interrupting processing being executed in a mounted display device.SOLUTION: An information processing device includes an information acquisition unit for acquiring a set of user's action information including line-of-sight information during display of an image about processing being executed in a mounted display device, and a control unit for causing the display device to display a setting screen corresponding to the acquired set of action information with reference to a storage unit which stores a setting screen associated with the set of user's action information including the line-of-sight information.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an information processing apparatus, a setting screen display method, and a setting screen display program.

  Wearing a wearable display device such as a head-mounted display (hereinafter referred to as “HMD”) connected to a game machine, and playing a game by operating a controller or the like while viewing a screen displayed on the HMD Has been done.

  With a normal stationary display connected to a game device, the user's field of view extends beyond the display screen, so it may not be possible to concentrate on the display screen or lack a sense of immersion in the game. . On the other hand, when the HMD is attached, the user does not see any video other than the video displayed on the HMD, so that the feeling of immersion in the video world is enhanced and the entertainment of the game is further enhanced.

  However, it is difficult for the user to see the hand when the HMD is worn, and it is difficult to operate the controller. Therefore, a technique is known that provides an image generation apparatus and an image generation method capable of detecting a user's movement and displaying an image according to the user's movement on an HMD (see, for example, Patent Document 1).

JP 2013-258614 A

  However, in Patent Document 1, it is difficult to display a desired setting screen without disturbing the predetermined process while displaying an image related to a predetermined process such as a game on the HMD.

  Therefore, in one aspect, an object of the present invention is to display a desired setting screen without interfering with a process being executed on a wearable display device.

  In one proposal, an information acquisition unit that acquires a set of user's motion information including line-of-sight information while displaying an image related to the process being executed on the wearable display device, and a user's motion information including the line-of-sight information An information processing apparatus is provided that includes a control unit that displays a setting screen corresponding to the acquired set of operation information on the display device with reference to a storage unit that stores a setting screen associated with the set.

  According to one aspect, it is possible to display a desired setting screen without interfering with the processing being executed on the wearable display device.

The figure which shows an example of the whole structure of the information processing system which concerns on one Embodiment. The figure which shows an example of the hardware constitutions of the information processing apparatus which concerns on one Embodiment. The figure which shows an example of a function structure of the information processing apparatus which concerns on one Embodiment. The figure which shows an example of the data item memorize | stored in the setting screen information DB which concerns on one Embodiment. The flowchart which shows an example of the setting screen display process which concerns on one Embodiment. The figure which shows an example of the setting screen displayed on HMD which concerns on one Embodiment, and a gaze movement. The figure which shows an example of the setting screen displayed on HMD which concerns on one Embodiment, and a gaze movement. The figure which shows an example of the setting screen displayed on HMD which concerns on one Embodiment, and a gaze movement.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, the duplicate description is abbreviate | omitted by attaching | subjecting the same code | symbol.

[Entire configuration of information processing system]
First, an example of the overall configuration of an information processing system according to an embodiment of the present invention will be described with reference to FIG. The information processing system according to the present embodiment includes an HMD (Head Mounted Display) 10 and an information processing apparatus 20. In the information processing system according to the present embodiment, an image of a predetermined game executed by the information processing apparatus 20 is displayed on the HMD 10. The user U wears the HMD 10 connected to the information processing apparatus 20 and plays the game by operating the controller 15 or the like while viewing the screen displayed on the HMD 10. The user U uses the controller 15 to start and end the game, select items, and perform operations during game execution.

  When the user U wears the HMD 10, for example, only the video displayed on the HMD 10 as shown in FIG. 1 enters the field of view of the user U, so that the sense of immersion in the video world is increased and the entertainment of the game is further enhanced. it can.

  The HMD 10 includes a position sensor 11, a posture sensor 12, and an acceleration sensor 13 built in or externally attached. The position sensor 11 measures the position information of the user U using a GPS (Global Positioning System) or the like. The posture sensor 12 and the acceleration sensor 13 measure posture information such as the direction and inclination of the neck of the user U wearing the HMD 10 and the direction and inclination of the head.

  The HMD 10 detects the line of sight of the user U by the mounted camera 14. In the information processing system according to the present embodiment, it is possible to play a game without operating the controller 15 by the sensor value detected by the sensor group. In this case, the controller 15 may not be provided in the information processing system.

  The HMD 10 is connected to the information processing apparatus 20 through an interface for connecting peripheral devices such as wireless communication or USB. The information processing apparatus 20 may be connected to the server 40 via a network 30 such as the Internet or a LAN (Local Area Network). In that case, the server 40 may provide the information processing apparatus 20 with an online application such as a game in which a plurality of users U can participate via the network 30.

  The HMD 10 is an example of a wearable display device. Other examples of the wearable display device include glasses, goggles, and a helmet-type display device that can be worn on the user's head and face, such as FMD (Face Mounted Display).

  The information processing apparatus 20 executes a predetermined process such as playing a game or a movie, and controls display of an image related to the process being executed on the wearable display device. The information processing apparatus 20 is not limited to a game device, but may be, for example, a PC (Personal Computer), a smartphone, a video camera, a digital camera, a PDA (Personal Digital Assistants), a mobile phone, a portable music playback device, a portable video processing device, or the like. It may be. The predetermined process is not limited to the reproduction of a game or a movie, and may be any process for displaying an image.

  Further, in the information processing system according to the present embodiment, the HMD 10 and the information processing apparatus 20 are configured as separate bodies, but the present invention is not limited to this. The information processing apparatus 20 according to the present embodiment may be integrated with the HMD 10. In this case, the information processing apparatus 20 is configured as the HMD 10 that displays an image or the like of the game in progress.

[Hardware configuration of information processing device]
Next, an example of the hardware configuration of the information processing apparatus 20 according to the present embodiment will be described with reference to FIG. The information processing apparatus 20 includes an input device 31, an output device 32, a drive device 33, an auxiliary storage device 34, a main storage device 35, a CPU (Central Processing Unit) 36, and a communication control device 37. Each unit is connected to each other by a system bus B.

  The input device 31 has operation devices such as a keyboard and a mouse, and accepts a game execution instruction, a setting screen startup instruction, and other various inputs in accordance with an operation by the user U of the operation device. The input device 31 may include a controller 15.

  The output device 32 has a display and displays images and setting screens according to the progress of the game. The HMD 10 is an example of the output device 32.

  The auxiliary storage device 34 is realized by, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like. The auxiliary storage device 34 stores programs and data. The stored programs and data include a program for controlling the entire information processing apparatus 20 and a program for executing a predetermined application.

  The main storage device 35 may be configured by a ROM (Read Only Memory) or a RAM (Random Access Memory). The CPU 36 reads a program and data from the auxiliary storage device 34 to the main storage device 35 and executes the game, thereby controlling the progress of the game and realizing other functions installed in the information processing device 20.

  The communication control device 37 performs wireless communication with the HMD 10 and communication with the server 40 and other external devices via the network 30.

  The information processing apparatus 20 can read and write the recording medium 38 via the drive device 33. The recording medium 38 includes a CD (Compact Disk), a DVD (Digital Versatile Disk), an SD memory card, a USB memory (Universal Serial Bus memory), and the like.

[Functional configuration of information processing device]
Next, an example of a functional configuration of the information processing apparatus 20 according to the present embodiment will be described with reference to FIG. The information processing apparatus 20 includes a communication unit 21, a storage unit 22, an information acquisition unit 23, a display control unit 24, a screen switching unit 25, and a control unit 26.

  The communication unit 21 transmits / receives data to / from an external device connected via the network 30. For example, the communication unit 21 transmits and receives various data used when downloading a game program from an external device or executing a setting screen display program for displaying a setting screen.

  The storage unit 22 stores a setting screen information DB (Data Base) 27. The storage unit 22 also stores a setting screen display program 28 and other programs.

  The information acquisition unit 23 acquires a set of motion information of the user U including line-of-sight information while displaying an image related to the game process being executed by the HMD 10. The information acquisition unit 23 acquires the line-of-sight information of the user U based on an image captured by the camera 14 mounted on the HMD 10. Further, the information acquisition unit 23 may acquire the position information of the user U from the sensor value detected by the position sensor 11 mounted on the HMD 10. Further, the information acquisition unit 23 acquires the posture information of the user U such as the direction and inclination of the user U's neck and the direction and inclination of the head from the sensor values detected by the posture sensor 12 and the acceleration sensor 13 mounted on the HMD 10. May be.

  The user operation information acquired by the information acquisition unit 23 is information related to an operation different from the operation that controls the process being executed. As an example, the user U line-of-sight information, the user U posture information, and the user U position information are included. Can be mentioned.

  The information acquisition unit 23 may acquire gesture information of the user U based on an image captured by the camera 14 mounted on the HMD 10. The information acquisition unit 23 may acquire the voice information of the user U using a voice input device such as a microphone. The information acquisition unit 23 may acquire a specific operation of the controller 15 as a specific operation of the user U. The gesture information, the voice information, and the instruction information from the controller are all examples of user operation information related to an operation different from the operation for controlling the game process being executed acquired by the information acquisition unit 23.

  The line-of-sight information included in the set of user motion information is information related to the line of sight directed in a different direction from the line of sight for visually recognizing the game process being executed. A set of user's motion information including line-of-sight information includes a group of a plurality of user's line-of-sight information, a group of user's line-of-sight information and posture information, a group of user's line-of-sight information and gesture information, and user's line-of-sight information and audio information And a set of the user's line-of-sight information and the instruction information from the controller 15.

  FIG. 4 shows an example of data items stored in the setting screen information DB 27. The setting screen information DB 27 includes data items of a set 271 of user operation information and setting screen information 272. The set of motion information 271 stores a set of first motion information 271a and second motion information 271b. However, the set of motion information 271 is not limited to a combination of two motion information, and may be a combination of three or more motion information. In the present embodiment, the description will be continued by taking a pair of two pieces of operation information as an example.

  The setting screen information 272 defines a setting screen to be displayed according to each of the operation information sets 271. For example, a set of user motions has occurred that includes a “motion with a line of sight directed toward the arm object M2” in the first motion information 271a and a “motion with a line of sight directed toward the setting item A” in the second motion information 271b. At this time, the setting screen of “menu selection screen 1” shown in the setting screen information 272 is selected. The selected setting screen is displayed on the HMD 10.

  The control unit 26 controls the entire information processing apparatus 20. For example, the control unit 26 controls the start and end of execution of the game process, performs a process for login authentication for each user, and executes a game corresponding to the user and a display control process when the authentication is successful. The control unit 26 controls to temporarily stop the game process being executed while executing a predetermined setting process from the display of the setting screen.

  In addition, the control unit 26 refers to the setting screen information DB 27 that stores a setting screen associated with a set of user motion information including line-of-sight information, and displays the setting screen corresponding to the acquired set of motion information. Display on the device. The display control unit 24 displays the selection screen selected by the control unit 26 on the HMD 10.

  The screen switching unit 25 switches and displays a screen for displaying an image of a predetermined scene of the game and a setting screen. The screen switching unit 25 may display the setting screen so as to overlap a screen that displays an image related to the game that is paused. That is, the screen switching includes a case where the game screen and the setting screen are completely switched, and a case where the setting screen is superimposed on the game screen.

  Note that FIG. 3 is a block diagram focusing on functions, and each unit indicated by these functional blocks can be realized by hardware alone, software alone, or a combination of hardware and software.

  With this configuration, in the information processing apparatus 20 according to the present embodiment, the CPU 36 executes a setting screen display process using the setting screen display program and various data stored in the main storage device 35 or the auxiliary storage device 34. The information stored in the setting screen information DB 27 can be stored in the main storage device 35, the auxiliary storage device 34, the server 40 connected to the information processing device 20 via the network 30, or other servers on the cloud. .

[Setting screen display processing]
Next, the setting screen display process according to the present embodiment will be described with reference to FIG. For example, when the user U operates the controller 15 to instruct the start of the game, the information processing apparatus 20 starts the processing by inputting the instruction information.

  When this process is started, in step S1, the information acquisition unit 23 receives a game execution instruction. Next, in step S1, the display control unit 24 causes the HMD 10 to display an image corresponding to the progress of the game.

  Next, in step S3, the control unit 26 determines whether a specific first line-of-sight movement has been detected. The control unit 26 has a specific line-of-sight movement including the first line-of-sight movement among the user movement information acquired by the information acquisition unit 23 is a line of sight directed in a direction different from the line of sight for viewing the game process being executed. . As an example of the specific line-of-sight movement, in the present embodiment, the line of sight of the user U directed to a specific object displayed on the HMD 10 will be described as an example. As an example of the specific object, there is an object M2 of the arm of the user U, which is VR (Virtual Reality) content shown in FIG. The arm M2 of the user U's arm is always displayed on the lower side of the display of the output device 32. However, the specific object is not limited to this, and may be, for example, the menu box M1 displayed at the lower right in FIG. The menu box M1 may function as a box that can transition to a comprehensive setting screen, and the arm object M2 of the user U may function as a box that can transition to an individual setting screen. The menu box M1 may be displayed or may not be displayed. When the menu box M1 is not displayed on the screen, the position of the menu box M1 on the screen may be indicated by a help operation. For example, as shown in FIG. 6A, when the position of the line of sight of the user U is represented by the line of sight pointer P, the line of sight pointer P changes from the current position to the object M2 of the user U's arm, When the gaze is focused on the object M2 (1-1), the control unit 26 determines that a specific first gaze movement has been detected.

  Returning to FIG. 5, when the control unit 26 determines in step S <b> 3 that the specific first line-of-sight movement is not detected, the process returns to step S <b> 2 until the ongoing game is continued and the specific first line-of-sight movement is detected. As the game progresses, the processes of steps S2 and S3 are repeated.

  On the other hand, if it is determined in step S3 that the control unit 26 has detected the specific first line-of-sight movement, the process proceeds to step S4, and the control unit 26 performs control so as to temporarily stop the screen display during the game. The display control unit 24 displays the temporarily stopped image on the HMD 10.

  Next, in step S5, the control unit 26 pops up a specific menu at the position of the line-of-sight pointer P. In response to this, a specific menu in which setting items A to D are arranged radially from the line-of-sight pointer P is popped up as shown in FIG.

  Returning to FIG. 5, in step S <b> 6, the control unit 26 determines whether a specific second eye movement is detected. When determining that the specific second line-of-sight movement is not detected, the control unit 26 determines whether or not the predetermined time has elapsed in Step S7, and repeats the process of Step S6 until the predetermined time elapses. In step S6, when the predetermined time in step S7 has elapsed without detecting the specific second line-of-sight movement, the process returns to step S2, and the control unit 26 advances the paused game and repeats the processes in and after step S2. .

  In step S6, when it determines with the control part 26 having detected the specific 2nd eyes | visual_axis movement, it progresses to step S8 and the screen switching part 25 switches a display screen to a menu selection screen. For example, in FIG. 6B, the user U turns his / her line of sight upward from the line-of-sight pointer P and keeps his / her eyes on the line of sight for a predetermined time (1-2). Thereby, the control unit 26 determines that a specific second line-of-sight movement has been detected.

  Based on the setting screen information DB 27 in FIG. 4, the control unit 26 searches for a set of the first line-of-sight movement and the second line-of-sight movement. The control unit 26, from the motion information set 271, the first line-of-sight movement that directs the line of sight toward the arm object M2 in FIG. A pair with eye movement is detected. The control unit 26 sets the “menu selection screen 1” of the setting screen information 272 according to the set of the first line-of-sight movement that directs the line of sight toward the arm object M2 and the second line-of-sight movement that directs the line of sight toward the setting item A. Select a screen. The display control unit 24 switches to and displays the selected “menu selection screen 1”. As a result, the “menu selection screen 1” is switched and displayed on the HMD 10. FIG. 6C shows an example of a setting screen displayed by switching.

  Returning to FIG. 5, in step S <b> 9, the control unit 26 further determines whether a specific eye movement is detected. When it is determined that the control unit 26 does not detect the specific line-of-sight movement, in step S10, the control unit 26 determines whether a predetermined time has elapsed, and repeats the process of step S9 until the predetermined time has elapsed. . In step S9, when the predetermined time of step S10 has elapsed without detecting a specific line-of-sight movement, the process returns to step S2, and the control unit 26 advances the game as it is, and repeats the processing after step S2.

  If it is determined in step S9 that the specific line-of-sight movement has been detected, the control unit 26 proceeds to step S11, and the screen switching unit 25 executes the menu operation selected by the detected line-of-sight movement, and then the step Returning to S2, the game in progress is resumed.

  For example, in FIG. 6C, the line of sight of the user U is directed from the line-of-sight pointer P to the “Yes” option in the area on the left side of the page, and a line-of-sight operation is performed in which the user gazes for a predetermined time (1 -3). Accordingly, the control unit 26 determines that the option “Yes” is selected for the setting item “Are you sure?”, And executes the action “00”. For example, an operation of acquiring equipment related to the game being executed can be considered as the operation of “OO”. When the “acquire” option is selected, the control unit 26 performs an operation of giving the equipment A to the user U in the game.

  As described above, according to the information processing system according to the present embodiment, a setting screen for a specific menu or the like can be obtained by designating a specific object in advance and detecting a set of user motion information for the specific object. Can be displayed. In this case, the specific object may be VR content.

  According to the information processing system according to the present embodiment, during the progress of the game, based on the operation including the user's line of sight, the area is set to the area where the user is gazing, depending on where the line of sight is directed and where Changes to items can be made each time. Further, the game progress is paused during the setting, but the game can be advanced immediately after the setting. For this reason, according to the information processing system which concerns on this embodiment, a desired setting screen can be displayed, without disturbing progress of a game, during progress of a game. Note that even if the set of user's line-of-sight information is the same, the items (options) set in the watched area may be changed depending on which game screen is displayed.

(A set of user actions)
Note that the set of user motion information is not limited to the set of user eye movements shown in FIG. The set of user motion information may be a set of line-of-sight information and user posture information, or may be a set of line-of-sight information and user gesture information. The set of user operation information may be a set of line-of-sight information and user's voice information, or may be a set of line-of-sight information and instruction information from the controller 15. One of the sets of user motion information is line-of-sight information, and the line-of-sight information may be detected in step S3 in FIG. 5 or step S6. In step S where the line-of-sight information is not detected, the user's motion information paired with the line-of-sight information is detected.

  A case where a set of “neck movement” which is an example of posture information and a line of sight movement is detected, or a case where a plurality of lines of sight movement are detected will be described with reference to the example of FIG. As shown in FIG. 7A, the user U raises his / her line of sight (or neck) once in the sky direction (upward) (2-1), and then looks down to the lower right of the ground shown in FIG. 7B. Or drop the line of sight to the menu box M1 displayed in the lower right (2-2). When a set of actions including the user's line-of-sight movements is detected, the screen displayed on the HMD 10 transitions to a setting screen that can confirm whether or not the game is ended as shown in FIG. The user U can select whether or not to end the game by gazing for a predetermined time toward the region of “ending” or “not ending”. In particular, if the user's action set is given a meaning (a set of gaze movements that give up and look up at the sky and discourage) as shown in FIG. Setting screens linked to the progress state can be set, and the user's operation can be made easier.

  The set of user motion information may be a user motion combining a gesture and a line of sight, or a user motion combining a voice and a line of sight. For example, the information processing apparatus 20 detects that the user U has said a specific keyword such as “setting” using the voice recognition function, and when there is a specific eye movement, a setting corresponding to the set A screen may be displayed. In this case, the accuracy of speech recognition is not limited as long as noise and the like can be distinguished from the keyword.

  Further, when a user action that combines the operation of the controller 15 and the line of sight is detected, a predetermined setting screen may be displayed. For example, a predetermined setting screen may be launched when a combination of pressing of the button of the controller 15 and a line-of-sight operation that are at hand but are not normally used is detected.

  In addition, after switching from the screen displaying the scene where the game is in progress to the setting screen to display the menu, selecting the necessary setting items from the setting screen using the controller 15 realizes simple menu access. Not. Therefore, it is preferable that the user operation for selecting a necessary setting item from the setting screen after starting the setting screen is completed using the line-of-sight operation without using the controller 15.

  As described above, according to the information processing apparatus 20 according to the present embodiment, the simple setting is performed by a plurality of user operations including the line of sight while the game is in progress. Thereby, a desired setting screen can be displayed without hindering the game process being executed in the HMD 10.

  Although the information processing apparatus, the setting screen display method, and the setting screen display program have been described in the above embodiment, the information processing apparatus, the setting screen display method, and the setting screen display program according to the present invention are limited to the above embodiment. Rather, various modifications and improvements can be made within the scope of the present invention. In addition, when there are a plurality of the above-described embodiments and modifications, they can be combined within a consistent range.

  For example, a specific example of display of the selected setting screen is not limited to the case where the screen during game execution in FIG. 6B is switched to the setting screen shown in FIG. For example, when it is not necessary to switch the screen as in the case of displaying a setting screen for matters directly related to the progress of the current game, only the setting items A to D may be displayed. In this case, it is possible to move the line of sight from the line-of-sight pointer P in FIG. 6B in any one of the setting items A to D and execute the operation set in the setting item. In this case, if “Yes” is determined in step S6 of FIG. 5, step S11 is executed.

  Further, in the case of setting the game end or the like, as shown in FIG. 8B, a semi-transparent display is displayed on the screen in progress of the game without switching from the screen of FIG. 7B to the screen of FIG. 7C. For example, the selected setting screen may be overlapped and displayed. Even in this case, the user can watch the area obtained by dividing the entire screen by the number of options (3-2), and select the option for that area ("Proceed to the next stage" in FIG. 8B). Can do. Thereby, the setting which the user U desires can be performed smoothly and a game can be restarted, without switching a screen to a setting screen. The area obtained by dividing the entire screen by the number of options is not limited to two, and may be any number of three or more.

  The setting of the setting screen information 272 for the user operation information set 271 stored in the setting screen information DB 27 is performed by assigning a specific function to an area set in the setting screen information 272 by the user U in advance. It can be carried out. However, it is possible that the user U cannot change the information set in the setting screen information DB 27.

10 HMD
DESCRIPTION OF SYMBOLS 11 Position sensor 12 Attitude sensor 13 Acceleration sensor 14 Camera 15 Controller 20 Information processing apparatus 21 Communication part 22 Memory | storage part 23 Information acquisition part 24 Display control part 25 Screen switching part 26 Control part 27 Setting screen information DB
28 Setting Screen Display Program 30 Network 40 Server 31 Input Device 32 Output Device 33 Drive Device 34 Auxiliary Storage Device 35 Main Storage Device 36 CPU
37 Communication control device 38 Recording medium U User

Claims (8)

An information acquisition unit that acquires a set of user operation information including line-of-sight information while displaying an image related to a process being executed on the wearable display device;
A control unit that displays a setting screen corresponding to the acquired set of motion information on the display device with reference to a storage unit that stores a setting screen associated with the set of user motion information including line-of-sight information;
An information processing apparatus. The control unit pauses the process being executed while executing a predetermined setting process from the display of the setting screen.
The information processing apparatus according to claim 1. The set of user motion information including the line-of-sight information includes a group of a plurality of user's line-of-sight information, a group of user's line-of-sight information and posture information, a group of user's line-of-sight information and gesture information, and user's line-of-sight information and audio. A set of information, and a set of user's line-of-sight information and instruction information from a controller that controls the processing being executed,
The information processing apparatus according to claim 1 or 2. The operation information of the user is information related to an operation different from an operation for controlling the process being executed.
The information processing apparatus according to any one of claims 1 to 3. The line-of-sight information included in the set of user's motion information is information regarding the line of sight directed in a different direction from the line of sight for visually recognizing the process being executed.
The information processing apparatus according to claim 4. The control unit causes a setting screen to be displayed so as to overlap with a screen that displays an image related to the process being executed.
The information processing apparatus according to any one of claims 1 to 5. While displaying an image related to the process being executed on the wearable display device, a set of user operation information including line-of-sight information is acquired,
With reference to a storage unit that stores a setting screen associated with a set of user motion information including line-of-sight information, a setting screen corresponding to the acquired set of motion information is displayed on the display device.
A setting screen display method in which processing is executed by a computer. While displaying an image related to the process being executed on the wearable display device, a set of user operation information including line-of-sight information is acquired,
With reference to a storage unit that stores a setting screen associated with a set of user motion information including line-of-sight information, a setting screen corresponding to the acquired set of motion information is displayed on the display device.
A setting screen display program for causing a computer to execute processing.

Images