JP7211778B2 - Application processing system, application processing method, and application processing program - Google Patents

Description

The disclosure herein relates to an application processing system, an application processing method, and an application processing program that provide interaction with a virtual character. More specifically, the disclosure herein relates to an application processing system, an application processing method, and an application processing program that provide interaction with a virtual character according to information detected by a head-mounted display.

A simulation game that allows interaction with a virtual character is known as an application that provides interaction with a virtual character. Such a simulation game is disclosed, for example, in Japanese Unexamined Patent Application Publication No. 2012-055644 (Patent Document 1).

In games that provide interaction with such virtual characters, a head-mounted display (hereinafter sometimes referred to as "HMD") is sometimes used in order to enhance the sense of immersion in the game. For example, a VR game is known in which a virtual space is displayed on an HMD, and a virtual character appearing in this virtual space can be interacted with using the HMD. As this type of VR game, there is "Summer Lesson" by BANDAI NAMCO Entertainment Inc. (see Non-Patent Document 1).

The HMD includes a tracking sensor for detecting the position and orientation of the user's head, as described in Japanese Patent Laying-Open No. 2017-058969 (Patent Document 2). In a VR game played using an HMD, a viewpoint in the virtual space is determined based on the detection information detected by the tracking sensor, and an image of the virtual space seen from the thus determined viewpoint is displayed on the HMD. be done.

In the above "summer lesson", an action for the virtual character is specified according to the detection information detected by the tracking sensor of the HMD, and by making the virtual character react to the specified action, the user and the virtual character Interaction between is realized. For example, when a user wearing the HMD nods in response to a question from a virtual character displayed on the HMD, the movement of the user's head corresponding to the nod is detected, and the detected head movement is detected. Based on this, an action from the user to the virtual character in the virtual space is specified. For example, when a movement of the user's head corresponding to a nod is detected, an action of giving an affirmative response to the question from the virtual character is specified as the action from the user to the virtual character in the virtual space. be. Further, when the HMD detects a motion of the user's head corresponding to the motion of shaking the head, the action to the virtual character is to give a negative response to the question from the virtual character in the virtual space. actions are identified. The virtual character's reaction to the user's action specified in this way is determined, and an image of the virtual character performing the determined reaction is displayed on the HMD.

The user's action in the virtual space, which is specified based on the HMD detection information, is associated with the user's head movement such as nodding or shaking his head, and the user's line of sight movement such as gazing at a specific part of the virtual character. is limited to In this way, the variation of actions from the user to the virtual character specified by the detection signal of the HMD is limited.

If an action associated with a part of the user's body other than the head (for example, a hand) can be specified as the action of the user in the virtual space, variations in interaction between the virtual character and the user will be enriched. be. Actions associated with parts of the user's body other than the head have conventionally been identified based on input from an operation controller separate from the HMD. A typical conventional manipulation controller is configured to be held by a user. For example, Japanese Patent No. 6189497 discloses that the position of a user's virtual hand is specified in a virtual space based on the operation of an operation controller held by the user, and the virtual space is operated by the virtual hand. It is disclosed to perform an action of touching an object within.

JP 2012-055644 A JP 2017-058969 A Patent No. 6189497

"Summer Lesson", [online], December 12, 2017, Internet (URL: HYPERLINK "http://summer-lesson.bn-ent.net/" http://summer-lesson.bn-ent .net/)

As described above, conventionally, in order to identify an action that is not associated with the movement of the user's head (for example, the action of touching with a hand in the virtual space) among the user's actions in the virtual space, of operation controllers are used. The use of such manipulation controllers may detract from the immersion in the application. Moreover, when the HMD is worn, the field of view in the real space is blocked, so there is a problem that it is difficult to operate the operation controller.

There is a need for an application processing system capable of performing actions on a virtual character that are not associated with the movement of the user's head in virtual space without using an operation controller separate from the HMD.

It is an object of the present disclosure to provide technical improvements that solve or alleviate at least some of the problems in the prior art discussed above.

One specific object of the present disclosure is to provide technical improvements in application processing systems that can perform actions on virtual characters that are not associated with user head movements.

One of the more specific objects of the present disclosure is to provide an application processing system that can perform actions unrelated to the movement of the user's head on a virtual character based on information detected by a head-mounted display. to provide.

One of the more specific purposes of the present disclosure is to enrich the variations of actions that are not associated with the movement of the user's head performed on a virtual character.

These and other objects of the disclosure will become apparent upon reference to the specification as a whole.

An application processing system according to an aspect of the present invention includes one or more computer processors, and outputs an image of a virtual space including a virtual character to a head-mounted display worn on the user's head to provide the user with the virtual space. Provide character interaction. In one aspect, the one or more computer processors execute computer readable instructions to perform on the virtual character based on one or more detected information detected by the head-mounted display. identifying a first action that is an action and is associated with movement of the user's head in the virtual space, and selecting a specific object included in the virtual space; based on the detection information, a second action that is an action performed on the virtual character and is not associated with the movement of the user's head in the virtual space.

In the application processing system according to one aspect of the present invention, the second action is an action associated with hand movement of the user in the virtual space.

In the application processing system according to one aspect of the present invention, the second action is an action associated with use of tools by the user in the virtual space.

In the application processing system according to one aspect of the present invention, one or more character gaze positions are set on the virtual character according to the selection of the specific object, and one or more character gaze positions set on the virtual character are set. The second action is identified responsive to at least one of the locations being gazed at.

In the application processing system according to one aspect of the present invention, the one or more character gaze positions include a first character gaze position and a second character gaze position, and the second action includes a first gaze position action and a second gaze position action. A position action is included, a first gaze position action is specified as the second action by the gaze at the first character gaze position, and a second gaze position action is specified as the second action by the gaze at the second character gaze position identified.

In the application processing system according to one aspect of the present invention, the specific object is associated with a first menu item and a second menu item, and the second action is the first menu item associated with the first menu item. and a second menu item action associated with the second menu item, wherein at least one of the one or more character gaze positions is gazed after the first menu item is gazed at, whereby the A first menu item action is identified as the second action, and at least one of the one or more character gaze positions is gazed after the second menu item is gazed, whereby the second menu item action is determined as the Identified as the second action.

In the application processing system according to an aspect of the present invention, the one or more computer processors specify a first main display area in the virtual space in the first scene, and the specified object is the first main display area in the first scene. 1 is located outside the main display area.

In the application processing system according to an aspect of the present invention, the one or more computer processors specify a second main display area in the virtual space in the second scene, and the specified object is the second main display area in the second scene. 2 Located outside the main display area.

In the application processing system according to one aspect of the present invention, the one or more computer processors are configured to perform the virtual space so that the virtual space includes the virtual character that reacts according to the first action or the second action. configure.

An application processing method according to one aspect includes outputting an image of a virtual space including a virtual character to a head-mounted display worn on a user's head by one or more computer processors executing computer-readable instructions. provides the user with interaction with the virtual character. The application processing method is an action performed on the virtual character and associated with the movement of the user's head in the virtual space based on one or more pieces of detection information detected by the head-mounted display. an action to be performed on the virtual character based on the one or more pieces of detection information in response to selection of a specific object contained in the virtual space; and identifying a second action that is not associated with movement of the user's head in the virtual space.

An application processing program according to an aspect of the present invention outputs an image of a virtual space including a virtual character to a head-mounted display worn on a user's head by being executed by one or more computer processors. Providing a user with interaction with said virtual character. The application processing program instructs the one or more computer processors, based on one or more pieces of detection information detected by the head-mounted display, to perform an action on the virtual character in the virtual space. identifying a first action associated with movement of the user's head; and in response to selection of a specific object included in the virtual space, based on the one or more pieces of detection information, and identifying a second action to be performed on the virtual character that is not associated with movement of the user's head in the virtual space.

Embodiments disclosed herein provide technical improvements in application processing systems that can perform actions on virtual characters that are not associated with user head movements.

1 is a block diagram illustrating an application processing system according to one embodiment; FIG. FIG. 2 is a schematic diagram showing a head-mounted display that can be used in the application processing system of FIG. 1; FIG. 4 is a diagram schematically showing a head-mounted display worn by a user; 1 is a diagram schematically showing a virtual space in an application processing system according to one embodiment; FIG. In Figure 4a, the user is in a standing position in the virtual space. 1 is a diagram schematically showing a virtual space in an application processing system according to one embodiment; FIG. In FIG. 4b, the user is in a supine position in virtual space. FIG. 4 is a diagram illustrating mode transition of a game processed by the application processing system according to one embodiment; It is a flow figure showing a flow of processing in chat mode in one embodiment. FIG. 10 is a diagram showing an example of a display image in chat mode in one embodiment; FIG. 4 is a flow diagram showing the flow of processing in VR mode in one embodiment; FIG. 4 is a diagram showing an example of a display image in VR mode in one embodiment; FIG. 10 is a flow diagram showing the flow of processing for identifying a second action in one embodiment; FIG. 4 is a diagram showing an example of a display image in VR mode in one embodiment; The display image of FIG. 11a includes an image showing a menu item icon. FIG. 4 is a diagram showing an example of a display image in VR mode in one embodiment; A plurality of character gaze positions are set for the virtual character in FIG. 11b. FIG. 4 is a diagram showing an example of a display image in VR mode in one embodiment; The display image of FIG. 11c includes an image showing the user's second action in the virtual space. FIG. 4 is a diagram showing an example of a display image in VR mode in one embodiment; The display image of FIG. 11d includes an image showing the user's second action in the virtual space. FIG. 4 is a block diagram illustrating an application processing system according to another embodiment of the invention;

Various embodiments of the present invention will now be described with appropriate reference to the drawings. Identical or similar components are provided with the same reference numerals in several drawings.

An application processing system according to one embodiment will be described with reference to FIGS. 1 to 3. FIG. FIG. 1 is a block diagram showing an application processing system 1 according to one embodiment, and FIG. 2 is a schematic diagram of a head-mounted display 10 (hereinafter referred to as "HMD 10") used in the application processing system 1. FIG. 3 is a diagram schematically showing the HMD 10 worn by the user 5. FIG.

The application processing system 1 according to one embodiment provides various applications by executing application processing programs. An example of this application is video games. A user can use the application processing system 1 to play various games using game media. A game medium is electronic data used in a game realized by the application processing system 1, and is acquired, owned, used, managed, exchanged, combined, enhanced, sold, discarded, gifted, or otherwise used in the game by a user. can be used in the manner of Game media include, for example, cards, items, virtual currency, tickets, characters, avatars, level information, status information, parameter information (physical strength, attack power, etc.), ability information skills, abilities, spells, jobs, and the like. Includes various electronic data that can be used in games other than The game media that can be used in the application processing system 1 and the mode of use thereof are not limited to those specified in this specification. The application processing system 1 can realize, for example, a game in which a virtual character and a user interact in a virtual space. A game realized by the application processing system 1 has a first mode and a second mode. An example of the first mode is chat mode and an example of the second mode is VR mode. The first mode and second mode will be described later.

The application processing system 1 may be configured to process applications other than games. The application processing system 1 includes, for example, simulations in driving and other fields, training applications in medical and other fields, applications for experiencing products and services, and applications for watching movies and various other forms of entertainment. , and may be configured to provide various applications other than these.

The application processing system 1 has an HMD 10 and a server 50 . The HMD 10 and the server 50 are connected via a network 40 so as to be able to communicate with each other.

As shown in FIG. 2, the HMD 10 includes an attachment 11 to be worn on the head of a person, and an information processing device 20 attached to the attachment 11 .

The wearing tool 11 has a goggle-shaped housing 11a formed with an opening 11d, a first belt 11b and a second belt 11c attached to the housing 11a, and a lid 11e. The lid 11e is attached to the housing 11a so as to be openable and closable. An opening 11d is formed in front of the housing 11a. In FIG. 2, the lid 11e is open. When the HME 10 is in use, the lid 11e is closed to block the opening 11d.

The information processing device 20 is detachably provided inside the lid 11 e of the HMD 10 . The information processing device 20 has a display 24 . The information processing device 20 is attached to the lid 11e such that the display 24 faces the inside of the housing 11a when the lid 11e is closed. The information processing device 20 is attached to the wearing tool 11 when the HMD 10 is used. The information processing device 20 is removed from the wearing tool 11 when the HMD 10 is not used.

In the illustrated embodiment, the information processing device 20 is a smart phone. The information processing device 20 may be a mobile phone, a tablet terminal, a personal computer, an electronic book reader, a wearable computer, a game console, or various information processing devices other than a smart phone. The information processing device 20 removed from the HMD 10 is used for its original purpose.

The information processing device 20 includes the display 24 as described above. In the illustrated embodiment, when the information processing device 20 is attached to the wearing tool 11 , the display 24 functions as a device for displaying images on the HMD 10 . Therefore, when playing a game using the HMD 10, the display 24 displays the virtual space of the game, the virtual characters, and other images related to the game.

The shape of the wearing tool 11 is not limited to the illustrated goggle shape. The wearer 11 may include a structure of any shape that follows the movement of the user's head to which it is worn, and which allows the display 24 to be placed in front of the user's eyes when worn. The wearing tool 11 may have, for example, a shape that imitates glasses, a shape that imitates a hat, or a shape that imitates a helmet. In order to improve the user's sense of immersion, the HMD 10 is desirably configured so that the display 24 covers the user's eyes when the wearer 11 is worn on the user's head.

When in use, the HMD 10 is attached to the head of the user 5 via the attachment 11, as shown in FIG. An information processing device 20 is attached to the wearing tool 11 worn on the head of the user 5 .

The information processing apparatus 20 will be further described with reference to FIG. 1 again. As illustrated, in one embodiment, the information processing device 20 includes a computer processor 21 , a memory 22 , a communication I/F 23 , a display 24 , a sensor unit 25 , a sound collector 26 and a storage 27 .

The computer processor 21 is an arithmetic device that loads various programs that implement an operating system and game logic from the storage 27 or other storage into the main memory 22 and executes instructions contained in the loaded programs. The computer processor 21 is, for example, a CPU, MPU, DSP, GPU, or any other arithmetic device. The processor 21 may be realized by integrated circuits such as ASIC, PLD, FPGA, and MCU. Although computer processor 21 is illustrated as a single component in FIG. 1, computer processor 21 may be a collection of a plurality of physically separate computer processors. In this specification, a program or instructions included in the program described as being executed by the computer processor 21 may be executed by a single computer processor, or may be executed by a plurality of computer processors in a distributed manner. good too. Also, the program executed by the computer processor 21 or instructions included in the program may be distributed and executed by a plurality of virtual computer processors.

The memory 22 is used to store instructions executed by the computer processor 21 and various other data. At least part of the application processing program according to one embodiment is loaded into the memory 22 from time to time as the game progresses. The memory 22 is, for example, a main storage device (main memory) that can be accessed at high speed by the computer processor 21 . The memory 22 is, for example, a RAM such as a DRAM or an SRAM.

The communication I/F 23 is implemented as hardware, firmware, communication software such as a TCP/IP driver or PPP driver, or a combination thereof. The information processing device 20 can transmit and receive data to and from other devices via the communication I/F 23 .

The display 24 has a display panel 24a and a touch panel 24b. For example, the touch panel 24b is laminated on the upper surface or the lower surface of the display panel 24a. The display panel 24a is a liquid crystal panel, an organic EL panel, an inorganic EL panel, or any other display panel capable of displaying an image. The touch panel 24b is configured to detect a user's touch operation. The touch panel 24b can detect various touch operations such as a user's tap, double tap, and drag.

The sensor unit 25 comprises one or more sensors. Sensor unit 25 includes, for example, at least one of a gyro sensor, an acceleration sensor, and a geomagnetic sensor. The sensor unit 25 may comprise an eye tracking sensor that directly detects the movement of the user's eyeballs. An eye tracking sensor is, for example, a line-of-sight sensor that receives near-infrared light into the iris and detects the reflected light. The position and orientation of the HMD 10 worn on the head of the user 5 are specified based on the detection information detected by the sensor unit 25 . The position and orientation of the HMD 10 are calculated as, for example, the position in a three-dimensional orthogonal coordinate system and the angle around each axis. This three-dimensional orthogonal coordinate system is, for example, an orthogonal coordinate system composed of a roll axis along the front-rear direction, a yaw axis along the vertical direction, and a pitch axis along the left-right direction. At least some of the various sensors that make up the sensor unit 25 may be provided on the mounting tool 11 and/or a member different from the information processing device 20 other than the mounting tool 11 .

The sound collector 26 is configured to be able to collect sound. The sound collecting device 26 is, for example, a microphone. The voice of the user 5 is detected based on the voice information collected by the sound collector 26 .

Storage 27 is an external storage device accessed by computer processor 21 . The storage 27 is, for example, a magnetic disk, an optical disk, a semiconductor memory, or various other storage devices capable of storing data. The storage 27 stores various programs such as an application processing program. The storage 27 also stores various data used in the game. At least part of the programs and various data that can be stored in the storage 27 may be stored in a storage that is physically separate from the information processing device 20 .

In the illustrated embodiment, the storage 27 stores image data 28a, chat data 28b, scenario data 28c, event data 28d, game progress data 28e, and various other data necessary for the progress of the game. be.

The image data 28a includes data for drawing the background of the virtual space in which the game is played, data for drawing virtual characters, and data for drawing objects other than the virtual characters used in the game. The image data 28a may include position information of the object in the virtual space. The above data may be stored in the image data 28a in association with each scenario, each event, or each scene.

The chat data 28b includes data for drawing the icon of the virtual character, data for drawing the icon of the user 5, data for specifying a plurality of messages of the virtual character, and each of the plurality of messages of the virtual character. and data used in other chat modes. A plurality of messages of the virtual character may be defined in a tree structure having nodes corresponding to each message and connecting the nodes with arcs. In this tree structure, for example, a plurality of arcs extend from the start message, which is the root node of the highest level, and each arc is connected to a lower node. Arcs also extend from this lower node, and the arcs are connected to further lower nodes. Each of the lower nodes corresponds to a candidate message of the virtual character that can be displayed after the starting message. Also, the chat data 28b may include a mode transition condition, which is a condition for starting transition from the chat mode to the VR mode. Mode transition conditions include, for example, that the elapsed time since the game was started in chat mode is equal to or longer than a predetermined time, that the chat has progressed to the terminal node in the message of the tree structure, and other conditions. be Chat data 28b may include data indicative of messages to be displayed in the resumed chat mode after incomplete selection of transition objects.

Scenario data 28c includes data defining the scenario experienced by user 5 in the game. If there are multiple scenarios experienced by the user 5 in the game, the scenario data 28c may define scenario data for each of the multiple scenarios. Scenario data 28c may be defined as a set of multiple events. A plurality of events included in the scenario data 28c may be defined in a tree structure having nodes corresponding to each event and connecting the nodes with arcs. In this tree structure, for example, a plurality of arcs extend from the initial event, which is the highest root node, and each arc is connected to lower nodes. Arcs also extend from this lower node, and the arcs are connected to further lower nodes. Each lower node in a scenario corresponds to an event candidate that can occur after the opening event in the scenario. The above data may be stored in the scenario data 28c in association with each scenario.

The event data 28d is data for defining each of the events forming the scenario data 28c. The event data 28d may be defined as a set of data for specifying multiple actions of the virtual character in the game. A plurality of actions of the virtual character may be defined in a tree structure having nodes corresponding to each action and connecting the nodes with arcs. In this tree structure, for example, a plurality of arcs extend from the top root node, the beginning action, and each arc is connected to a lower node. Arcs also extend from this lower node, and the arcs are connected to further lower nodes. Each subordinate node in an event corresponds to a candidate action that can occur after the initial action in the event. The event data 28d includes, for each node, moving image data corresponding to the action corresponding to the node, data regarding actions that the user can take with respect to the virtual character, and data regarding other actions of the virtual character. can be done.

The game progress data 28e includes data used for managing the progress of the game. The game progress data 28e may be updated as the game progresses. The game progress data 28e may include, for example, data relating to the user's favorability of the virtual character, data relating to points earned by the user in the game, and various other data that may change according to the progress of the game.

The components and functions of the information processing device 20 shown in FIG. 1 are exemplary. The information processing device 20 applicable to the present invention may include various components not shown. For example, the information processing device 20 may include a speaker that outputs game sound effects and virtual character voices.

The components and functions of the information processing device 20 shown in FIG. 1 are exemplary. The information processing device 20 applicable to the present invention may include various components not shown. For example, the information processing device 20 may include a speaker that outputs game sound effects and virtual character voices.

Next, functions of the HMD 10 will be described. In the illustrated embodiment, various functions of the HMD 10 are implemented by executing computer readable instructions in the computer processor 21 of the information processing device 20 . The instructions executed by computer processor 21 include instructions contained in an application processing program according to one embodiment.

A game having a first mode and a second mode different from the first mode is realized in the application processing system 1 by executing the application processing program according to one embodiment by the computer processor 21 . A game implemented in the application processing system 1 may have modes other than the first mode and the second mode.

In the first mode of the game realized in the application processing system 1, the information processing device 20 performs the A game-related process is performed according to the detected first detection information. The first detection information includes information about the touch operation of the user 5 detected by the touch panel 24b of the information processing device 20, information about the user's voice detected by the sound collector 26, and whether the HMD 10 is worn by the user 5. It may include detection information other than the above that can be detected by the information processing device 20 when it is not present. In the first mode, the user 5 can perform game-related operations using the information processing device 20 removed from the wearing tool 11 . In the first mode, it is assumed that the game progresses when the HMD 10 is not worn by the user 5, so in one embodiment, the display 24 displays an image other than the stereoscopic image.

In the second mode of the game implemented in the application processing system 1, the game proceeds using the second detection information detected by the HMD 10 worn on the user's 5 head. This second detection information is, for example, detection information detected by the sensor unit 25 . Based on this second detection information, head tracking information for specifying the position and orientation of the head of the user 5 is calculated. The processing for progressing the game in the second mode is executed based on, for example, head tracking information calculated based on the second detection information. The processing for progressing the game in the second mode may be executed using information other than the head tracking information in addition to the head tracking information. In preparation for playing the game in the second mode, the user 5 attaches the information processing device 20 to the wearing tool 11 and wears the wearing tool 11 to which the information processing device 20 is attached on the head. Thus, in the second mode, it is assumed that the game progresses while the HMD 10 is worn by the user 5. Therefore, in one embodiment, in the second mode, the user 5 stereoscopically A stereoscopic image is displayed on the display 24 . A stereoscopic image is displayed on the display 24 by, for example, a parallax barrier method. In the parallax barrier method, a left-eye image and a right-eye image are displayed on the display 24 . Thus, a stereoscopic image is a set of images including a left-eye image and a right-eye image that are configured to be stereoscopically viewed using the parallax of the left and right eyes by being displayed on a display.

The first mode is, for example, chat mode. Chat mode is an example of the first mode. The chat mode implements the ability for users to chat with virtual characters via text messages. In the first mode, the user can experience interaction with the virtual character by chatting with the virtual character. Specific contents of processing in the chat mode will be described later. Interaction, in a broad sense, means that a virtual character reacts to an action performed by a player. Interaction with a virtual character includes interaction made as communication such as conversation with a virtual character. In this specification, an interaction performed as communication with a virtual character is sometimes called a communicative interaction. Interactions with virtual characters can include, in addition to communicative interactions, battles with virtual characters, cooperative play in which a game is played in cooperation with virtual characters, and interactions with other virtual characters. In this specification, an interaction performed as a battle with a virtual character is sometimes referred to as a battle interaction. In this specification, an interaction performed as cooperative play with a virtual character may be referred to as a cooperative interaction.

The second mode is VR mode, for example. In the VR mode, functions for performing various interactions with virtual characters appearing in the virtual space displayed on the display of the information processing device 20 are implemented. Specific contents of processing in the VR mode will be described later. The VR mode is an example of the second mode, and the second mode may include any game mode in which processing for progressing the game is performed using head tracking information.

In one embodiment, the game having the first mode and the second mode realized by the application processing system 1 may be a game in which interaction other than communicative interaction with a virtual character is performed. A game realized by the application processing system 1 does not have to involve communicative interaction. In one embodiment, the game is played in a two-dimensional space in a first mode and in a three-dimensional space in a second mode. In one embodiment, the game is played in a three-dimensional space in the first mode, and displayed in a display mode different from the three-dimensional space in the first mode in the second mode (or similar to the three-dimensional space in the first mode). are played in a three-dimensional space (configured in different ways). In one embodiment, the game realized by the application processing system 1 may use common game media in the first mode and the second mode. Parameters associated with this game content may be carried over between the first mode and the second mode. For example, if the parameters of a specific game medium are changed during play in the first mode, and then the mode is shifted to the second mode, the specific game medium in which the changed parameters are set is played in the second mode. Used. In one embodiment, certain game content parameters may be changed during transitions from the first mode to the second mode and/or during transitions from the second mode to the first mode. This change in the parameters of the game media may be a change that is advantageous or disadvantageous to the player 5 in terms of the progress of the game. In one embodiment, the result of playing in the first mode may be reflected in the game in the second mode, and the result of playing in the second mode may be reflected in the game in the first mode. For example, the experience value of player 5 acquired in the first mode may be inherited in the second mode.

As described above, the first mode and the second mode of the game implemented in the application processing system 1 are modes that are distinguished from each other. That is, the first mode is a mode different from the second mode. In one embodiment, the HMD 10 is not worn on the head of the user 5 when the game realized in the application processing system 1 is played in the first mode, whereas the game is played in the second mode. In this case, the HMD 10 is worn on the head of the user 5 . In this case, in the first mode, game processing is performed according to the first detection information detected by the information processing device 20 that is not worn on the head of the user 5, and in the second mode, the Game processing is performed according to the second detection information detected by the HMD 10 mounted on the head.

In one embodiment, a stereoscopic image is displayed on the display 24 in the second mode, whereas an image other than the stereoscopic image is displayed on the display 24 in the first mode, as described above. In the second mode, at least the virtual character image among the game images is displayed as a stereoscopic image, and in the first mode, the virtual character image is displayed as an image other than the stereoscopic image.

In one embodiment, in the first mode, processing is performed to progress the game without using head tracking information calculated based on the second detection information, whereas in the second mode, the head tracking information is not used. Based on this, processing for advancing the game is performed.

In one embodiment, in the first mode, the game progresses according to the touch operation detected by the touch panel 24b, whereas in the second mode, there is no process according to the touch operation of the touch panel 24b. not.

In one embodiment, when interaction with a virtual character is provided in a game implemented in the application processing system 1, in the first mode, the information processing device 20 that is not worn on the head of the user 5 detects In the second mode, interaction with the virtual character is provided according to the detected first detection information, and in the second mode, interaction with the virtual character is provided according to the second detection information detected by the HMD 10 mounted on the head of the user 5. interaction is provided.

After starting the game, it is possible to switch between the first mode and the second mode, which are distinguished as described above. With reference to FIG. 5, an outline of mode transition of the game processed by the application processing system 1 will be described. As shown in the figure, when the game is started in step S1, chat mode S2, which is the first mode, is started in step S2. In this chat mode, when transition processing for transitioning to the VR mode, which is the second mode, is started, the process proceeds to step S3, and transition processing to the second mode is performed. When the transition process is completed, the VR mode, which is the second mode, is started in step S4. When the VR mode ends or is interrupted, processing for returning to the chat mode is performed.

Next, functions realized by the computer processor 21 will be described more specifically. The computer processor 21 functions as a chat mode execution unit 21a, a transition processing unit 21b, and a VR mode execution unit 21c by executing computer-readable instructions. At least part of the functions implemented by computer processor 21 may be implemented by a computer processor other than computer processor 21 of game system 1 . At least part of the functions implemented by the computer processor 21 may be implemented by a computer processor installed in the server 50, for example.

The chat mode execution unit 21a performs processing for providing a game in the chat mode, which is the first mode, by executing an application processing program according to one embodiment. The chat mode execution unit 21a realizes a function for chatting between the user 5 and the virtual character. After logging into the chat mode, the chat mode execution unit 21a displays the message of the virtual character and the message input or selected by the user 5 on the display 24, and causes the virtual character and the user 5 to chat. can be done. The chat mode execution unit 21a may display on the display 24, following the message of the virtual character, a plurality of options regarding the user's 5 response to the message. A message for the user 5 is specified from the plurality of options according to the operation of the user 5, and the specified message is displayed as the message for the user 5 following the message of the virtual character. The user 5 can select a desired one of the displayed options by performing a touch operation on the touch panel 24b. The virtual character's message and the user's options for the message can be specified by referring to the chat data 28b according to the application processing program. The message of the virtual character may be displayed together with the image of the virtual character, and the message of the user 5 may be displayed together with the image of the user 5 . The virtual character's message includes a message prompting the transition to the VR mode, which is the second mode, a message for allowing the user 5 to select settings in the VR mode, and other messages associated with the VR mode. It may be

In the chat mode, a transition start object for starting transition to the VR mode, which is the second mode, is displayed according to the progress of the chat. The transition start object is displayed when the mode transition condition is satisfied according to the application processing program. When it is detected that the transition start object has been selected based on a detection signal from the touch panel 24b or another user interface, processing for transitioning to the VR mode, which is the second mode, is started.

The transition processing unit 21b performs transition processing for transitioning from the first mode to the second mode. This transition processing includes displaying on the display 24 guidance prompting the user to attach the information processing device 20 to the mounting tool 11, and displaying on the display 24 a transition object that can be selected by the user's gaze. It's okay. After displaying this transitional object, the transition processing unit 21b receives a detection signal from the sensor unit 25 or other sensors, and determines whether or not the transitional object has been selected by gaze based on this detection signal. can be done. For this determination, the transition processing unit 21b calculates the position and orientation of the HMD 10 based on the detection signal from the sensor unit 25, and specifies the gaze position of the user 5 based on the calculated position and orientation of the HMD 10. do. Identification of the gaze position can be performed by various known methods. For example, the gaze position may be specified based on the detection signal of an eye tracking sensor. The transition processing unit 21b measures, for example, the time the gaze position is on the transition start object (gazing time), and determines that the selection of the transition start object has been completed when the gaze time reaches a predetermined time. When the migration processing unit 21b determines that the selection of the migration object is completed, it starts the VR mode, which is the second mode.

The VR mode execution unit 21c constructs a virtual space including virtual characters and other objects, and outputs image information for displaying this virtual space to the display 24. FIG. The VR mode execution unit 21c can generate the image information of the virtual space 60 so as to correspond to the scene determined according to the progress of the game.

The virtual space constructed by the VR mode execution unit 21c will be described with further reference to FIG. 4a. FIG. 4a is a diagram schematically showing the virtual space 60 configured by the VR mode execution section 21c. FIG. 4a shows the virtual space 60 at the time of initialization in the first scene. The VR mode execution unit 21c specifies the position of the user 61 and the reference line of sight 62 in the virtual space 60 based on the initial settings of the first scene, and also specifies a predetermined position centered on the reference line of sight 62 in the entire virtual space 60. is specified as the initial display area 63 . The initial display area 63 is, for example, a range within a predetermined angle of view centered on the reference line of sight 62 . The initial display area 63 is defined by the non-display area and the boundary line 63a. When the processing of the first scene is started, the VR mode execution unit 21c generates image information indicating an image of the virtual space within the initial display area 63 of the entire virtual space 60, and displays the generated image information on the display 24. output to An image of the virtual space 60 corresponding to the initial display area 63 is displayed on the display 24 based on the image information thus generated. Image data 28a, scenario data 28c, event data 28d, game progress data 28e, and other data stored in the storage 27, for example, can be used to generate image information.

When the motion of the head of the user 5 is detected by the HMD 10 after starting the processing of the first scene in the VR mode, the VR mode execution unit 21c follows the detected motion of the head of the user 5. , the reference line of sight 62 is moved, and a range within a predetermined angle of view centered on the reference line of sight 62 after movement is specified as a new display area. In this way, the VR mode execution unit 21c updates the display area as needed according to the movement of the user's 5 head by the HMD 10, and outputs the image of the virtual space 60 corresponding to the updated display area to the display 24. . As a result, the image of the virtual space 60 displayed on the display 24 also changes according to the display area. The user 5 can thus see an image of the virtual space 60 that changes according to the movement of his head. When the user 61 moves within the virtual space 60 , the orientation of the reference line of sight 62 changes according to the position of the user 61 within the virtual space 60 .

A main display area 64 is provided in the virtual space 60 . The main display area 64 is an example of a first main display area. The main display area 64 is provided at a predetermined position in the virtual space 60 determined for each scene. The main display area 64 is separated from other areas by a boundary line 64a. The main display area 64 is, for example, an area including a position to which the line of sight of the user 5 is likely to be directed in each scene. The main display area 64 is arranged, for example, inside the initial display area 63 determined based on the initial settings of each scene. A main display area 64 in a certain scene is set, for example, so that at least part of an object representing a virtual character that performs interaction is included in an initial display area 63 determined based on the initial settings of the scene. The main display area 64 may be an area that forms part of the initial display area 63 or may be an area that coincides with the initial display area 63 . In the first scene shown in FIG. 4a, a main display area 64 is provided around a reference line of sight 62 of a user 61 in a standing posture.

A virtual character object and various other objects are placed in the virtual space 60 . Objects included in the virtual space 60 may include a specific object that can be selected by the user 5's gaze. A menu object 66 is shown in FIG. 4a as an example of such a specific object.

The menu object 66 may be associated with one or more menu items that can be selected by the gaze of the user 5 . Each of the one or more menu items may be associated with an action of user 61 within virtual space 60 . The menu items included in the menu object 66 are actions that are not associated with movement of the user's 61 head in the virtual space 60, such as hand actions associated with movement of the user's 61 hand in the virtual space 60, or It may be associated with a tool action related to the use of a tool by user 61 at 60 . A tool action is, for example, a photograph action in which the user 61 takes a photograph in the virtual space 60 using a camera. When multiple menu items are associated with the menu object 66, only some of them may be selectable as the game progresses.

As shown in FIG. 4a, the menu object 66 is positioned in the virtual space 60 at a location not included in the main display area 64. As shown in FIG. That is, the menu object 66 is arranged outside the main display area 64 . The menu object 66 may be placed inside the initial display area 63 or outside the initial display area 63 .

Figure 4b shows the virtual space 60 in the second scene. The second scene is a game scene different from the first scene. The second scene is executed before the first scene or after the second scene. 64 are provided. Also in this second scene, the menu object 66 is arranged outside the main display area 64 .

The VR mode execution unit 21c displays an image of the virtual character in the virtual space according to the application processing program, and realizes interaction between the virtual character and the user 5. FIG. The VR mode execution unit 21c determines the action of the virtual character in the virtual space according to the application processing program, and generates image information and audio information corresponding to the action of the virtual character. The image information is output to the display 24 , and the audio information is output to the speaker of the HMD 10 . As a result, an image in which the virtual character moves in the virtual space is displayed on the display 24, and sound corresponding to the movement of the virtual character is output from the speaker. For example, when the virtual character speaks to the user 5, an image in which the head and mouth of the virtual character move is displayed on the display 24, and the voice corresponding to the speech of the virtual character is output from the speaker. For example, image data 28a, scenario data 28c, event data 28d, game progress data 28e, and other data stored in the storage 27 are used to generate image information and audio information corresponding to actions of the virtual character. obtain. In one embodiment, the virtual character that interacts with user 5 in each of VR mode and chat mode is the same virtual character. For example, a virtual character appearing in the VR mode and a character appearing in the chat mode have the same name, and have similarities in appearance to the extent that they can be recognized as the same character. However, while the virtual character image is displayed as a stereoscopic image in the VR mode, the virtual character image is displayed as an image other than the stereoscopic image in the chat mode. The images are different from each other. However, such a difference in representation does not prevent the identity of the virtual characters.

In addition to the functions described above, the VR mode execution unit 21 c has a function of identifying actions of the user 61 in the virtual space 60 based on detection signals from the sensor unit 25 . Actions of the user 61 in the virtual space 60 are, for example, actions by the user 61 with respect to virtual characters appearing in the virtual space 60 . Since the action of the user 61 in the virtual space 60 is virtual in the virtual space 60, it may or may not match the body movement of the user 5 in the real space or the movement of the user 5 in the real space. be.

Actions of the user 61 in the virtual space 60 include a first action associated with the movement of the user's 61 head and a second action not associated with the movement of the user's 61 head. The first action associated with the movement of the head of the user 61 in the virtual space 60 includes an action in which the user 61 nods in the virtual space 60, an action in which the user 61 shakes his head in the virtual space 60, and other actions. Actions associated with head movements of user 61 in virtual space 60 are included. The VR mode execution unit 21c calculates the position and orientation of the HMD 10 based on the detection information detected by the sensor unit 25 provided in the HMD 10 worn on the head of the user 5, thereby allowing the user 5 in the real space. head movements can be identified.

The VR mode execution unit 21c detects movement of the head of the user 5 in the real world based on one or more pieces of detection information detected by the HMD 10, and detects the movement of the head of the user 5 in the real world. A matching or corresponding action is identified as the first action of user 61 in virtual space 60 . For example, when it is detected that the user 5 has performed a nodding motion, the action of the user 61 nodding (in the virtual space 60 ) is specified as the first action of the user 61 in the virtual space 60 . The first action of the user 61 in the virtual space 60 is not included in the image of the virtual space 60 when the virtual space 60 is displayed on the display 24 as a first-person viewpoint image.

The second action of user 61 in virtual space 60 is not associated with the movement of the head of user 61 in virtual space 60 . This second action is associated with movement of a part other than the head of the user 61 in the virtual space 60, for example movement of the hand of the user 61. FIG. The second action includes an action of "stroking" the user 61 in the virtual space 60, an action of "poking" the user 61 in the virtual space 60, and movements of parts other than the head of the user 61 in the virtual space 60 other than these. Contains actions associated with The action "stroking" is an example of a first gaze position action, and the action "poking" is an example of a second gaze position action.

In one embodiment, the VR mode execution unit 21c is configured to specify the second action in response to the menu object 66 being selected. The VR mode execution unit 21c can detect that the menu object 66 has been selected based on the gaze position calculated based on the detection signal of the HMD 10. FIG. For example, the VR mode execution unit 21c can determine whether or not the selection of the menu object 66 has been completed based on whether or not the gaze position is on the menu object 66 for a predetermined time or longer. For example, if the gaze position is on the menu object 66 for a predetermined time or longer, it is determined that the selection of the menu object 66 has been completed.

When the menu object 66 includes a plurality of menu items, the VR mode execution unit 21c selects the menu item when the gaze position is on one of the plurality of menu items for a predetermined time or longer. can be detected.

When the VR mode execution unit 21c detects the selection of the menu object 66 or the selection of one of the menu items included in the menu object 66, the action corresponding to the selected menu object 66 or the selected menu item is This is set as a selection action that the user 61 performs on the virtual character. As described above, the menu items include menu items corresponding to hand actions related to hand movements of the user 61 in the virtual space 60 and menu items corresponding to photo actions related to taking a picture in the virtual space 60. may be included. When the menu item associated with this hand action is selected, the VR mode execution unit 21c sets this hand action as the selected action for the virtual character.

When the second action is set, the VR mode execution unit 21c may set one or more character gaze positions on the virtual character. When at least one of the one or more gaze positions is selected, the VR mode execution unit 21c specifies an action for the virtual character based on the selected gaze position and the selected action. be able to. For example, when a menu item corresponding to a hand action is selected and a gaze position set to the head of a virtual character is selected, an action of stroking the head of the virtual character is specified, and this specified action is specified. Actions take place in virtual space 60 .

The VR mode execution unit 21c can determine the action (reaction) of the virtual character in the virtual space according to the action of the user 61. FIG. The VR mode execution unit 21c generates image information for displaying the action of the virtual character determined according to the action of the user 5, and outputs the generated image information to the display 24 as a stereoscopic image. As a result, the display 24 displays an image of the virtual character performing the action determined according to the action of the user 5 . In this way, the VR mode execution unit 21c determines the action (reaction) of the virtual character in response to the action of the user 61, and outputs image information of the virtual character corresponding to the determined action to the display 24. FIG. In the VR mode, interaction between the user 5 and the virtual character displayed as a stereoscopic image is realized in this way.

The VR mode execution unit 21c performs termination processing for terminating the VR mode when the termination condition is satisfied. The end conditions are, for example, that a predetermined time (for example, one minute) has elapsed since the start of the VR mode, that an end target operation has been detected, and that the last event included in the scenario being executed in the VR mode has ended. , and other conditions. The termination process performed when the termination condition is satisfied includes displaying guidance on the display 24 prompting the user to remove the information processing device 20 from the wearing tool 11, and displaying a login screen for the chat mode, which is the first mode. Displaying may be included.

Next, chat processing in the chat mode will be described with reference to FIGS. 6 and 7. FIG. FIG. 6 is a flowchart showing the flow of processing in chat mode in one embodiment, and FIG. 7 shows an example of a display image in chat mode. At the start of the chat mode, it is assumed that the HMD 10 is not worn on the head of the user 5 and the information processing device 20 is removed from the wearing tool 11 .

In the application processing system 1, the game is started in chat mode. In step S<b>11 , a login screen for logging into the chat mode is displayed on the display 24 of the information processing device 20 . When the user 5 performs the login process, the chat process proceeds to step S12.

In step S12, messages are exchanged between the user 5 and the virtual character, whereby a chat is conducted between them. When the chat mode is logged in, a chat display image for displaying a chat between the user 5 and the virtual character is generated and displayed on the display 24 . FIG. 7 shows a chat display image 70, which is an example of a chat display image displayed on the display 24. As shown in FIG. The chat display image 70 includes an icon 73 corresponding to the user 5, an icon 74 corresponding to the virtual character, a message 75a of the user 5, messages 76a and 76b of the virtual character, and a transition start object 78. Includes area 71 . The chat display image 70 also includes a menu area 72 arranged above the chat display area 71 . The virtual character messages 76a and 76b are specified based on the chat data 28b and other data stored in the storage 72 according to the application processing program. For example, virtual character messages are displayed in order from the message at the root node with reference to the chat data 28b defined in the tree structure. At a node branch point, a node is selected according to the branch condition, and a message corresponding to the selected node is displayed.

User 5 can select one of the options included in chat display image 70 . This selection is made by touching with a finger the area where the option to be selected is displayed on the display 24 . When this selection is made, a message 75b corresponding to the selected option is newly displayed in the chat display area 71 as the message for the user 5. FIG.

In step S13, it is determined whether or not the condition for mode transition from the first mode to the second mode is satisfied. As a mode transition condition from the first mode to the second mode, for example, a condition that a predetermined time (for example, one minute) has passed since the start of the chat mode is used. Elapsed time from the start of the chat mode is measured using, for example, the system clock. The process returns to step S12 and the chat process continues until it is determined that the mode transition condition is satisfied. On the other hand, if it is determined that the mode transition condition is satisfied, the chat process proceeds to step S14.

In step S14, the transition start object 78 is displayed on the display 24, and the chat process proceeds to step S15. As shown in FIG. 7, the transition start object 78 is displayed, for example, in the chat area 71 as an object with a circular outer edge. In step S15, when the selection of the transition start object is confirmed, the chat process proceeds to step S16. On the other hand, if the selection of the transition start object is not confirmed even after a predetermined time (for example, 10 seconds) has elapsed since the transition start object was displayed, it is determined that the transition start object was not selected, and the chat process ends. do. When it is determined that the transition start object has not been selected, the process may return to step S12 and perform processing for resuming the chat. Whether or not the transition start object has been selected may be determined according to the operation on the touch panel 24b. For example, when it is detected from the detection signal from the touch panel 24b that a touch operation (for example, a tap operation) has been performed on a position overlapping the display area of the transition start object, it is determined that the transition start object has been selected. .

In step S16, processing for shifting to the VR mode, which is the second mode, is started. Once the transition process is started, the chat process ends. In the chat mode, scene options in the destination VR mode may be displayed on the display 24 . The user 5 can select a favorite scene from these options.

The chat processing described above is executed by the chat mode execution unit 21a. The chat mode execution unit 21a can execute the above-described chat processing independently or appropriately in cooperation with other functions. When executing the chat process, in addition to the data stored in the storage 27, data stored in other storages, detection information detected by various sensors, or data other than these are used as necessary. can be

In the mode transition processing from the chat mode, which is the first mode, to the VR mode, which is the second mode, guidance prompting the user to wear the HMD 10 is displayed on the display 24 of the information processing device 20 . Following this guidance, display 24 displays transition objects. When this transition object is selected by user 5, the VR mode is started. The user 5 can select this transitional object by wearing the HMD 10 and continuously gazing at the transitional object for a predetermined period of time.

Next, VR processing in the VR mode will be described with reference to FIGS. 8 and 9. FIG. FIG. 8 is a flow diagram showing the flow of VR processing in VR mode in one embodiment, and FIG. 9 shows an example of a display image in VR mode. It is assumed that the HMD 10 is worn on the head of the user 5 at the start of the VR mode. In the following description of the VR mode, it is assumed that the user 5 has selected the first scene in the chat mode. Various scenes can be set as the first scene. For example, the first scene may be a scene for having a conversation anywhere indoors or outdoors. The VR processing in FIG. 8 is repeatedly performed until the termination condition is satisfied. The end conditions are, for example, that a predetermined time (for example, one minute) has elapsed since the start of the VR mode, that an end target operation has been detected, and that the last event included in the scenario being executed in the VR mode has ended. , and other conditions.

When the VR mode is started, image information of the virtual space corresponding to the first scene is generated and an image corresponding to the image information is output to the display 24 in step S21. The image information of the virtual space 60 in the first scene is generated by referring to the image data 28a, for example. In step S<b>21 , first, image information of the initial display area 63 in the virtual space 60 in the first scene is generated, and an image corresponding to the image information is output to the display 24 . FIG. 9 shows an example of the display image of the initial display area 63 of the first scene. The VR image 80 shown in FIG. 9 includes images showing a virtual character 65, a menu object 66, objects 82a to 82c corresponding to the background, and various other objects. A menu object 66 is arranged outside the main display area 64 . Since the VR image 80 is a first-person viewpoint image, an image representing the user 61 is not displayed.

The virtual character image 65 can perform various actions within the virtual space. For example, the actions of the virtual character image 65 include speaking to the user 61 (toward the position of the user 61 in the virtual space), moving in the virtual space, and picking up an object placed in the virtual space. , and various other actions in the virtual space.

When the sensor unit 25 of the HMD 10 detects the movement of the head of the user 5 while the VR image 80 is being displayed on the display 24, the movement in the virtual space 60 is performed based on the detected movement of the head. A first action of user 61 is identified. Then, a behavior (reaction) of the virtual character according to the specified first action of the user 61 is determined, and image information of the virtual character 65 performing the determined behavior is generated. The image information generated in this manner is output to the display 24 . For example, when it is detected that the user 5 performs a motion of shaking the head in the vertical direction (nodding motion), the action of nodding is specified as the first action of the user 61 in the virtual space 60 based on the detection information of the HMD 10 . be done. Then, image information of the virtual character image 65 that takes the reaction to the first action of nodding by the user 61 is generated, and the image information is displayed on the display 24 . Thus, in the VR mode, interaction between the user and the virtual character 65 is realized using the stereoscopic image of the virtual character.

In step S22, it is determined whether or not an end condition for ending the VR image display processing is satisfied. If it is determined that the termination condition is not satisfied, the process returns to step S21 to continue the VR image display processing. On the other hand, if it is determined that the end condition is met, the VR process proceeds to step S23.

In step S23, VR mode end processing is performed. This termination process may include displaying on the display 24 guidance prompting the user to remove the information processing device 20 from the wearing tool 11, and displaying a login screen for the chat mode, which is the first mode. .

The above VR processing is executed by the VR mode execution unit 21c. The VR mode execution unit 21c can execute the above-described VR processing independently or appropriately in cooperation with other functions.

Next, a second action identifying process for identifying a second action will be described with reference to FIGS. 10 and 11a to 11d. FIG. 10 is a flow diagram showing the flow of the second action specifying process in one embodiment, and FIGS. 11a to 11d show examples of display images in the process for specifying the second action. The processing for identifying the second action is performed in parallel with the VR processing described with reference to FIG.

First, in step S31, it is determined whether or not the menu object 66 has been selected. For example, when the gaze time for gazing at the menu object 66 reaches a predetermined time, it is determined that the menu object 66 has been selected. If the menu object 66 has not been selected, the determination of step 31 is repeated. On the other hand, if it is determined that the menu object 66 has been selected, the second action specifying process proceeds to step S32.

At step S32, the menu items associated with the menu object 66 are displayed on the display 24. FIG. FIG. 11a shows an example image of a virtual space including menu items associated with menu object 66. FIG. The VR image 80 shown in FIG. 11a includes icons corresponding to two menu items respectively. Specifically, the VR image 80 includes a menu image 67, which includes a menu item icon 68a and a menu item icon 68b. The menu item icon 68a is an icon indicating a menu item corresponding to a hand action related to movement of the hand of the user 61 in the virtual space 60, and the menu item icon 68b is an icon representing a photo action related to taking a picture in the virtual space 60. It is an icon indicating the corresponding menu item. After the processing for displaying the menu item on the display 24 is performed, the second action specifying processing proceeds to step S33. The menu item icon 68a and the menu item icon 68b displayed in the virtual space may be displayed parallel to one plane of the virtual structure arranged in the virtual space. For example, in the illustrated embodiment, menu item icon 68a and menu item icon 68b may be displayed parallel to the floor, wall, or ceiling of the room in the virtual space shown in Figure 11a. good. Such arrangement of the menu item icon 68a and the menu item icon 68b allows the menu item icon 68a and the menu item icon 68b to be displayed as if they were virtual objects arranged in the virtual space. Thereby, it is possible to prevent the user 61 from impairing the sense of immersion. The menu item icon 68a may be arranged on a plane different from the plane on which the menu item icon 68b is arranged in the virtual space. Components of the menu image 67 other than the menu item icon 68a and the menu item icon 68b may also be displayed parallel to one plane of the virtual structure arranged in the virtual space.

At step S33, it is determined whether or not any of the menu item icons displayed at step S32 has been selected. For example, when the gaze time for gazing at any one of the menu item icons included in the VR image 80 reaches a predetermined time, it is determined that the menu item icon has been selected. In the illustrated example, it is determined that the menu item icon 68a has been selected when the gaze time for gazing at the menu item icon 68a reaches the predetermined time, and the gazing time for gazing at the menu item icon 68b reaches the predetermined time. It is determined that the selection of the menu item icon 68b is performed when the menu item icon 68b is selected. More than one menu item icon may be selected. If the selection of the menu item icon is not confirmed in step 33, the display of the menu item icon is stopped, and the second action specifying process returns to step S31. On the other hand, if it is determined that the menu item icon has been selected, the second action specifying process proceeds to step S34.

In step S<b>34 , one or more character gaze positions are set on the virtual character 65 . The setting of the character gaze position on the virtual character 65 will be described with reference to FIG. 11b. FIG. 11b schematically shows the character gaze position within the VR image 80 in order to explain the character gaze position set on the virtual character 65 . The character gaze position may be set so as to be visible by the user 5 in the VR image 80, or may be set so as not to be visible. In the example shown in FIG. 11b, a plurality of character gaze positions 81a to 81d are set on the virtual character 65. In the example shown in FIG. One or more character gaze positions set on the virtual character 65 are each associated with a specific part of the virtual character 65 . In the example of FIG. 11b, the character gaze positions 81a-81d are respectively associated with the head, hair, shoulders, and arms of a virtual character 65 modeled after a female person. When the character gaze position is set on the virtual character 65, the second action specifying process proceeds to step S35.

In step S35, it is determined whether or not any of the one or more character gaze positions set on the virtual character 65 has been gazed at. In the example shown in FIG. 11b, when the gaze time for gazing at any of the character gaze positions 81a to 81d reaches a predetermined time, it is determined that the character gaze position has been selected. In step S35, if no character gaze position is gazed at, the setting of the character gaze position is cancelled, and the second action specifying process returns to step S31. On the other hand, if it is determined that any character gaze position has been gazed at, the second action specifying process proceeds to step S36.

In step S36, an action for the virtual character 65 is specified based on the menu item selected in step S33 and the character gaze position gazed in step S35, and the specified action is performed for the virtual character 65. For example, when the menu item icon 68a associated with the hand action is selected in step S33 and the character gaze position 81a associated with the head of the virtual character 65 is gazed in step S35, the virtual character The action of stroking the head of 65 is specified as the second action for the virtual character 65 of the user 61 in the virtual space 60 .

When the menu item icon 68a is selected, different second actions may be specified depending on the gaze position of the gazed character. For example, even if the menu item icon 68a is selected as in the above example, if the character gaze position 81c is gazed instead of the character gaze position 81a, an action different from the action of "stroking" will be performed. may be specified as the second action. When the menu item icon 68a is selected and the character gaze position 81c is gazed upon, for example, an action of "poking" the shoulder of the virtual character 65 may be specified as the second action.

When the second action is specified, the specified second action is performed on the virtual character 65 . For example, when the action of stroking the head of the virtual character 65 is specified as the second action, an image representing the second action is displayed as a VR image 80 on the display 24 as shown in FIG. 11c. . In the example shown in FIG. 11c, a virtual hand of user 61 is shown near the head of virtual character 65 to illustrate the action of user 61 stroking the head of virtual character 65 in virtual space 60. An image 83a is displayed. In another example, when the action of pecking the shoulder of the virtual character 65 is specified as the second action, the action of pecking the shoulder of the virtual character 65 by the user 61 in the virtual space 60 as shown in FIG. 11d. , an image 83b showing a virtual hand of the user 61 is displayed near the shoulder of the virtual character 65. As shown in FIG. The images 83a and 83b may be different from each other in order to represent different actions performed on the virtual character 65 . The virtual character 65 behaves (reacts) according to the application processing program in response to the second action of the user 61 .

In step S33, the menu item icon 68b associated with the photo action is selected, and one of the character gaze positions set on the virtual character 65 in step S35 (for example, the character gaze position 81a) is gazed at. good too. In this case, the action of taking a picture including the virtual character 65 is specified as the second action. In response to the menu item icon 68b being selected, the virtual character 65 may perform an action of posing for the camera held by the user 61. FIG. Further, an action of releasing the shutter may be performed according to selection of one of the character gaze positions set on the virtual character 65 . In this case, the image that is being output to the display 24 when one of the character gaze positions is selected is stored in the storage 27 as an action taken by the photo action.

The second action specifying process described above is executed by the VR mode execution unit 21c. The VR mode execution unit 21c can execute the above-described second action specifying process independently or appropriately in cooperation with other functions.

Next, an application processing system according to another embodiment will be described with reference to FIG. FIG. 12 is a block diagram illustrating application processing system 101 according to another embodiment. The application processing system 101 includes an information processing device 20 , an HMD 110 and an HMD 110 . The application processing system 101 differs from the application processing system 1 in that the VR mode can be provided without attaching the information processing device 20 to the HMD 110 . The application processing system 101 will be described below, focusing on the differences from the application processing system 1 .

The HMD 110, the information processing device 20, and the server 50 are connected via a network 40 so as to be able to communicate with each other. The HMD 110 and the information processing apparatus 20 may be communicably connected according to a short-range wireless system such as Bluetooth (registered trademark) without going through the network 40 . The HMD 110 differs from the HMD 10 of the application processing system 1 in that the VR mode can be provided without the information processing device 20 attached.

The HMD 110 includes a computer processor 121, a memory 122, a communication I/F 123, a display 124, a sensor unit 125, a sound collector 126, and a storage 127. The computer processor 121, the memory 122, the communication I/F 123, the display 124, the sensor unit 125, the sound collector 126, and the storage 127 are the computer processor 21, the memory 22, the communication I/F 23, the display 24, and the sensor of the information processing device 20. It is configured similarly to the unit 25 , sound collector 26 and storage 27 . However, the display 124 may not have a touch panel. Each component of these HMD 110 is assembled in the housing. This housing may be formed in a goggle shape like the housing 11a. Image data 28a, chat data 28b, scenario data 28c, event data 28d, and game progress data 28e may be stored in storage 127, storage 27, or other storage.

The functions of the chat mode execution unit 21a, the transition processing unit 21b, and the VR mode execution unit 21c are shared between the information processing device 20 and the HMD 110. FIG. Specifically, the function of the chat mode execution unit 21 a is implemented in the information processing device 20 , and the function of the VR mode execution unit 21 c is implemented in the HMD 110 . A part of the function of the transition processing unit 21 b is realized by the information processing device 20 and the remaining part is realized by the HMD 110 .

When starting the game in the application processing system 101 , the user 5 uses the information processing device 20 to start the chat mode. This chat mode processing is performed by the chat mode execution unit 21 a of the information processing device 20 .

When the mode transition condition is satisfied in the chat mode and the transition start object is selected, mode transition processing to the VR mode is started.

When the VR mode is started, the VR mode processing is performed by the VR mode execution unit 21 c of the HMD 110 .

In this manner, application processing system 101 can provide user 5 with interaction with a virtual character via HMD 110 .

According to the above embodiment, the following effects are obtained. According to the application processing systems 1 and 101 described above, by gazing at the character gazing positions 81a to 81d on the virtual character 65 after selecting the menu object 66, movement of the head of the user 61 in the virtual space 60 can be correlated. It is possible to specify a second action that does not exist. As a result, the second display that is not associated with the movement of the head of the user 61 in the virtual space 60 can be performed without using an operation device other than the head-mounted displays 10 and 110 such as an operation controller designed to be held by hand. can take action.

The second actions associated with the motions of the user's hand in the virtual space 60 include, for example, actions of "stroking", "hitting", "poking", and "pinching" the virtual character 65 in the virtual space 60; , an action of handling an item with the hand (for example, an action of taking a picture), and other actions associated with hand movements of the user 61 in the virtual space 60 .

According to the application processing systems 1 and 101 described above, the second action associated with the movement of the hand of the user 61 in the virtual space 60 is set to the virtual character 65 instead of the position and movement of the hand of the user in the real world. This is specified by gazing at the character gazing positions 81a to 81d. Thereby, the user can perform the second action that is not associated with the movement of the user's head without using any operation device other than the head mounted displays 10 and 110 .

According to the application processing systems 1 and 101 described above, it is possible to perform a second action that is not associated with the movement of the user's head by selecting a specific object.

According to the application processing systems 1 and 101 described above, it is possible to selectively use a plurality of second actions including the first gaze position action and the second gaze position action according to the character gaze positions 81a to 81d. As a result, there are many variations of the second action that are not associated with the movement of the user's 61 head.

According to the application processing systems 1 and 101 described above, it is possible to selectively use various types of second actions according to the selected menu item. As a result, there are many variations of the second action that are not associated with the movement of the user's 61 head.

In the above embodiment, menu object 66 is a functional object for enabling the second action. The user 5 may unintentionally select the menu object 66 depending on the arrangement and display mode in the virtual space. Also, functional objects such as the menu object 66 can reduce the user's sense of immersion in the virtual space. According to the application processing systems 1 and 101 described above, since the menu object 66 is arranged outside the main display area 63 in the virtual space 61, it is possible to suppress erroneous selection and a decrease in the sense of immersion.

According to the application processing systems 1 and 101 described above, an appropriate main display area can be determined according to the scene. For example, between the first scene and the second scene, the position of the main display area 63 is changed according to the posture of the user 61 .

According to the application processing systems 1 and 101 described above, it is possible to make the virtual character 65 react according to the action of the user 61 , thereby realizing an interaction between the user and the virtual character 65 .

Embodiments of the present invention are not limited to the embodiments described above, and various modifications are possible without departing from the scope of the invention. For example, some or all of the functions performed by computer processor 21 and computer processor 121 may be implemented by computer processors not shown in the above embodiments without departing from the spirit of the invention. For example, the application processing system 1 and the application processing system 101 may include a game machine that executes at least part of an application processing program, and part of the functions realized by the processing of the computer processor 21 or the computer processor 121 is It may be realized by processing in the game machine.

Embodiments of the present invention may include various apparatus, devices, and electronic components in addition to those described above. For example, the application processing system 1 and the application processing system 101 may include an operation device for accepting the operation of the user 5 in addition to the information processing device 20 and the HMDs 10 and 110 . The application processing system 1 and the application processing system 101 may detect an operation by the user 5 using the operation device, and perform game processing according to the detected operation.

In the processing procedures described in this specification, especially in the processing procedures described using flow diagrams, some of the steps that make up the processing procedure are omitted, and as a step that makes up the processing procedure It is possible to add unspecified steps and/or change the order of the steps, and the processing procedures in which such omissions, additions, and order changes do not deviate from the spirit of the present invention. Included within the scope of the invention.

The invention described in the scope of claims as originally filed in the original application of the present application will be additionally described below.
[1]
An application processing system that includes one or more computer processors and provides the user with interaction with the virtual character by outputting an image of a virtual space containing the virtual character to a head-mounted display worn on the user's head. There is
The one or more computer processors, by executing computer readable instructions,
A first action, which is an action to be performed on the virtual character and is associated with movement of the user's head in the virtual space, based on one or more pieces of detection information detected by the head-mounted display. identify,
an action performed on the virtual character based on the one or more pieces of detection information in response to the selection of a specific object contained in the virtual space, the action being the user's head in the virtual space; identify secondary actions not associated with the movement of
Application processing system.
[2]
The second action is an action associated with the movement of the user's hand in the virtual space.
The application processing system according to [1].
[3]
the second action is an action associated with use of a tool by the user in the virtual space;
The application processing system according to [1].
[4]
one or more character gaze positions are set on the virtual character according to the selection of the specific object;
The second action is specified in response to at least one of one or more character gaze positions set on the virtual character being gazed at;
The application processing system according to any one of [1] to [3].
[5]
the one or more character gaze positions include a first character gaze position and a second character gaze position;
the second action includes a first gaze position action and a second gaze position action;
A first gaze position action is specified as the second action by the gaze at the first character gaze position, and a second gaze position action is specified as the second action by the gaze at the second character gaze position,
The application processing system according to [4].
[6]
the specific object is associated with a first menu item and a second menu item;
the second action includes a first menu item action associated with the first menu item and a second menu item action associated with the second menu item;
By gazing at least one of the one or more character gazing positions after gazing at the first menu item, the first menu item action is identified as the second action and gazing at the second menu item. Later, at least one of the one or more character gaze positions is gazed at, thereby specifying the second menu item action as the second action;
The application processing system according to [5].
[7]
The one or more computer processors identify a first main display region in the virtual space in the first scene,
wherein the specific object is arranged outside the first main display area in the first scene;
The application processing system according to any one of [1] to [6].
[8]
The one or more computer processors identify a second main display region in the virtual space in a second scene,
the specific object is arranged outside the second main display area in the second scene;
The application processing system according to [7].
[9]
The one or more computer processors configure the virtual space so that the virtual space includes the virtual character that reacts according to the first action or the second action, [1] to [8]. The application processing system according to any one of Claims 1 to 3.
[10]
One or more computer processors execute computer-readable instructions to output an image of a virtual space including the virtual character to a head-mounted display worn on the user's head, thereby providing the user with the virtual character. An application processing method that provides an interaction of
A first action, which is an action to be performed on the virtual character and is associated with movement of the user's head in the virtual space, based on one or more pieces of detection information detected by the head-mounted display. a step of identifying;
an action performed on the virtual character based on the one or more pieces of detection information in response to the selection of a specific object contained in the virtual space, the action being performed on the user's head in the virtual space; identifying a second action not associated with body movement;
A game processing method comprising:
[11]
By being executed by one or more computer processors, an image of a virtual space including a virtual character is output to a head-mounted display worn on the user's head, thereby providing the user with an interaction with the virtual character. An application processing program,
to the one or more computer processors;
A first action, which is an action performed on the virtual character and is associated with movement of the user's head in the virtual space, based on one or more pieces of detection information detected by the head-mounted display. a step of identifying;
an action performed on the virtual character based on the one or more pieces of detection information in response to the selection of a specific object contained in the virtual space, the action being performed on the user's head in the virtual space; identifying a second action not associated with body movement;
, the application processing program that causes the .

1,101 application processing system

Claims (9)

One or more computer processors for processing an application that provides the user with interaction with the virtual character by outputting an image of a virtual space containing the virtual character to a head-mounted display worn on the user's head An application processing system,
The one or more computer processors, by executing computer readable instructions,
detecting movement of the user's head in the real world based on one or more pieces of detection information detected by the head-mounted display; is specified as the first action performed by the user on the virtual character in
In response to determining that a specific object included in the image of the virtual space output to the head-mounted display has been selected based on the one or more pieces of detection information, a line is executed for the virtual character. a menu item corresponding to a second action in the virtual space which is an action to be performed and which is not associated with the movement of the user's head, wherein the first menu item action and the first menu item are included in the second action; By displaying a menu item including a first menu item and a second menu item corresponding to a second menu item action different from the item action in a selectable manner based on the one or more pieces of detection information, the second action is performed. is identifiable based on the one or more detected information,
When it is determined that the first menu item or the second menu item has been selected based on the one or more detection information, the first menu item action or the second menu item corresponding to the selected first menu item identifying the second menu item action corresponding to the second menu item as the second action;
setting one or more character gaze positions on the virtual character in response to determining that the first menu item or the second menu item has been selected after the selection of the specific object;
when it is determined that at least one of the one or more character gaze positions set on the virtual character has been selected after the selection of the first menu item based on the one or more detection information; One menu item action is specified as the second action, and it is determined that at least one of the one or more character gaze positions is selected after the second menu item is selected based on the one or more detection information. identifying the second menu item action as the second action if
Application processing system. the second action is an action in the virtual space associated with the movement of the user's hand in the virtual space;
The application processing system of claim 1. the second action is an action in the virtual space associated with use of a tool by the user in the virtual space;
The application processing system of claim 1. the one or more character gaze positions include a first character gaze position and a second character gaze position;
the second action includes a first gaze position action and a second gaze position action;
When it is determined that the first character gaze position has been selected, the first gaze position action is specified as the second action, and when it is determined that the second character gaze position has been selected, the second character gaze position action is specified. identifying a gaze position action as the second action;
The application processing system of claim 1. the image of the virtual space is an image in a display area centered on the reference line of sight of the user;
The virtual space includes a first virtual space defined for a first scene executed in the application and a second virtual space defined for a second scene executed after the first scene in the application. and including
The one or more computer processors set a first main display area at a specific position in the first virtual space of the first scene ,
wherein the specific object is arranged outside the first main display area in the first scene;
The application processing system according to any one of claims 1 to 4. The one or more computer processors set a second main display area at a specific position in the second virtual space of the second scene ,
the specific object is arranged outside the second main display area in the second scene;
The application processing system according to claim 5. 7. The one or more computer processors configure the virtual space so that the virtual space includes the virtual character that reacts according to the first action or the second action. The application processing system according to any one of Claims 1 to 3. One or more computer processors execute computer-readable instructions to output an image of a virtual space including the virtual character to a head-mounted display worn on the user's head, thereby providing the user with the virtual character. An application processing method for processing an application that provides an interaction of
detecting movement of the user's head in the real world based on one or more pieces of detection information detected by the head-mounted display; a step of specifying as a first action performed by the user on the virtual character in
performed on the virtual character in response to determining that a specific object included in the image of the virtual space output to the head-mounted display has been selected based on the one or more pieces of detection information A menu item corresponding to a second action in the virtual space that is an action and is not associated with movement of the user's head, wherein the first menu item action and the first menu item are included in the second action. By displaying menu items including a first menu item and a second menu item respectively corresponding to a second menu item action different from the action in a selectable manner based on the one or more pieces of detection information, the second action is performed. identifiable based on the one or more pieces of detected information;
setting one or more character gaze positions on the virtual character in response to determining that the first menu item or the second menu item has been selected based on the one or more detection information;
when it is determined that at least one of the one or more character gaze positions set on the virtual character has been selected after the selection of the first menu item based on the one or more detection information; One menu item action is specified as the second action, and it is determined that at least one of the one or more character gaze positions is selected after the second menu item is selected based on the one or more detection information. then identifying the second menu item action as the second action;
A game processing method comprising: By being executed by one or more computer processors, an image of a virtual space including a virtual character is output to a head-mounted display worn on the user's head, thereby providing the user with interaction with the virtual character. An application processing program for processing an application,
to the one or more computer processors;
detecting movement of the user's head in the real world based on one or more pieces of detection information detected by the head-mounted display; a step of specifying as a first action performed by the user on the virtual character in
In response to determining that a specific object included in the image of the virtual space output to the head-mounted display has been selected based on the one or more pieces of detection information, a line is executed for the virtual character. a menu item corresponding to a second action in the virtual space which is an action to be performed and which is not associated with the movement of the user's head, wherein the first menu item action and the first menu item are included in the second action; By displaying a menu item including a first menu item and a second menu item corresponding to a second menu item action different from the item action in a selectable manner based on the one or more pieces of detection information, the second action is performed. is identifiable based on the one or more detected information;
setting one or more character gaze positions on the virtual character in response to determining that the first menu item or the second menu item has been selected based on the one or more detection information;
when it is determined that at least one of the one or more character gaze positions set on the virtual character has been selected after the selection of the first menu item based on the one or more detection information; One menu item action is specified as the second action, and it is determined that at least one of the one or more character gaze positions is selected after the second menu item is selected based on the one or more detection information. then identifying the second menu item action as the second action;
The application processing program that causes the .

Images