JP7325833B2 - Game program, game processing method, and game device - Google Patents

Description

The present invention relates to a game program, a game processing method, and a game device.

Among music games, there is a dance game that detects the movement of the user's body and evaluates the quality of the dance. For example, Patent Literature 1 discloses that a user (player) guides and displays a trajectory and timing to be drawn by moving hands and feet in time with music on a game screen displayed facing the user, and the guide display is displayed. Also disclosed is a dance game in which users move their hands and feet. This dance game can be played, for example, on a home-use game machine.

Similarly, Patent Document 2 discloses a dance game in which the user steps on an operation panel arranged in real space in accordance with instructions displayed on the game screen in time with music. there is This dance game requires an operation panel to be installed at the user's feet for determining the position of the user's foot in the real space, and is configured as a so-called arcade game installed in amusement facilities such as game arcades. For example.

JP 2012-196286 A JP 2016-193006 A

However, in the game described in Patent Document 1 mentioned above, although the game screen can guide what kind of body movement (trajectory) the user should make and at what timing, the position where the user should move in the real space (for example, , foot stepping position) is not suitable for games. For example, if the game described in Patent Document 2 mentioned above is to be realized with a simple configuration like a home-use game machine without installing an operation panel at the user's feet, the user will not be able to find any position in the real space. It was sometimes difficult to play intuitively because it was difficult to understand if you should move your feet.

SUMMARY OF THE INVENTION An object of some aspects of the present invention is to provide a game program, a game processing method, and a game device that guide the user to what actions should be performed so that a more intuitive play is possible with a simple configuration. be one.

Another object of the present invention is to provide a game program, a game processing method, and a game device that can achieve the effects described in the embodiments described later.

In order to solve the above-described problems, one aspect of the present invention provides a video output device that outputs a video that can be visually recognized by the user and that can visually recognize the real space by wearing it on the user's head. a step of obtaining a captured image of the real space in a computer that executes possible game processing; a step of generating a virtual space corresponding to the real space from the captured image; arranging a pointing object that instructs the action of the user at a position based on a reference position corresponding to the user so that the user can see the pointing object; detecting motion of at least a part of the user's body from the captured image; and determining the detected motion in timing and position based on the pointing object placed in the virtual space. and a step of evaluating based on.

Further, according to one aspect of the present invention, a video output device that is worn on the user's head outputs a video that can be visually recognized by the user and that can visually recognize the real space. A game processing method executed by a computer, comprising the steps of: obtaining a captured image of the real space; generating a virtual space corresponding to the real space from the captured image; arranging, at a position based on a reference position corresponding to the user, a pointing object that instructs the action of the user so as to be visible to the user; detecting motion of at least a part of the user's body from the captured image; and timing the detected motion based on the pointing object placed in the virtual space. and evaluating based on the position.

Further, according to one aspect of the present invention, a video output device that is worn on the user's head outputs a video that can be visually recognized by the user and that can visually recognize the real space. a game device comprising: an acquisition unit that acquires a captured image obtained by imaging the real space; a generation unit that generates a virtual space corresponding to the real space from the captured image acquired by the acquisition unit; an arrangement unit for arranging an instruction object that instructs the user's action at a position based on a reference position corresponding to the user in the virtual space generated by the unit so that the instruction object can be visually recognized by the user; a display control unit that displays the arranged virtual space in association with the real space; a detection unit that detects motion of at least a part of the user's body from the captured image acquired by the acquisition unit; and an evaluation unit that evaluates the action detected by the detection unit based on timing and position based on the pointing object placed in the virtual space.

FIG. 4 is a diagram showing an overview of game processing by the game device according to the first embodiment; 4A and 4B are diagrams showing definitions of spatial coordinates in a virtual space according to the first embodiment; FIG. 1 is a block diagram showing an example of the hardware configuration of a game device according to the first embodiment; FIG. 1 is a block diagram showing an example of a functional configuration of a game device according to a first embodiment; FIG. 4 is a flowchart showing an example of pointing object placement processing according to the first embodiment; 4 is a flowchart showing an example of pointing object display processing according to the first embodiment; 4 is a flowchart showing an example of play evaluation processing according to the first embodiment; FIG. 11 is a diagram showing an overview of game processing by the game device according to the second embodiment; FIG. 10 is a diagram showing the definition of spatial coordinates in a virtual space and the position of a user image according to the second embodiment; The block diagram which shows an example of the functional structure of the game device which concerns on 2nd Embodiment. FIG. 11 is a flowchart showing an example of pointing object placement processing according to the second embodiment; FIG. FIG. 11 is a block diagram showing an example of a hardware configuration of a game system according to a third embodiment; FIG. FIG. 11 is a block diagram showing an example of the functional configuration of a game device according to a third embodiment; The figure which shows the outline|summary of the game processing by the game device which concerns on 4th Embodiment. FIG. 12 is a block diagram showing an example of the hardware configuration of a game device according to the fourth embodiment; FIG. The block diagram which shows an example of the functional structure of the game device which concerns on 4th Embodiment. FIG. 14 is a flowchart showing an example of pointing object placement processing according to the fourth embodiment; FIG. 14 is a flowchart showing an example of pointing object display processing according to the fourth embodiment; 14 is a flowchart showing an example of play evaluation processing according to the fourth embodiment;

An embodiment of the present invention will be described below with reference to the drawings.
[First Embodiment]
First, a first embodiment of the present invention will be described.

[Overview of game device]
First, an overview of an example of game processing executed by the game device according to the present embodiment will be described. The game device according to this embodiment can typically be a home-use game machine, but it may also be used in amusement facilities such as game arcades.

FIG. 1 is a diagram showing an overview of game processing by the game device according to this embodiment. This diagram shows a bird's-eye view of a play situation in which the user U plays a dance game (an example of a music game) using the game device 10 . The game device 10 is configured to include a video output device. The image output device may display images on a display, or may project images. For example, the game device 10 is configured as an HMD (Head Mounted Display) that, when worn on the user's head, outputs an image so that the user can visually recognize it, and also allows the user to visually recognize the real space.

In the illustrated example of the dance game, the user U moves at least part of the body according to the timing and position of the pointing object displayed on the HMD in time with the music. The instruction object is an object displayed to guide the user U to instruct the timing and position to operate in the real space. In this embodiment, the user can intuitively play by displaying the pointing object arranged in the virtual space on the HMD in association with the real space.

For example, the game device 10 is configured as an HMD (a so-called optical transmission (optical see-through) HMD) that allows optical recognition of real space. The game device 10 displays a pointing object arranged in a virtual space on a transmissive display positioned in front of the user's eyes while being worn on the user's head. As a result, the user can visually recognize an image in which the pointing object displayed on the display is superimposed on the real space visible through the display.

Note that the game device 10 may be configured as a retinal projection type optically transmissive HMD. In the case of the retinal projection type, the game device 10 has a video projection device instead of a display that directly projects a video onto the user's retina. A pointing object placed in the user's virtual space is visually displayed by being directly projected onto the retina.

Further, the game device 10 may be configured as an HMD (so-called video see-through type HMD) that displays in real time an image obtained by imaging the real space. In this case, the game device 10 displays a real-time image of the real space on a display positioned in front of the user's head while being worn on the user's head, and also displays the pointing object arranged in the virtual space as a real-time image. It is superimposed and displayed.

The game device 10 is worn on the head of the user U, and generates a virtual space from a captured image obtained by capturing the line-of-sight direction of the user U in the real space. For example, the virtual space is defined in a three-dimensional coordinate space of XYZ by mutually orthogonal X-axis and Y-axis parallel to the floor (plane) and vertical Z-axis orthogonal to the floor (plane). The generated virtual space includes positions corresponding to at least some of the objects (eg, user U, floor, walls, etc.) in the real space. In the following description, as the direction of the Z-axis, the direction toward the ceiling is also referred to as the upward direction, and the direction toward the floor surface is also referred to as the downward direction.

The game device 10 uses the position of the user U in the virtual space as a reference position, and arranges an instruction object that instructs the user's action at a position based on the reference position (for example, a predetermined position around the reference position). . For example, the pointing object includes a judgment object and a moving object. A determination object is a pointing object placed at a determination position that serves as a determination reference when evaluating a user's motion. For example, the determination object is a position (height) corresponding to the floor surface in the Z coordinate in the virtual space, and a reference position (position of the user U) in the XY coordinate (for example, a range reached by the user U taking a step). ). In the illustrated example, the determination object HF is placed in front of the reference position (the position of the user U), the determination object HB is placed behind, the determination object HR is placed to the right, and the determination object HL is placed to the left. Here, the reference position (the position of the user U) and the forward, backward, rightward, and leftward directions with respect to the reference position are the directions initialized at the start of the dance game, and the direction of the user U during the play. is fixed even if is changed.

The moving object appears from the ceiling side in the Z coordinate in the virtual space and gradually moves downward toward the determination object (determination position) arranged at the position (height) corresponding to the floor surface. The appearance position may be preset based on, for example, the position of the user U's head (the position of the game device 10), or may change according to a predetermined rule. The moving object NF is a moving object that moves toward the determination object HF (the determination position of the moving object NF). The moving object NB is a moving object that moves toward the determination object HB (the determination position of the moving object NB). The moving object NR is a moving object that moves toward the determination object HR (the determination position of the moving object NR). The moving object NL is a moving object that moves toward the determination object HL (the determination position of the moving object NL).

The timing and position at which each moving object gradually moves and reaches the respective determination object (determination position) is the timing and position to be operated by the user U. For example, the moving object NF has reached the determination object HF. The user is requested to step on the determination object HF with his/her foot at the appropriate timing. The user's action is evaluated based on the timing and position at which the moving object reaches the determination object, and the score is updated according to the evaluation. For example, if it is determined that the timing and position of the moving object reaching the determination object match the timing and position of the user's motion, the score is added, and if it is determined that they do not match, the score is not added. For example, whether or not this timing and position match is determined by determining whether the moving object reaches the position within a predetermined time corresponding to the timing when the moving object reaches the determination object (for example, within 0.5 seconds before or after the arrival timing). It is determined whether or not the user has stepped on at least part of the corresponding determination area (for example, the area of the determination object HR). Note that the score to be added may vary depending on the degree of coincidence between the timing and position at which the moving object reaches the determination object and the timing and position of the user's motion.

Note that FIG. 1 shows the correspondence relationship between the real space in which the user U is included and the virtual space in which the pointing object is included. is different. Each pointing object does not exist in the real space, but only in the virtual space, and can be visually recognized through the game device 10 . A pointing object that the user U can actually see while playing is present within a field of view (Fov) that can be seen through the display portion of the game device 10 . By displaying the pointing object included in the field of view on the game device 10 (HMD), the pointing object is superimposed on the real space and can be visually recognized by the user U. The game device 10 also displays display information related to the game (score, information about music to be played, etc.) other than the instruction object.

FIG. 2 is a diagram showing the definition of spatial coordinates in the virtual space according to this embodiment. As described above, in this embodiment, the vertical axis is the Z-axis, and the X-axis and the Y-axis are the horizontal planes orthogonal to the Z-axis and perpendicular to each other. Further, in initialization at the start of play of this dance game, a reference position K1 (an example of a first reference position based on the position of the game device 10) corresponding to the position of the user U is defined as the coordinate origin, and the X axis is defined as It is defined as the axis of the user U's gaze direction. During play, the reference position K1 (coordinate origin), X-axis, Y-axis and Z-axis are fixed. A change in the rotational direction about the Z axis is also called a change in the yaw direction (horizontal direction), and a change in the rotational direction about the Y axis is also called a change in the pitch direction (vertical direction). is also referred to as a change in the roll direction.

When the direction of the head of the user U wearing the game apparatus 10 changes, the change is detected as a change in the rotation direction of each axis (yaw direction, pitch direction, roll direction) using a built-in acceleration sensor or the like. The game device 10 changes the field of view (Fov) shown in FIG. 1 based on the detected change in the rotational direction of each axis, and changes the display of the pointing object included in the virtual space. Thereby, even if the orientation of the user U's head changes, the game device 10 can display the pointing object included in the virtual space on the display in accordance with the change in the field of view.
A change in the yaw direction is sometimes called a change in the horizontal direction, and a change in the pitch direction is sometimes called a change in the vertical direction.

Note that the illustrated reference position K1 is an example and is not limited to this position. Further, although the reference position K1 is defined as the coordinate origin of the spatial coordinates, the coordinate origin may be defined at another position.

[Hardware Configuration of Game Device 10]
Next, an overview of the hardware configuration of the game device 10 according to this embodiment will be described.
FIG. 3 is a block diagram showing an example of the hardware configuration of the game device 10 according to this embodiment. The game device 10, as an optically transmissive HMD, includes an imaging unit 11, a display unit 12, a sensor 13, a storage unit 14, a CPU (Central Processing Unit) 15, a communication unit 16, and a sound output unit 17. is composed of

The image capturing unit 11 is a camera that captures an image of the line-of-sight direction of the user U who uses the game device 10 (HMD) while wearing it on the head. That is, the imaging unit 11 is provided in the game device 10 (HMD) so that the optical axis corresponds to the line-of-sight direction when the imaging unit 11 is worn on the head. The imaging unit 11 may be a monocular camera or a dual camera. The imaging unit 11 outputs a captured image.

The display unit 12 is, for example, a transmissive display in an optical transmissive HMD. For example, the display unit 12 displays at least a pointing object. The display unit 12 may be configured to include two displays for the right eye and the left eye, or may be configured to include one display that can be viewed with both eyes regardless of whether the display is for the right eye or the left eye. If the game device 10 is a retinal projection type optically transmissive HMD, the display unit 12 is an image projection device that directly projects an image onto the user's retina.

Note that if the game device 10 is a video transmissive HMD, the display unit 12 is a non-transmissive display that makes the real space optically invisible.

The sensor 13 is a sensor that outputs a detection signal regarding the direction of the game device 10 . For example, the sensor 13 is a gyro sensor that detects the angle, angular velocity, angular acceleration, etc. of an object. Note that the sensor 13 may be a sensor that detects a change in direction, or may be a sensor that detects the direction itself. For example, the sensor 13 may include an acceleration sensor, an inclination sensor, a geomagnetic sensor, etc. instead of or in addition to the gyro sensor.

The storage unit 14 includes, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory), a ROM (Read-Only Memory), a Flash ROM, a RAM (Random Access Memory), etc., and stores the dance game program, data, and generated data. Stores virtual space data and the like.

The CPU 15 functions as a control center that controls each part of the game device 10 . For example, the CPU 15 executes game processing by executing a game program stored in the storage unit 14, and generates a virtual space corresponding to the real space from the captured image as described with reference to FIG. a process of placing a pointing object in the generated virtual space; and a process of detecting a user's motion and evaluating it based on the timing and position of the pointing object.

The communication unit 16 includes, for example, a communication device that performs wireless communication such as Bluetooth (registered trademark) and Wi-Fi (registered trademark). The communication unit 16 may include a digital input/output port such as a USB (Universal Serial Bus), a video input/output port, and the like.

The sound output unit 17 outputs the performance sound of the play music of the dance game, the effect sound of the game, and the like. For example, the sound output unit 17 may include a speaker, an earphone, a headphone, or a terminal connectable thereto. Note that the sound output unit 17 may output various sounds to external speakers, earphones, headphones, etc. via wireless communication such as Bluetooth (registered trademark).

The hardware components included in the game device 10 described above are connected to communicate with each other via a bus.

[Functional Configuration of Game Device 10]
Next, with reference to FIG. 4, the functional configuration of the game device 10 will be described.
FIG. 4 is a block diagram showing an example of the functional configuration of the game device 10 according to this embodiment. The illustrated game device 10 includes a control unit 150 as a functional configuration realized by the CPU 15 executing a program stored in the storage unit 14 . The control unit 150 executes the dance game processing described with reference to FIGS. 1 and 2 . For example, the control unit 150 includes an image acquisition unit 151, a virtual space generation unit 152, an object placement unit 154, a gaze direction detection unit 155, a display control unit 156, a motion detection unit 157, and an evaluation unit 158. I have.

The video acquisition unit 151 (an example of the acquisition unit) acquires the captured video of the real space captured by the imaging unit 11 . For example, the game device 10 instructs the user U to look in a predetermined direction (for example, to look up, down, left, and right) before starting to play the dance game. The game device 10 causes the display unit 12 to display this instruction, for example. Thereby, the video acquisition unit 151 acquires a captured video in which the surroundings of the user U in the real space captured by the imaging unit 11 are captured.

The virtual space generation unit 152 (an example of the generation unit) generates a virtual space corresponding to the real space from the captured image acquired by the image acquisition unit 151 . For example, the virtual space generation unit 152 detects the position of an object (floor, wall, etc.) that exists in the real space from the acquired captured image, and includes position information of at least part of the detected object (floor, wall, etc.). 3D coordinate space data is generated as virtual space data. As an example, a reference position K1 (see FIG. 2) corresponding to the user U based on the position of the game device 10 itself worn on the head of the user U is defined as the coordinate origin of the virtual space (three-dimensional coordinate space). be done. The virtual space generation unit 152 includes position information corresponding to objects (floors, walls, etc.) existing in the real space in a virtual space (three-dimensional coordinate space) with the reference position K1 corresponding to the user U as the coordinate origin. Generate virtual space data. The virtual space generation unit 152 causes the storage unit 14 to store the generated virtual space data.

Here, any known technique can be applied to the detection method for detecting the position of an object (floor, wall, etc.) existing in the real space from the captured image. For example, if the imaging unit 11 is a dual camera (stereo camera), the position of an object (floor, wall, etc.) may be detected by analyzing captured images using the parallax of the left and right cameras. In addition, when the imaging unit 11 is a monocular camera, it is possible to perform detection using parallax in the same way as a dual camera by using images captured from two locations with the monocular camera shifted by a prescribed distance. Also, in place of or in addition to image analysis, laser light, sound waves, or the like may be used to detect the position of an object (floor, wall, etc.) existing in the real space.

The object arranging unit 154 (an example of the arranging unit) arranges an instruction object that instructs the action of the user U at a position based on the reference position K1 corresponding to the user U in the virtual space so that the user U can see the instruction object. Specifically, the object placement unit 154 places determination objects (see determination objects HF, HB, HR, and HL in FIG. 1) at determination positions in the virtual space corresponding to the position of the floor. Further, the object placement unit 154 places moving objects (see moving objects NF, NB, NR, and NL in FIG. 1) at the appearance positions in the virtual space at preset timings in accordance with music, Move (change the placement position) toward it. The object placement unit 154 updates the virtual space data stored in the storage unit 14 based on the coordinate information of the placement position in the virtual space when placing the pointing object (the determination object and the moving object).

The line-of-sight direction detection unit 155 detects the direction of the game device 10 , that is, the line-of-sight direction of the user U based on the detection signal output from the sensor 13 . Note that the line-of-sight direction detection unit 155 may detect the direction of the game device 10 , that is, the line-of-sight direction of the user U by analyzing the captured image of the real space captured by the imaging unit 11 . For example, the line-of-sight direction detection unit 155 detects the position and inclination of an object or an edge of an object by analyzing a captured image, and detects the orientation of the game device 10, that is, the line-of-sight direction of the user U, based on the detection result. may In addition, by detecting the position and inclination of the object or the edge of the object in each frame of the captured image, the difference in the position and inclination of the object or the edge of the object between each frame is detected, and based on the detection result The orientation of the game device 10, that is, the change in the line-of-sight direction of the user U may be detected. Note that the line-of-sight direction detection unit 155 may detect the orientation of the game device 10, that is, the line-of-sight direction of the user U, based on both the detection signal output from the sensor 13 and the analysis of the captured image of the real space. .

The display control unit 156 refers to the virtual space data stored in the storage unit 14 and causes the display unit 12 to display the virtual space in which at least the pointing object is arranged in association with the real space. Here, associating the virtual space with the real space includes associating the coordinates of the virtual space generated based on the real space with the coordinates of the real space. When displaying the virtual space, the display control unit 156 determines the viewpoint position and line-of-sight direction in the virtual space based on the position and orientation of the game device 10 (HMD) in the real space, that is, the position and orientation of the user U. . For example, the display control unit 156 controls the pointing object arranged in the range of the virtual space corresponding to the range of the field of view (Fov) (range of the real space) determined by the line-of-sight direction of the user U detected by the line-of-sight direction detection unit 155. is displayed on the display unit 12 (see FIG. 1).

The motion detection unit 157 (an example of the detection unit) detects motion of at least part of the body of the user U from the captured image. For example, the motion detection unit 157 detects motions of the user U's feet playing a dance game. Any known technique can be applied as the recognition technique for recognizing at least a part of the body of the user U (that is, the recognition target) from the captured image. For example, the motion detection unit 157 recognizes a recognition target video area from the captured video using recognition target feature information (for example, foot feature information). The motion detection unit 157 detects a recognition target motion (for example, leg motion) by extracting and tracking a recognition target video region from each frame of the captured video.

The evaluation unit 158 evaluates the motion of at least part of the body of the user U detected by the motion detection unit 157 based on the timing and position based on the pointing object arranged in the virtual space. For example, the evaluation unit 158 compares the timing and position at which the moving object reaches the determination object with the timing and position of the motion of the user U's foot (the motion of stepping on the determination object), and evaluates the play by the motion of the user U. do. Based on the comparison result, the evaluation unit 158 adds the score when it can be determined that the timing and position of both match, and does not add the score when it can determine that they do not match.

Note that the evaluation unit 158 may evaluate the play by the action of the user U by comparing the position of the user U's foot at the timing when the moving object reaches the determination object with the position of the determination object.

[Operation of pointing object placement processing]
Next, in the dance game processing executed by the CPU 15 of the game device 10, a pointing object placement process for generating a virtual space and placing a pointing object will be described. FIG. 5 is a flowchart showing an example of pointing object placement processing according to the present embodiment.

First, the CPU 15 acquires a captured image of the real space captured by the imaging unit 11 (step S101). For example, the CPU 15 causes the display unit 12 to display an instruction for the user U to look in a predetermined direction (for example, an instruction to look up, down, left, and right) before starting to play the dance game, so that the surroundings of the user U in the real space are displayed. Acquire the captured image.

Next, the CPU 15 generates a virtual space corresponding to the real space from the captured image acquired in step S101 (step S103). For example, the CPU 15 detects the position of an object (floor, wall, etc.) existing in the real space from the captured image. The CPU 15 creates a three-dimensional coordinate space containing position information of at least a part of a detected object (floor, wall, etc.) in a virtual space (three-dimensional coordinate space) with the reference position K1 corresponding to the user U as the coordinate origin. Generate virtual space data. Then, the CPU 15 causes the storage unit 14 to store the generated virtual space data.

Subsequently, the CPU 15 places the determination objects (the determination objects HF, HB, HR, HL) are placed (step S105). When arranging the judgment object, the CPU 15 adds the position information of the arranged judgment object to the virtual space data stored in the storage unit 14 .

Further, when the play of the dance game is started, the CPU 15 determines whether or not there is a moving object appearance trigger (step S107). Appearance triggers are generated at timings preset according to music. When the CPU 15 determines that there is an appearance trigger in step S107 (YES), the process proceeds to step S109.

In step S109, the CPU 15 arranges the moving object at the appearance position based on the reference position K1 in the virtual space (one or more of the moving objects NF, NB, NR, and NL in FIG. 1), determines the determination position (each position of the determination object corresponding to the moving object). When arranging the moving object, the CPU 15 adds the position information of the arranged moving object to the virtual space data stored in the storage unit 14 . Further, when moving the arranged moving object, the CPU 15 updates the position information of the moving object added to the virtual space data stored in the storage unit 14 . Then, the process proceeds to step S111. On the other hand, when the CPU 15 determines that there is no appearance trigger in step S107 (NO), it proceeds to the process of step S111 without performing the process of step S109.

In step S111, the CPU 15 determines whether or not the moving object has reached the determination position. The CPU 15 erases from the virtual space the moving object that has reached the determination position (YES) in step S111 (step S113). When deleting a moving object from the virtual space, the CPU 15 deletes the position information of the moving object to be deleted from the virtual space data stored in the storage unit 14 .

On the other hand, the CPU 15 continues to gradually move the moving object determined (NO) that it has not reached the determination position in step S111 toward the determination position (step S115). When moving the arranged moving object, the CPU 15 updates the position information of the moving object to be moved among the virtual space data stored in the storage unit 14 .

Next, the CPU 15 determines whether or not the dance game has ended (step S117). For example, the CPU 15 determines that the dance game has ended when the music being played ends. When the CPU 15 determines that the dance game has not ended (NO), the process returns to step S107. On the other hand, when the CPU 15 determines that the dance game has ended (YES), it ends the pointing object placement process.

The order of placement of the determination object and the placement of the moving object that appears first may be the same, the determination object may be placed first, or conversely, the determination object may be placed after (the moving object that appears first may be placed first). until it reaches the judgment position).

[Operation of Pointer Object Display Processing]
Next, in the dance game processing executed by the CPU 15 of the game device 10, a pointing object display process for displaying pointing objects placed in the virtual space will be described. FIG. 6 is a flowchart showing an example of pointing object display processing according to the present embodiment.

The CPU 15 detects the line-of-sight direction of the user U (orientation of the game device 10) based on the detection signal output from the sensor 13 (step S201).

The CPU 15 refers to the virtual space data stored in the storage unit 14, and displays the virtual space corresponding to the range of the field of view (Fov) (range of the real space) based on the line-of-sight direction detected in step S201 on the display unit 12. display. For example, the CPU 15 causes the display unit 12 to display the pointing object (the determination object and the moving object) arranged in the range of the virtual space corresponding to the range of the field of view (Fov) based on the line-of-sight direction (step S203). As a result, the moving object is displayed on the display unit 12 at the timing set in advance according to the music.

Next, the CPU 15 determines whether or not the dance game has ended (step S205). For example, the CPU 15 determines that the dance game has ended when the music being played ends. When the CPU 15 determines that the dance game has not ended (NO), the process returns to step S201. On the other hand, when the CPU 15 determines that the dance game has ended (YES), it ends the pointing object display process.

[Operation of play evaluation process]
Next, in the dance game processing executed by the CPU 15 of the game device 10, the operation of the play evaluation process for evaluating the play by the motion of at least part of the body of the user U will be described. FIG. 7 is a flowchart showing an example of play evaluation processing according to this embodiment.

The CPU 15 acquires a captured image of the real space captured by the imaging unit 11 (step S301). Next, the CPU 15 detects motion of at least a part of the body of the user U from the captured image acquired in step S301 (step S303). For example, the CPU 15 detects the motion of the user U's feet playing a dance game.

Then, the CPU 15 evaluates the motion of at least part of the body (for example, foot) of the user U detected in step S303 based on the timing and position based on the pointing object placed in the virtual space (step S305). ). For example, the CPU 15 compares the timing and position at which the moving object reaches the determination object with the timing and position of the motion of the user U's foot (motion of stepping on the determination object), and evaluates the play by the motion of the user U's foot. do.

Also, the CPU 15 updates the score of the game based on the evaluation result in step S305 (step S307). For example, if the CPU 15 can determine that the timing and position at which the moving object reaches the determination object and the timing and position of the motion of the user U's foot (motion to step on the determination object) match, the score is added. However, if it can be determined that they do not match, the score is not added.

Next, the CPU 15 determines whether or not the dance game has ended (step S309). For example, the CPU 15 determines that the dance game has ended when the music being played ends. When the CPU 15 determines that the dance game has not ended (NO), the process returns to step S301. On the other hand, when the CPU 15 determines that the dance game has ended (YES), it ends the play evaluation process.

[Summary of the first embodiment]
As described above, the game device 10 according to the present embodiment, when worn on the head of the user U, outputs an image that can be visually recognized by the user U and allows the user U to visually recognize the real space. An example of an output device) is used to execute processing of a game that can be played. For example, the game device 10 acquires a captured image of a real space, and generates a virtual space corresponding to the real space from the acquired captured image. Then, the game device 10 arranges an instruction object for instructing the action of the user U at a position based on the reference position K1 corresponding to the user U in the virtual space so that at least the instruction object is arranged. The virtual space is displayed in association with the real space. In addition, the game device 10 detects motion of at least a part of the body of the user U from the captured image, and evaluates the detected motion based on the timing and position of the pointing object placed in the virtual space. do.

As a result, in the game process of evaluating the motion of the user U based on the timing and position of the pointing object that instructs the motion of the user U, the game device 10 attaches the pointing object to the real space by wearing it on the head. Since the user U can visually recognize the information in association with the user U, it is possible to guide the user to what action should be performed with a simple configuration so that the user can play the game more intuitively.

For example, the reference position K1 is the first reference position in the virtual space corresponding to the position of the user U wearing the game device 10 (an example of the video output device), and is the position of the transmissive HMD in the virtual space. based on For example, the reference position K1 is a position in the virtual space corresponding to the position of the user U (the position of the transmissive HMD) in the real space, and is defined as the coordinate origin of the virtual space (three-dimensional coordinate space).

As a result, the game device 10 can display the instruction object in association with the real space based on the position of the user U playing the game, so that the instruction of the action to the user U can be made to feel more realistic and intuitive. play is possible.

In addition, the game device 10 moves the pointing object (eg, moving object) arranged at a predetermined position (appearance position) in the virtual space toward a predetermined determination position (eg, position of the determination object). Then, the game device 10 detects at least a part of the body of the user U (the For example, to evaluate the movement of the foot).

Thereby, the game device 10 can use the captured image to evaluate whether or not the user U has performed the action as instructed.

In the game device 10, the user U can only visually recognize the pointing object within the range of the visual field based on the direction of the user's line of sight. Therefore, the game device 10 may restrict the position where the pointing object is placed to a part of the virtual space according to the orientation of the user U wearing the game device 10 (an example of the video output device). For example, based on the orientation of the user U (reference position K1) at the time of initialization, the game device 10 arranges only the front, right, and left pointing objects, and does not arrange the pointing object behind. good too.

Accordingly, since the game apparatus 10 does not issue an instruction to move outside the range of the user U's field of vision (for example, the rear), the user U may not be aware of the outside of the range of the field of vision (for example, the rear) during play. can be played without Therefore, the game device 10 can prevent the play from becoming too difficult.

Further, when restricting the position where the pointing object is placed to a part of the virtual space according to the orientation of the user U, the game device 10 changes the restricted direction according to the orientation of the user U during play. good too. For example, when the user U is facing forward, the game device 10 arranges only the front, right, and left pointing objects with respect to the user U (reference position K1), and does not arrange pointing objects behind the user U. You may do so. In addition, when the user U faces right, the game device 10 is positioned forward, to the right, and to the left of the user U after facing right (the reference position K1). , front, and rear) may be placed, and no pointing object may be placed behind (to the left before turning to the right). Similarly, when the user U faces leftward or backward, the pointing object may not be placed in the opposite direction (rightward or forward before changing the direction).

As a result, the game device 10 follows the change in the direction of the user U and does not always issue an instruction to operate outside the range of the user U's field of view, so that the difficulty level of the play can be suppressed. .

[Second embodiment]
Next, a second embodiment of the invention will be described.
In the example described in the first embodiment, the pointing object that is actually visible to the user U is the pointing object arranged in the range of the visual field based on the line-of-sight direction of the user U among the pointing objects arranged in the virtual space. Limited. Therefore, for example, when the pointing object exists behind the user U, it may be difficult to recognize it. In terms of gameplay, this difficult element can be used, but on the other hand, there is a concern that it will be difficult for beginners and the like to play. In the first embodiment, it has been described that the difficulty may be suppressed by restricting the position where the pointing object is placed to a part of the virtual space according to the orientation of the user U. In the case of , the types of actions to be instructed to the user U during play are reduced. In addition, when the user U actually plays, it is necessary to play while looking at the user U's feet and the pointing object, which are located below the user U. There is concern that it will become Therefore, in this embodiment, the above concern is resolved by using a mirror.

FIG. 8 is a diagram showing an overview of game processing by the game device according to this embodiment. This figure shows a bird's-eye view of a play situation in which a user U plays a dance game using the game device 10A according to the present embodiment. Similar to FIG. 1, this figure summarizes the correspondence relationship between the real space in which the user U is included and the virtual space in which the pointing object is included. It is different from the normal play screen.

In the illustrated example, the user U is playing a dance game facing the mirror MR. Around the user U in the virtual space, similarly to FIG. 1, pointing objects (determination objects and moving objects) are arranged. Furthermore, the user U is reflected in the mirror MR facing the user U. Here, the virtual image of the user U reflected in the mirror MR is referred to as "user image UK". The game device 10A detects the user image UK corresponding to the user U from the captured image captured in the direction of the mirror MR, and the detected user image UK is displayed as if the pointing object arranged around the user U is reflected in the mirror MR. A pointing object is also placed around the image UK.

FIG. 9 is a diagram showing the definition of the spatial coordinates of the virtual space and the position of the user image UK according to this embodiment. This diagram is a diagram obtained by adding the position of the user image UK detected from the captured image to the definition of the spatial coordinates of the virtual space shown in FIG. A reference position K2 (an example of a second reference position) corresponding to the position of the user image UK in the virtual space is positioned relative to a reference position K1 (for example, coordinate origin) corresponding to the position of the user U in the X-axis direction (line-of-sight direction). is detected at the front (rear) position of the mirror MR in the position. For example, if the position where the mirror surface of the mirror MR intersects the X-axis is the mirror surface position M1, the distance from the reference position K1 to the mirror surface position M1 and the distance from the mirror surface position M1 to the reference position K2 are the same in the X-axis direction. A reference position K2 is detected at such a position. The reference position K2 may be defined as any position, such as a position corresponding to the center of the head of the user image UK or a position corresponding to the center of gravity of the user image UK.

Returning to FIG. 8, the game device 10A detects the image area (contour) and distance of the user image UK from the captured image, and separates the user image UK from the virtual space from the reference position K1 corresponding to the position of the user U. A reference position K2 corresponding to the position is detected. Then, the game device 10A arranges the pointing object around the reference position K1 and the reference position K2 based on the reference position K1 and the reference position K2, respectively. At this time, since the user image UK is a virtual image of the user U reflected in the mirror MR, the front and back direction of the user U is reversed. Therefore, in the game device 10A, the pointing object placed around the reference position K2 (the position of the user image UK) is oriented forward and backward (spatial coordinates) from the pointing object placed around the reference position K1 (the position of the user U). ) are reversed.

For example, assuming that the direction from the reference position K1 to the reference position K2 on the X-axis is the positive direction, the determination object HF and the moving object NF placed in front of the reference position K1 (the position of the user U) are located at the reference position K1. is arranged in the positive direction of the X-axis with respect to On the other hand, the determination object HF' and the moving object NF' placed in front of the reference position K2 (the position of the user image UK) are placed in the negative direction of the X axis with respect to the reference position K2. Also, the determination object HB and the moving object NB, which are arranged behind the reference position K1 (the position of the user U), are arranged in the negative direction of the X axis with respect to the reference position K1. On the other hand, the determination object HB' and the moving object NB', which are arranged behind the reference position K2 (the position of the user image UK), are arranged in the positive direction of the X axis with respect to the reference position K2.

On the other hand, the determination object HR and the moving object NR that are placed on the right side of the reference position K1 (the position of the user U), and the determination object that is placed on the right side of the reference position K2 (the position of the user image UK). HR' and moving object NR' are arranged in the same direction (for example, positive direction) of the Y-axis with respect to their respective reference positions. Also, the determination object HL and the moving object NL are arranged to the left of the reference position K1 (the position of the user U), and the judgment object is arranged to the left of the reference position K2 (the position of the user image UK). HL' and moving object NL' are arranged in the same direction (for example, negative direction) of the Y-axis with respect to their respective reference positions. Further, the upward and downward positional relationships between the pointing object arranged with respect to the reference position K1 and the pointing object arranged with respect to the reference position K2 are also the same.

[Configuration of game device 10A]
The game device 10A according to the present embodiment may be a device including an optical transmissive HMD or a device including a video transmissive HMD, like the game device 10 described in the first embodiment. good. Here, as in the first embodiment, the game device 10A will be described as an optically transmissive HMD. Since the hardware configuration of the game device 10A is the same as the configuration example shown in FIG. 3, the description thereof is omitted.

FIG. 10 is a block diagram showing an example of the functional configuration of the game device 10A according to this embodiment. The illustrated game device 10A includes a control section 150A as a functional configuration realized by the CPU 15 executing a program stored in the storage section 14 . The control unit 150A includes a video acquisition unit 151, a virtual space generation unit 152, a user image detection unit 153A, an object placement unit 154A, a gaze direction detection unit 155, a display control unit 156, a motion detection unit 157, and an evaluation unit 158 . In this figure, the same reference numerals are given to the configurations corresponding to the respective parts in FIG. 4, and the description thereof will be omitted as appropriate. The functional configuration of the game device 10A differs from that of the game device 10 shown in FIG. 4 in that a user image detection unit 153A for detecting a reference position corresponding to the user image UK reflected in the mirror MR is added. differ mainly from

The user image detection unit 153A detects a user image UK (an example of an image) corresponding to the user U from the captured image acquired by the image acquisition unit 151 . For example, the user image UK, which is a virtual image of the user U reflected in the mirror MR that faces the user U, is detected. This detection needs to recognize that the user image UK is a virtual image of the user U playing the dance game. As a recognition method, for example, an identifiable marker (mark, sign, etc.) is attached to the body of the user U or the game device 10A (HMD) worn on the head of the user U, and the user image detection unit 153A It may be recognized that the marker is a virtual image of the user U by detecting the marker from the captured image. In addition, by instructing the user U to perform a specific action (for example, raising and lowering the right hand), the user image detection unit 153A detects a person who performs the action according to the instruction from the captured image. You may recognize that it is the user's U virtual image.

The virtual space generation unit 152 generates three-dimensional coordinate space data including the position information of the user image UK in addition to the position information of at least a part of the object (floor, wall, etc.) detected from the captured image as virtual space data. do. For example, the virtual space generator 152 detects the position of an object (floor, wall, etc.) existing in the real space from the captured image. In addition, the virtual space generator 152 detects the position (reference position K2) of the user image UK detected by the user image detector 153A. A method of detecting the position of the user image UK may be a detection method using parallax of a camera (imaging unit), like the method of detecting the position of an object (floor, wall, etc.) existing in the real space described above. Then, the virtual space generation unit 152 generates data in a three-dimensional coordinate space including position information of at least part of the detected object (floor, wall, etc.) and position information of the reference position K2. Generate as spatial data. As an example, the coordinate origin of the virtual space (three-dimensional coordinate space) is assumed to be the reference position K1 corresponding to the user U, as in the first embodiment. The virtual space generation unit 152 causes the storage unit 14 to store the generated virtual space data.

The object placement unit 154A places the pointing object at a position based on the reference position K1 corresponding to the user U in the virtual space, and also places the pointing object at a position based on the reference position K2 corresponding to the user image UK ( 8 and 9). In addition, the object placement unit 154A reverses the front-rear direction with respect to the reference position K2 when placing the pointing object at the position based on the reference position K2 in the virtual space.

Here, the object placement unit 154A determines whether the detected user image UK is the image reflected in the mirror MR by instructing the above-described specific action (for example, raising and lowering the right hand). A person performing the action may be detected from a captured image and determined. In addition, the object placement unit 154A selects a previously selectable mirror mode (a mode in which the user is played while looking at the image reflected in the mirror MR), so that the detected user image UK is reflected in the mirror MR. It may be determined that the image is an image that

For example, when the line-of-sight direction of the user U is the direction of the mirror MR, the display control unit 156 causes the display unit 12 to display the pointing object arranged in the range of the virtual space corresponding to the range of the visual field in the direction of the mirror MR. . That is, the display control unit 156 can display the pointing object arranged at the position based on the reference position K2 corresponding to the user image UK reflected in the mirror MR so that the user U can view and visually recognize the pointing object. .

The motion detection unit 157 detects the motion of at least part of the body of the user U by detecting the motion of at least part of the body of the user image UK reflected in the mirror MR from the captured image.

The evaluation unit 158 converts the motion of at least a part of the body of the user image UK (the user image UK reflected in the mirror MR) detected by the motion detection unit 157 to a position based on the reference position K2 corresponding to the user image UK. Evaluate using the pointing object that is placed. Specifically, the evaluation unit 158 evaluates the movement of at least a part of the body of the user image UK (the user image UK reflected in the mirror MR) at a position based on the user image UK reflected in the mirror MR. Evaluate based on timing and position based on the pointing object that is present. That is, the user U can play while looking at the direction of the mirror MR without looking at the feet of the user U and the pointing object existing below.

In the example shown in FIG. 8, the pointing object is arranged at both the position based on the reference position K1 corresponding to the user U and the position based on the reference position K2 corresponding to the user image UK, but the present invention is limited to this. not a thing For example, when the pointing object is placed at the position based on the reference position K2 corresponding to the user image UK, the object placing unit 154A does not have to place the pointing object at the position based on the reference position K1 corresponding to the user U. good too. That is, when the pointing object is displayed at the position based on the reference position K2, the pointing object at the position based on the reference position K1 may not be displayed. As a result, the pointing object displayed on the mirror MR is not hidden by the pointing object displayed around the user U, and the visibility of the pointing object can be improved.

Further, when the pointing object is placed at the position based on the reference position K2, the object placement unit 154A reduces the visibility by making the pointing object placed at the position based on the reference position K1 translucent or reducing it. It may be displayed in an inconspicuous manner. Note that the display control unit 156 may perform the process of changing the display mode of the pointing object.

Further, the object placement unit 154A (or the display control unit 156) hides or translucents the pointing object around the user U only when the mirror MR is within the field of view of the user U, and the mirror MR When it is out of the range of the field of view, the pointing objects around the user U may be displayed as usual. As a result, even when the mirror MR is out of the field of view (for example, when the user U faces the direction opposite to the direction of the mirror MR), the pointing object can be visually recognized.

[Operation of pointing object placement processing]
Next, in the dance game processing executed by the CPU 15 of the game device 10A, a pointing object placement process for generating a virtual space and placing a pointing object will be described. Here, the pointing object arrangement processing corresponding to the user U (reference position K1) is the same as the processing shown in FIG. A pointing object arrangement process for arranging a pointing object will be described. FIG. 11 is a flowchart showing an example of pointing object placement processing according to the present embodiment.

First, the CPU 15 acquires a captured image of the real space captured by the imaging unit 11 (step S401). For example, the CPU 15 acquires a captured image including the user image UK (see FIG. 8) reflected in the mirror MR in the line-of-sight direction of the user U playing the dance game.

Next, the CPU 15 detects a virtual image (user image UK) of the user U playing the dance game from the imaged video acquired in step S401 (step S403).

Also, the CPU 15 generates a virtual space corresponding to the real space from the captured image acquired in step S401 (step S405). For example, the CPU 15 detects the position of an object (floor, wall, etc.) existing in the real space from the captured image and the position (reference position K2) of the user image UK detected in step S403, and detects the detected object (floor, wall, etc.). , walls, etc.) and the position information of the reference position K2 are generated as virtual space data. As an example, the CPU 15 stores positional information of at least a part of a detected object (floor, wall, etc.) and a reference position K2 in a virtual space (three-dimensional coordinate space) whose coordinate origin is the reference position K1 corresponding to the user U. to generate virtual space data including the position information of Then, the CPU 15 causes the storage unit 14 to store the generated virtual space data.

Subsequently, the CPU 15 places determination objects (determination objects HF′, HB′, HR' and HL') are placed (step S407). When arranging the judgment object, the CPU 15 adds the position information of the arranged judgment object to the virtual space data stored in the storage unit 14 .

Further, when the play of the dance game is started, the CPU 15 determines whether or not there is a moving object appearance trigger (step S409). Appearance triggers are generated at timings preset according to music. When the CPU 15 determines in step S409 that there is an appearance trigger (YES), the process proceeds to step S411.

In step S411, the CPU 15 arranges a moving object (one or more of the moving objects NF', NB', NR', and NL' in FIG. 8) at the appearance position based on the reference position K2 in the virtual space, Movement is started toward the determination position (the position of the determination object corresponding to each moving object). When arranging the moving object, the CPU 15 adds the position information of the arranged moving object to the virtual space data stored in the storage unit 14 . Further, when moving the arranged moving object, the CPU 15 updates the position information of the moving object added to the virtual space data stored in the storage unit 14 . Then, the process proceeds to step S413. On the other hand, when the CPU 15 determines that there is no appearance trigger in step S409 (NO), the process proceeds to step S413 without performing the process of step S411.

In step S413, the CPU 15 determines whether or not the moving object has reached the determination position. The CPU 15 erases from the virtual space the moving object that has reached the determination position (YES) (step S415). When deleting a moving object from the virtual space, the CPU 15 deletes the position information of the moving object to be deleted from the virtual space data stored in the storage unit 14 .

On the other hand, the CPU 15 continues to gradually move the moving object that has determined (NO) that it has not reached the determination position toward the determination position (step S417). When moving the moving object, the CPU 15 updates the position information of the moving object to be moved among the virtual space data stored in the storage unit 14 .

Next, the CPU 15 determines whether or not the dance game has ended (step S419). For example, the CPU 15 determines that the dance game has ended when the music being played ends. When the CPU 15 determines that the dance game has not ended (NO), the process returns to step S409. On the other hand, when the CPU 15 determines that the dance game has ended (YES), it ends the pointing object placement process.

The order of placement of the determination object and the placement of the moving object that appears first may be the same, the determination object may be placed first, or conversely, the determination object may be placed after (the moving object that appears first may be placed first). until it reaches the judgment position).

[Summary of the second embodiment]
As described above, the game device 10A according to the present embodiment further detects a user image UK, which is a virtual image (an example of an image) corresponding to the user U, from captured video of the real space. Then, the game device 10A arranges an instruction object that instructs the action of the user U at a position based on the reference position K2 of the user image UK corresponding to the user U in the virtual space so that the instruction object can be visually recognized by the user.

As a result, the game device 10A, when worn on the head, can display an instruction object that instructs the action of the user U, for example, around the virtual image (user image UK) of the user U reflected in the mirror MR. Therefore, with a simple configuration, it is possible to guide the user on what actions should be taken so that more intuitive play is possible. For example, the game device 10A can simultaneously overlook and visually recognize the pointing objects displayed around the user image UK (for example, front, back, left, and right) without limiting the position of the pointing object to a part of the virtual space. Therefore, it is possible to diversify the types of actions to be instructed to the user U during play. In addition, the game device 10A allows the user U to play while looking at the direction of the mirror MR facing the user U without looking at his/her own feet and pointing object located below, so that it is not difficult to dance. can be done. In addition, the game device 10A displays pointing objects around the user U playing the game and around a virtual image of the user (user image UK) reflected in the mirror MR, for example, so that the user can It is possible to play while arbitrarily selecting one of the pointing objects that is easier to play.

Note that the mirror MR may be anything other than a mirror as long as it has a mirror effect (specular reflection). For example, when the user U plays in a place where the room is brightened at night (outside is dark) and faces the windowpane, the windowpane can be used as the mirror MR and the virtual image of the user U reflected on the windowpane can be used. good.

In addition, the game device 10A reverses the front-rear orientation with respect to the reference position K2 when arranging the pointing object at the position (surrounding the user image UK) based on the reference position K2 in the virtual space. As a result, the game device 10A can display the pointing object corresponding to the direction of the user image UK reflected in the mirror MR, so that the user can be guided as to what action should be taken so as to enable more intuitive play. can be done.

When arranging the pointing object at the position (around the user image UK) based on the reference position K2 in the virtual space, the game device 10A arranges the pointing object at the position (around the user U) based on the reference position K1. visibility may be reduced. For example, the game device 10A may make the pointing object placed at the position based on the reference position K1 semi-transparent or reduced in size, so that the visibility is reduced and the display mode is less conspicuous. Further, when arranging the pointing object at the position (around the user image UK) based on the reference position K2 in the virtual space, the game device 10A places the pointing object at the position (around the user U) based on the reference position K1. You don't have to.

As a result, the game device 10A can prevent the pointing object displayed on the mirror MR from being hidden by the pointing object displayed around the user U, thereby improving the visibility of the pointing object. can.

In the present embodiment, the user U arranges the pointing object in the virtual space in association with the user image UK (virtual image of himself/herself) reflected in the mirror MR. ), the pointing object may be arranged in the virtual space in association with the image of the user U being displayed. For example, the game device 10A further includes a camera (imaging device) that captures an image of the user U in real space and a monitor (display device) that displays the captured image in real time on the side facing the user U. The captured image of the user U is displayed on the monitor. Then, instead of the user image UK (a virtual image of itself) reflected in the mirror MR, the game device 10A detects the image of the user U from the image displayed on the monitor, associates the detected image of the user U, A pointing object may be placed in the virtual space. In this case, the position of the image of the user U displayed on the monitor is the reference position. Note that the image of the user U displayed on the monitor is left-right reversed with respect to the user image UK in which the user U is reflected in the mirror MR. Therefore, the game device 10A reverses the direction of the instruction object arranged in association with the image of the user U displayed on the monitor not only in the front-rear direction but also in the left-right direction with respect to the instruction object arranged in correspondence with the user U.

In addition, a game mode in which the pointing object is placed using the mirror MR in this embodiment, a game mode in which the pointing object is placed using the above-described monitor, and a mirror and monitor mode described in the first embodiment. The game mode (the mode of game processing by the game device 10) in which the pointing object is placed without using either one of them has its own display mode of the pointing object (the reference position when the pointing object is placed, whether the pointing object is reversed front to back or left to right). no, etc.) are different. Therefore, when there are two or more game modes among these game modes (for example, when the configuration of the game device 10 and the configuration of the game device 10A are combined), the user selects which mode in advance before starting the dance game. It may be configured such that it is possible to select whether to use it. By doing so, the user image UK reflected in the mirror MR and the image of the user U displayed on the monitor can be detected smoothly, and erroneous recognition can be reduced.

[Third embodiment]
Next, a third embodiment of the invention will be described.
In the above-described first and second embodiments, the example in which the game device 10 (10A) is configured as one complete device as a transmissive HMD has been described. may be configured as a device of

FIG. 12 is a block diagram showing an example of the hardware configuration of a game system including the game device 10C according to this embodiment. The game device 10C does not include a video output device. The illustrated game system 1C includes a game device 10C and an HMD 20C as a video output device. For example, the HMD 20C is a transmissive HMD.

The HMD 20C includes an imaging section 21C, a display section 22C, a sensor 23C, a storage section 24C, a CPU 25C, a communication section 26C, and a sound output section 27C. The imaging unit 21C, the display unit 22C, the sensor 23C, and the sound output unit 27C respectively correspond to the imaging unit 11, the display unit 12, the sensor 13, and the sound output unit 17 shown in FIG. The storage unit 24C temporarily stores the data of the captured video imaged by the imaging unit 21C, the display data acquired from the game device 10C, and the like. The storage unit 24C also stores programs and the like necessary for controlling the HMD 20C. CPU25C functions as a control center which controls each part with which HMD20C is provided. The communication unit 26C communicates with the game device 10C using wired or wireless communication. The HMD 20C transmits the captured image captured by the imaging section 21C, the detection signal of the sensor 23C, and the like to the game device 10C via the communication section 26C. The HMD 20C also acquires display data, sound data, and the like of the dance game from the game device 10C via the communication unit 26C.

The game device 10C includes a storage section 14C, a CPU 15C, and a communication section 16C. The storage unit 14C stores dance game programs and data, generated virtual space data, and the like. The CPU 15C functions as a control center that controls each part of the game device 10C. For example, the CPU 15C executes the game program stored in the storage unit 14C to execute game processing, processing to generate a virtual space corresponding to the real space from the captured image, and placement of a pointing object in the generated virtual space. Placement processing, detection of the user's motion and evaluation based on the timing and position of the pointing object, and the like are executed. The communication unit 16C communicates with the HMD 20C using wired or wireless communication. The game device 10C acquires, through the communication unit 16C, the imaged video imaged by the imaging unit 21C of the HMD 20C, the detection signal of the sensor 23C, and the like. The game device 10C also transmits display data, sound data, and the like of the dance game to the HMD 20C via the communication unit 16C.

FIG. 13 is a block diagram showing an example of the functional configuration of the game device 10C according to this embodiment. The illustrated game device 10C includes a control section 150C as a functional configuration realized by the CPU 15C executing a program stored in the storage section 14C. The control unit 150C follows the control shown in FIG. 4 except that it exchanges data with each unit (the imaging unit 21C, the display unit 22C, the sensor 23C, the sound output unit 27, etc.) of the HMD 20C via the communication unit 16C. The configuration is the same as that of the section 150 or the control section 150A shown in FIG.

Thus, the game device 10C may be configured as another device that communicates with the HMD 20 as an external device. Note that the game device 10C can be, for example, a smart phone, a PC (Personal Computer), a home game machine, or the like.

[Fourth embodiment]
Next, a fourth embodiment of the invention will be described.
In the above-described first to third embodiments, a mode using an HMD worn on the head has been described, but in this embodiment, a mode without using an HMD will be described.

FIG. 14 is a diagram showing an overview of game processing by the game device according to this embodiment. This figure shows a bird's-eye view of the play situation in which the user U plays the dance game using the game device 10D. The illustrated game device 10D is an example to which a smart phone is applied. In this embodiment, the pointing object arranged in the virtual space is displayed on the display unit 12D or the monitor 30D of the game device 10D in association with the image of the user U captured by the front camera 11DA of the game device 10D. Therefore, the user can play intuitively. The monitor 30D is an external display unit (display device) connectable to the game device 10D by wire or wirelessly. For example, the monitor 30D has a larger screen display than the display unit 12D provided in the game device 10D.

The game device 10D recognizes the image area of the user U from the captured image of the user U. FIG. Then, the game device 10D defines a reference position K3 corresponding to the position of the user U in the virtual space, and generates virtual space (XYZ three-dimensional space) data in which the pointing object is arranged at a position based on the reference position K3. , superimposed on the captured image. Note that the reference position K3 may be a position corresponding to the center of the head of the user U, or a position corresponding to the center of gravity of the user U, and can be defined as an arbitrary position.

In the illustrated example, the pointing object is also displayed on the display unit 12D of the game device 10D, but by displaying it on the monitor 30D having a screen larger than that of the game device 10D, the visibility of the pointing object for the user U becomes higher. Description will be made with reference to the display screen of the monitor 30D. A user image UV indicates an image of the user U included in the captured image. Around the user image UV, an image is displayed in which a pointing object placed at a position based on the reference position K3 of the user U in the virtual space is superimposed on the captured image.

For example, the captured image captured by the front camera 11DA can be a left-right reversed image like a mirror. A determination object HR and a moving object NR that instruct the user U to move rightward are displayed on the right side of the user image UV facing the screen of the monitor 30D, and a leftward movement of the user U is displayed on the left side of the user image UV. A determination object HL and a moving object NL for instructing actions are displayed. In addition, a determination object HF and a moving object NF that instruct the user U to move forward are displayed on the front side of the user image UV facing the screen of the monitor 30D, and a movement object NF behind the user U is displayed on the back side of the user image UV. A judgment object HB and a moving object NB are displayed for instructing an action to .

As described above, in the present embodiment, the pointing object arranged in the virtual space is associated with the image of the user U, similarly to the case where the pointing object is displayed around the user image UK reflected in the mirror MR shown in FIG. Since it can be displayed, it is possible to guide the user to what actions should be taken so that intuitive play is possible.

Note that the display of this pointing object may be performed on either one of the game device 10D and the monitor 30D.

[Hardware Configuration of Game Device 10D]
A hardware configuration of the game device 10D will be described with reference to FIG.
FIG. 15 is a block diagram showing an example of the hardware configuration of the game device 10D according to this embodiment. The game device 10D includes two imaging units of a front camera 11DA and a back camera 11DB, a display unit 12D, a sensor 13D, a storage unit 14D, a CPU 15D, a communication unit 16D, a sound output unit 17D, and a video output unit. 18D.

The front camera 11DA is provided on the surface (front side) of the game apparatus 10D on which the display section 12D is provided, and captures an image in the direction facing the display section 12D. The back camera 11DB is provided on the side opposite to the side on which the display unit 12D of the game device 10D is provided (back side), and captures an image in the direction facing the back side.

The display unit 12D includes a liquid crystal display, an organic EL display, and the like. For example, the display unit 12D may be configured as a touch panel that detects touch operations on the display screen.

The sensor 13D is a sensor that outputs a detection signal regarding the direction of the game device 10D. For example, the sensor 13D may include one or a plurality of sensors such as a gyro sensor, an acceleration sensor, an inclination sensor, and a geomagnetic sensor.

The storage unit 14D includes, for example, EEPROM, ROM, Flash ROM, RAM, etc., and stores the dance game program and data, generated virtual space data, and the like.

The CPU 15D functions as a control center that controls each part of the game device 10D. For example, the CPU 15D executes game processing by executing the game program stored in the storage unit 14D, and as described with reference to FIG. , superimposing and displaying the pointing object placed in the .

The communication unit 16D includes, for example, a communication device that performs wireless communication such as Bluetooth (registered trademark) and Wi-Fi (registered trademark).

The sound output unit 17D outputs the performance sound of the play music of the dance game, the effect sound of the game, and the like. For example, the sound output unit 17 includes a speaker, a phone terminal to which earphones and headphones are connected, and the like.

The video output section 18D includes a video output terminal for outputting the video to be displayed on the display section 12D to an external display device (for example, the monitor 30D shown in FIG. 14). The video output terminal may be a dual-purpose terminal including output other than video output, or may be a terminal dedicated to video output.

[Functional Configuration of Game Device 10D]
Next, with reference to FIG. 16, the functional configuration of the game device 10D will be described.
FIG. 16 is a block diagram showing an example of the functional configuration of the game device 10D according to this embodiment. The illustrated game device 10D includes a control section 150D as a functional configuration realized by the CPU 15D executing a program stored in the storage section 14D. The control unit 150D includes a video acquisition unit 151D, a virtual space generation unit 152D, a user detection unit 153D, an object placement unit 154D, a display control unit 156D, a motion detection unit 157D, and an evaluation unit 158D. .

The image acquisition unit 151D (an example of an acquisition unit) acquires a captured image of the real space captured by the front camera 11DA. For example, the video acquisition unit 151D acquires a captured video including a user U playing a dance game, as shown in FIG.

The virtual space generation unit 152D (an example of the generation unit) generates a virtual space corresponding to the real space from the captured image acquired by the image acquisition unit 151D. For example, the virtual space generation unit 152D detects the position of an object (floor, wall, etc.) existing in the real space from the acquired captured image, and includes position information of at least part of the detected object (floor, wall, etc.). 3D coordinate space data is generated as virtual space data. As an example, the virtual space generation unit 152D determines that the reference position K3 corresponding to the user U detected from the captured image by the user detection unit 153D is the virtual space (XYZ three-dimensional coordinate space) to generate virtual space data. During play, the reference position K3 (coordinate origin) and the X, Y and Z axes are fixed. The virtual space generation unit 152D stores the generated virtual space data in the storage unit 14D.

The user detection unit 153D detects the image of the user U from the captured image acquired by the image acquisition unit 151D. For this detection, it is necessary to recognize that the image of the person detected from the captured image is the image of the user U playing the dance game. As a method of recognizing the image of the user U, for example, an identifiable marker (mark, mark, etc.) is attached to the body of the user U, and the user detection unit 153D detects this marker from the captured image. Therefore, it may be recognized that the image is that of the user U. Further, by instructing the user U to perform a specific action (for example, raising and lowering the right hand), the user detection unit 153D detects a person performing the action according to the instruction from the captured image, thereby detecting the user U. You may recognize that it is an image of U.

The object placement unit 154D (an example of the placement unit) places the pointing object at a position based on the reference position K3 corresponding to the user U in the virtual space so that the user U can see it. Specifically, the object placement unit 154D places determination objects (see determination objects HF, HB, HR, and HL in FIG. 14) at determination positions in the virtual space corresponding to the position of the floor. Further, the object arrangement unit 154D arranges moving objects (see moving objects NF, NB, NR, and NL in FIG. 14) at the appearance positions in the virtual space at preset timings according to the music, Move (change the placement position) toward it. The object placement unit 154D updates the virtual space data stored in the storage unit 14D based on the coordinate information of the placement position in the virtual space when placing the pointing object (the determination object and the moving object).

The display control unit 156D generates a composite image by synthesizing the captured image acquired by the image acquisition unit 151D and the image of the pointing object placed in the virtual space by the object placement unit 154D. Then, the display control unit 156D causes the display unit 12D to display the generated composite image. Also, the display control unit 156D outputs the generated composite image from the image output unit 18D. For example, the display control unit 156D causes the display unit 12D to display the generated composite video in a horizontally reversed manner. Similarly, the display control unit 156D horizontally inverts the generated synthesized image and outputs it from the image output unit 18D.

The motion detection unit 157D (an example of the detection unit) detects motion of at least part of the body of the user U from the captured image acquired by the image acquisition unit 151D. For example, the motion detection unit 157D detects motions of the user U's feet playing a dance game. The motion detection unit 157D detects the motion of the foot by extracting and tracking the image area of the foot from each frame of the captured image.

The evaluation unit 158D evaluates the motion of at least part of the body of the user U detected by the motion detection unit 157D based on the timing and position based on the pointing object arranged in the virtual space. For example, the evaluation unit 158D compares the timing and position at which the moving object reaches the determination object with the timing and position of the motion of the user U's foot (the motion of stepping on the determination object), and evaluates the play by the motion of the user U. do. Based on the comparison result, the evaluation unit 158D adds the score when it can be determined that the timing and position of both match, and does not add the score when it can determine that they do not match.

Note that the evaluation unit 158D may evaluate the play by the user U's action by comparing the position of the user U's foot at the timing when the moving object reaches the determination object with the position of the determination object.

[Operation of pointing object placement processing]
Next, in the dance game processing executed by the CPU 15D of the game device 10D, a pointing object placement process for generating a virtual space and placing a pointing object will be described. FIG. 17 is a flowchart showing an example of pointing object placement processing according to the present embodiment.

First, the CPU 15D acquires a captured image of the real space captured by the front camera 11DA (step S501). For example, as shown in FIG. 14, the CPU 15 acquires a captured image including a user U playing a dance game.

Next, the CPU 15D detects an image of the user U playing the dance game from the captured image acquired in step S501 (step S503).

Next, the CPU 15D generates a virtual space corresponding to the real space from the captured image acquired in step S501 (step S505). For example, the CPU 15D detects the position of an object (floor, wall, etc.) existing in the real space from the captured image, and the data of the three-dimensional coordinate space containing at least part of the position information of the detected object (floor, wall, etc.). is generated as virtual space data. As an example, the CPU 15D stores a detected object (floor, wall, etc.) in a virtual space (three-dimensional coordinate space) whose coordinate origin is the reference position K3 corresponding to the user U detected from the captured image by the user detection unit 153D. generates virtual space data including at least part of the position information of Then, the CPU 15D stores the generated virtual space data in the storage section 14D.

Subsequently, the CPU 15D places the determination objects (the determination objects HF, HB, HR, HL) are placed (step S507). When arranging the judgment object, the CPU 15D adds the position information of the arranged judgment object to the virtual space data stored in the storage unit 14D.

Further, when the play of the dance game is started, the CPU 15D determines whether or not there is a moving object appearance trigger (step S509). Appearance triggers are generated at timings preset according to music. When the CPU 15D determines in step S509 that there is an appearance trigger (YES), the process proceeds to step S511.

In step S511, the CPU 15D arranges the moving object (one or more of the moving objects NF, NB, NR, and NL in FIG. 14) at the appearance position based on the reference position K3 in the virtual space, and determines the determination position (each position of the determination object corresponding to the moving object). When arranging the moving object, the CPU 15D adds the position information of the arranged moving object to the virtual space data stored in the storage unit 14D. Further, when moving the arranged moving object, the CPU 15D updates the position information of the moving object added to the virtual space data stored in the storage unit 14D. Then, the process proceeds to step S513. On the other hand, when the CPU 15D determines that there is no appearance trigger in step S509 (NO), the process proceeds to step S513 without performing the process of step S511.

In step S513, the CPU 15D determines whether or not the moving object has reached the determination position. The CPU 15D erases from the virtual space the moving object that has reached the determination position (YES) (step S515). When deleting a moving object from the virtual space, the CPU 15D deletes the position information of the moving object to be deleted from the virtual space data stored in the storage unit 14D.

On the other hand, the CPU 15D continues to gradually move the moving object that has determined (NO) that it has not reached the determination position toward the determination position (step S517). When moving the moving object, the CPU 15D updates the position information of the moving object to be moved among the virtual space data stored in the storage unit 14D.

Next, the CPU 15D determines whether or not the dance game has ended (step S519). For example, the CPU 15D determines that the dance game has ended when the music being played ends. When the CPU 15D determines that the dance game has not ended (NO), the process returns to step S509. On the other hand, when the CPU 15D determines that the dance game has ended (YES), it ends the pointing object placement process.

The order of placement of the determination object and the placement of the moving object that appears first may be the same, the determination object may be placed first, or conversely, the determination object may be placed after (the moving object that appears first may be placed first). until it reaches the judgment position).

[Operation of Pointer Object Display Processing]
Next, in the processing of the dance game executed by the CPU 15D of the game device 10D, the operation of pointing object display processing for displaying pointing objects arranged in the virtual space will be described. In this embodiment, the instruction object is displayed as a composite image in which the instruction object is superimposed on the imaged image of the user U.
FIG. 18 is a flowchart showing an example of pointing object display processing according to the present embodiment.

The CPU 15D acquires the captured image of the real space captured by the front camera 11DA, and also acquires the virtual space data from the storage unit 14D (step S601).

Then, the CPU 15D generates a synthesized image by synthesizing the obtained imaged image and the pointing object included in the virtual space data, and causes the display unit 12D to display the generated synthesized image (step S603). Further, the CPU 15D outputs the generated synthesized image to the image output unit 18D and displays it on the monitor 30D connected to the image output unit 18D (step S603). As a result, a composite image in which the instruction object is superimposed on the imaged image of the user U is displayed on the display unit 12D and the monitor 30D in real time. Note that the CPU 15D may display the composite image on either the display section 12D or the monitor 30D.

Next, the CPU 15D determines whether or not the dance game has ended (step S605). For example, the CPU 15D determines that the dance game has ended when the music being played ends. When the CPU 15D determines that the dance game has not ended (NO), the process returns to step S601. On the other hand, when the CPU 15D determines that the dance game has ended (YES), it ends the pointing object display process.

[Operation of play evaluation process]
Next, in the dance game processing executed by the CPU 15D of the game device 10D, the operation of the play evaluation process for evaluating the play by the motion of at least part of the body of the user U will be described. FIG. 19 is a flowchart showing an example of play evaluation processing according to this embodiment.

The CPU 15D acquires a captured image of the real space captured by the front camera 11DA (step S701). Next, the CPU 15D detects motion of at least a part of the body of the user U from the captured image acquired in step S701 (step S703). For example, the CPU 15D detects the motion of the user U's feet playing the dance game.

Then, the CPU 15D evaluates the movement of at least part of the body (for example, foot) of the user U detected in step S703 based on the timing and position based on the pointing object placed in the virtual space (step S705). ). For example, the CPU 15D compares the timing and position at which the moving object reaches the determination object with the timing and position of the motion of the user U's foot (motion of stepping on the determination object), and evaluates the play by the motion of the user U's foot. do.

Also, the CPU 15D updates the score of the game based on the evaluation result in step S705 (step S707). For example, the CPU 15D adds a score if it can determine that the timing and position at which the moving object reaches the determination object and the timing and position of the motion of the user U's foot (stepping motion on the determination object) match. However, if it can be determined that they do not match, the score is not added.

Next, the CPU 15D determines whether or not the dance game has ended (step S709). For example, the CPU 15D determines that the dance game has ended when the music being played ends. When the CPU 15D determines that the dance game has not ended (NO), the process returns to step S701. On the other hand, when the CPU 15D determines that the dance game has ended (YES), it ends the play evaluation process.

[Summary of the fourth embodiment]
As described above, the game device 10D according to the present embodiment obtains a captured image of the real space and generates a virtual space corresponding to the real space from the obtained captured image. Then, the game device 10D arranges an instruction object that instructs the action of the user U at a position based on the reference position K3 corresponding to the user in the generated virtual space so that the user U can see the captured image and the virtual space. A synthesized image obtained by synthesizing the image of the pointing object placed inside is displayed on the display unit 12D (an example of the display unit). Note that the game device 10D may display the composite image on the monitor 30D (an example of the display unit). In addition, the game device 10D detects the movement of at least a part of the body of the user U from the acquired captured image, and evaluates the detected movement based on the timing and position of the pointing object arranged in the virtual space. do.

As a result, the game device 10D performs synthesis in which the instruction object is combined with the image of the user U in the game processing for evaluating the action of the user U based on the timing and position of the instruction object that instructs the action of the user U. In order to visually display the video on the game device 10D (for example, a smartphone) or an externally connected monitor 30D (for example, a home TV), the user operates to enable more intuitive play with a simple configuration. I can guide you on what to do.

For example, the game device 10D horizontally reverses the composite image and displays it on the display unit 12D or the monitor 30D.

Accordingly, the game device 10D can be played while looking at the display unit 12D or the monitor 30D as if the user U were looking at a mirror.

In addition, the game device 10 moves the pointing object (eg, moving object) arranged at a predetermined position (appearance position) in the virtual space toward a predetermined determination position (eg, position of the determination object). Then, the game device 10 detects at least a part of the body of the user U (the For example, to evaluate the movement of the foot).

Thereby, the game device 10D can evaluate whether or not the user U has performed the action as instructed using the captured image.

[Modification]
Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configuration is not limited to the above-described embodiments, and includes design and the like within the scope of the gist of the present invention. For example, the configurations described in the above embodiments can be combined arbitrarily.

It should be noted that the instruction object described in each of the above-described embodiments is an example, and various modes are possible as long as the instruction object instructs the user U to perform an action. For example, the content of the action to be instructed to the user U differs depending on the type (mode) of the instruction object. For example, by changing the thickness (width in the Z-axis direction) of the moving object, the time from when the bottom of the moving object reaches the determination object to when the top of the moving object reaches the determination object changes. Alternatively, the thickness of the moving object may be used to indicate how long the determination object should continue to be stepped on. The moving object does not necessarily appear in the vertical direction of the determination object that is the destination of the movement, and may appear in a position away from the vertical direction. Also, the moving direction of the moving object and the position of the determination object can be arbitrarily set.

Also, the determination object may not be displayed at the determination position. For example, when the position of the floor surface is the determination position, the timing and position at which the moving object reaches the floor surface are the instruction contents for instructing the user U's action. For example, a moving object having a certain thickness (e.g., a length similar to the height of the user U) not in the vertical direction but in an oblique direction (e.g., a direction tilted 45° with respect to the vertical direction) direction toward the floor (judgment position), from the position on the XY plane when the bottom of the moving object reaches the floor to the XY plane when the top of the moving object reaches the floor , the position at which the moving object reaches the floor changes over time. Therefore, a moving object having a certain thickness in an oblique direction may be used to instruct to move the stepping position.

Also, the determination position is not limited to the floor surface, and can be set to any position between the floor surface and the ceiling, for example. Note that the height of the determination position may be set according to the height of the user U by detecting the height. Alternatively, the displayed moving object itself may indicate the action of the user U without providing the determination position. For example, the position at which the moving object appears or the position at which the moving object is moving and its timing may indicate the action of the user U. For example, the movement trajectory of the moving object may indicate the movement trajectory of the user U (for example, the hand movement trajectory).

Note that although the game device 10 and the game device 10A configured as the transmissive HMD described in the first and second embodiments are provided with the imaging unit 11, the imaging unit 11 is And, as a device different from the game device 10A, it may be installed in another place where the image of the user U playing the dance game can be captured. In this case, a device including the imaging unit 11 installed at another location is connected to the game device 10 and the game device 10A for wired or wireless communication. Further, in the game system 1C including the game device 10C and the HMD 20C described in the third embodiment, the configuration in which the HMD 20C is provided with the imaging unit 21C corresponding to the imaging unit 11 has been described. Similarly, the unit 21C may be installed in another place where it is possible to capture an image of the user U playing the dance game as a device separate from the HMD 20C. In this case, a device including the imaging unit 21C installed at another location is connected to the HMD 20C or the game device 10C for wired or wireless communication. Also, the imaging unit 21C may be provided in the game device 10C.

Further, in the game device 10D described in the fourth embodiment, the front camera 11DA provided as an imaging unit is used to capture an image of the user U playing the dance game. A configuration in which an image of the user U is captured using a device including an imaging unit installed at another location may be employed. In this case, a device including an imaging unit installed at another location is connected to the game device 10D by wire or wirelessly.

In addition, a program for realizing the functions of the control units 150 (150A, 150C, 150D) described above is recorded in a computer-readable recording medium, and the program recorded in this recording medium is read and executed by the computer system. By doing so, processing as the control unit 150 (150A, 150C, 150D) may be performed. Here, "loading and executing the program recorded on the recording medium into the computer system" includes installing the program in the computer system. The "computer system" here includes hardware such as an OS and peripheral devices. A "computer system" may also include a plurality of computer devices connected via a network including communication lines such as the Internet, WAN, LAN, and dedicated lines. The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard discs incorporated in computer systems. Thus, the recording medium storing the program may be a non-transitory recording medium such as a CD-ROM. Recording media also include internal or external recording media accessible from the distribution server for distributing the program. The program code stored in the recording medium of the distribution server may be different from the program code in a format executable by the terminal device. That is, as long as it can be downloaded from the distribution server and installed in a form that can be executed on the terminal device, the form stored in the distribution server does not matter. It should be noted that the program may be divided into a plurality of parts, and the divided programs may be downloaded at different timings and then merged in the terminal device, or the distribution servers that distribute the divided programs may be different. In addition, "computer-readable recording medium" is a volatile memory (RAM) inside a computer system that acts as a server or client when the program is transmitted via a network, and retains the program for a certain period of time. It shall also include things. Further, the program may be for realizing part of the functions described above. Further, it may be a so-called difference file (difference program) that can realize the above functions by combining with a program already recorded in the computer system.

Also, some or all of the functions of the control units 150 (150A, 150C, 150D) described above may be implemented as an integrated circuit such as an LSI (Large Scale Integration). Each of the functions described above may be individually processorized, or part or all of them may be integrated and processorized. Also, the method of circuit integration is not limited to LSI, but may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integration circuit technology that replaces LSI appears due to advances in semiconductor technology, an integrated circuit based on this technology may be used.

Further, in the above embodiment, at least part of the data stored in the storage units 14 (14C, 14D) included in the game devices 10 (10A, 10C, 10D) may be stored in an externally connected storage device. . The externally connected storage device is a storage device that is wired or wirelessly connected to the game device 10 (10A, 10C, 10D). For example, the externally connected storage device may be a storage device connected via a USB (Universal Serial Bus), a wireless LAN (Local Area Network), a wired LAN, or the like, or may be a storage device connected via the Internet ( data server). A storage device (data server) connected via the Internet or the like may be used using cloud computing.

A server connected via the Internet or the like may have a configuration corresponding to at least a part of each unit included in the control unit 150 (150A, 150C, 150D). For example, the above embodiment can be applied to a so-called cloud game in which game processing such as a dance game is executed by a server.

In the above embodiment, a dance game, which is an example of a music game, has been described as an example, but the present invention is not limited to dance games. For example, it can be applied to general music games in which objects appearing in time with music are operated. In addition to music games, the present invention can also be applied to games in which operations such as punching, kicking, brushing off, or hitting with a weapon are performed on objects that appear at predetermined timings.

[Appendix]
From the above description, the present invention can be grasped, for example, as follows. In order to facilitate understanding of the present invention, reference numerals in the accompanying drawings are added in parentheses for convenience, but the present invention is not limited to the illustrated embodiments.

(Appendix A1) A game program according to one aspect of the present invention is a video output device (10 , 10A, 20C) to obtain a captured image of the real space (S101, S301, S401), and corresponding to the real space from the captured image. a step of generating a virtual space (S103, S405) to generate a virtual space for performing a motion of the user; arranging so as to be visible (S105, S109, S407, S411); displaying the virtual space in which at least the pointing object is arranged in association with the real space (S203); detecting motion of at least part of the user's body (S303); and evaluating the detected motion based on timing and position based on the pointing object placed in the virtual space (S305). and let it run.

According to the configuration of appendix A1, the game program wears a video output device such as an HMD on the head in the game processing for evaluating the user's motion based on the timing and position based on the instruction object that instructs the user's motion. Accordingly, since the pointing object is associated with the real space and can be visually recognized by the user, it is possible to guide the user to the action to be performed with a simple configuration so that the user can play more intuitively.

(Appendix A2) Further, one aspect of the present invention is the game program according to Appendix A1, wherein the reference position is the user (U) wearing the video output device (10, 10A, 20C). a first reference position (K1) in the virtual space corresponding to the position of , wherein the first reference position is based on the position of the video output device in the virtual space.

According to the configuration of appendix A2, the game program can display the instruction object in association with the real space based on the position of the user playing the game. A more intuitive play is possible.

(Appendix A3) Further, one aspect of the present invention is the game program according to Appendix A2, wherein in the disposing step (S105, S109, S407, S411), the video output device (10, 10A, 20C) according to the orientation of the user (U) who is wearing the .

According to the configuration of appendix A3, the game program does not issue an instruction to move outside the range of the user's field of vision (for example, behind). You can play without worrying about , and you can prevent the difficulty of play from becoming too high.

(Appendix A4) Further, one aspect of the present invention is the game program according to Appendix A1, wherein the computer detects an image (UK) corresponding to the user (U) from the captured image (S403). ), wherein the reference position includes a second reference position (K2) in the virtual space of the detected image corresponding to the user.

According to the configuration of appendix A4, by wearing a video output device such as an HMD on the head, the game program generates a virtual image of the user (user image UK) can be displayed around the player, and with a simple configuration, it is possible to guide the user on what actions should be taken so that more intuitive play is possible. For example, the game program can simultaneously overlook and visually recognize the pointing objects displayed around the user image UK (for example, front, back, left, and right) without restricting the positions where the pointing objects are arranged to a part of the virtual space. Therefore, it is possible to diversify the types of actions to be instructed to the user during play. In addition, the game program makes it possible for the user to evaluate the motion of the user without looking at the user's own feet and the pointing object located below, so that it is possible to prevent difficulty in dancing.

(Appendix A5) Further, one aspect of the present invention is the game program according to Appendix A2 or Appendix A3, wherein the computer detects (UK) corresponding to the user (U) from the captured image. (S403), wherein the reference position includes a second reference position (K2) in the virtual space of the detected image corresponding to the user.

According to the configuration of appendix A5, by wearing a video output device such as an HMD on the head, the game program generates a virtual image of the user (user image UK) can be displayed around the player, and with a simple configuration, it is possible to guide the user on what actions should be taken so that more intuitive play is possible. For example, the game program can simultaneously overlook and visually recognize the pointing objects displayed around the user image UK (for example, front, back, left, and right) without restricting the positions where the pointing objects are arranged to a part of the virtual space. Therefore, it is possible to diversify the types of actions to be instructed to the user during play. In addition, the game program makes it possible for the user to evaluate the motion of the user without looking at the user's own feet and the pointing object located below, so that it is possible to prevent difficulty in dancing. In addition, the game program displays instruction objects around the user playing the game and around a virtual image of the user reflected in a mirror, for example, so that the user can easily play the instruction objects. It is possible to make it possible to play while arbitrarily selecting one of them.

(Appendix A6) Further, one aspect of the present invention is the game program according to Appendix A5, wherein in the placing step (S105, S109, S407, S411), the second reference position ( K2), reducing visibility of the pointing object placed at the position based on the first reference position (K1), or based on the first reference position (K1) Do not place the pointing object at the position.

According to the configuration of Appendix A6, the game program causes the pointing object displayed at the position (around the user U) based on the first reference position (for example, the reference position K1) to move to the second reference position (for example, the reference position K2 ) (surrounding the virtual image of the user reflected in the mirror MR), the visibility of the pointing object can be improved.

(Appendix A7) Further, one aspect of the present invention is the game program according to any one of appendices A4 to A6, wherein the detected image (UK) corresponding to the user (U) is a is an image of the user reflected in a mirror (MR) present in the virtual space, and in the step of arranging (S105, S109, S407, S411), at a position based on the second reference position (K2) in the virtual space. When arranging the pointing object, the forward and backward direction with respect to the second reference position is reversed.

According to the configuration of appendix A7, the game program can display the pointing object corresponding to the orientation of the user's virtual image (user image UK) reflected in the mirror, so the user can intuitively play while looking at the mirror. In this way, the user can be guided as to what should be done.

(Supplementary Note A8) Further, one aspect of the present invention is the game program according to any one of Supplementary Notes A1 to A7, wherein in the placing step (S105, S109, S407, S411), The pointing object placed at the predetermined position is moved toward the predetermined determination position, and in the evaluating step (S305), the timing at which the pointing object moving in the virtual space reaches the determination position and the Evaluate the detected motion based on the determined position.

According to the configuration of appendix A8, the game program can use the captured image to evaluate whether or not the user has performed the action as instructed.

(Appendix A9) Further, one aspect of the present invention is the game program according to any one of appendices A1 to A8, wherein the action to be instructed to the user (U) differs depending on the type of the pointing object. .

According to the configuration of appendix A9, the game program can diversify the contents of the user's actions during play, and can provide a highly entertaining game.

(Appendix A10) In addition, the game processing method according to one aspect of the present invention outputs an image that can be visually recognized by the user (U) and outputs an image that allows the real space to be visually recognized by the user (U) wearing the game processing method. A game processing method executed by a computer that executes processing of a game that can be played using a device (10, 10A, 20C), the method comprising steps of obtaining a captured image of the real space (S101, S301, S401 ), a step of generating a virtual space corresponding to the real space from the captured image (S103, S405); arranging an instruction object that instructs a user's action so that the user can see it (S105, S109, S407, S411); and displaying at least the virtual space in which the instruction object is arranged in association with the real space. detecting motion of at least a part of the user's body from the captured image (S303); and determining the detected motion based on the pointing object placed in the virtual space. Evaluating (S305) based on timing and location.

According to the configuration of appendix A10, the game processing method evaluates the user's motion based on the timing and position of an instruction object that instructs the user's motion, in which a video output device such as an HMD is mounted on the head. By doing so, since the pointing object is associated with the real space and can be visually recognized by the user, it is possible with a simple configuration to guide the user as to what action should be performed so that the user can play more intuitively.

(Appendix A11) Further, the game device (10, 10A, 10C) according to one aspect of the present invention outputs an image that can be visually recognized by the user (U) by being worn on the head of the user (U) and A game device that executes processing of a playable game using a video output device (10, 10A, 20C) that can visually recognize the real space, and an acquisition unit (151, S101, S301, S401); a generation unit (152, S103, S405) for generating a virtual space corresponding to the real space from the captured image acquired by the acquisition unit; an arrangement unit (154, S105, S109, S407, S411) for arranging an instruction object for instructing an action of the user at a position based on the reference position (K1, K2) corresponding to the user so that the user can see the instruction object. a display control unit (156, S203) for displaying the virtual space in which at least the pointing object is arranged in association with the real space; A detection unit (157, S303) that detects at least part of the motion, and an evaluation unit that evaluates the motion detected by the detection unit based on the timing and position based on the pointing object placed in the virtual space. (158, S305).

According to the configuration of appendix A11, the game device wears a video output device such as an HMD on the head in game processing for evaluating the user's motion based on the timing and position based on the instruction object that instructs the user's motion. Accordingly, since the pointing object is associated with the real space and can be visually recognized by the user, it is possible to guide the user to the action to be performed with a simple configuration so that the user can play more intuitively.

(Appendix B1) Further, the game program according to one aspect of the present invention provides a computer with a step of obtaining a captured image of a real space (S501, S701), and a virtual space corresponding to the real space from the captured image. and placing an instruction object for instructing the user's action at a position based on the reference position (K3) corresponding to the user (U) in the virtual space so that the user can see the instruction object (S505). step (S507, S511); displaying a synthesized image obtained by synthesizing the imaged image and the image of the pointing object placed in the virtual space on the display unit (12D, 30D) (S603); detecting motion of at least part of the user's body from the video (S703); and evaluating the detected motion based on timing and position based on the pointing object placed in the virtual space. (S705) and are executed.

According to the configuration of Appendix B1, the game program synthesizes the instruction object with the image captured by the user in the game processing for evaluating the user's action based on the timing and position based on the instruction object that instructs the user's action. Since the synthesized video is visually displayed on the display unit of a smartphone or a home-use TV, it is possible to guide the user to what actions should be performed with a simple configuration so that more intuitive play is possible.

(Appendix B2) Further, one aspect of the present invention is the game program according to Appendix B1, wherein in the displaying step (S603), the synthesized image is horizontally reversed and displayed on the display unit (12D, 30D). display.

According to the configuration of appendix B2, the game program can be played while looking at the display section (monitor) as if the user were looking at a mirror.

(Appendix B3) Further, one aspect of the present invention is the game program according to Appendix B1 or Appendix B2, wherein in the step of arranging (S507, S511), the game program is arranged at a predetermined position in the virtual space. The pointing object is moved toward a predetermined determination position, and in the evaluating step (S705), the detection is performed based on the timing when the pointing object moving in the virtual space reaches the determination position and the determination position. Evaluate the action taken.

According to the configuration of Appendix B3, the game program can use the captured image to evaluate whether or not the user has performed the action as instructed.

(Appendix B4) Further, one aspect of the present invention is the game program according to any one of appendices B1 to B3, wherein the content of the action to be instructed to the user (U) differs depending on the type of the instruction object. .

According to the configuration of Appendix B4, the game program can diversify the contents of the user's actions during play, and can provide a highly entertaining game.

(Appendix B5) Further, a game processing method according to an aspect of the present invention is a game processing method executed by a computer, comprising steps of obtaining a captured image of a real space (S501, S701); a step of generating a virtual space corresponding to the real space from the image (S505); and an instruction to direct the user's movement to a position based on the reference position (K3) corresponding to the user (U) in the virtual space. a step of arranging an object so that it can be visually recognized by the user (S507, S511); a step of displaying (S603); a step of detecting motion of at least part of the user's body from the captured image (S703); and evaluating (S705) based on the timing and position based.

According to the configuration of appendix B5, in the game processing for evaluating the user's action based on the timing and position of the instruction object that instructs the user's action, the instruction object is combined with the image captured by the user. Since the synthesized video is visually displayed on a display unit such as a smartphone or a home-use TV, it is possible to guide the user to what actions should be performed with a simple configuration so that more intuitive play is possible.

(Appendix B6) Further, the game device (10D) according to an aspect of the present invention includes an acquisition unit (151D, S501, S701) that acquires a captured video image of a real space; a generating unit (152D) that generates a virtual space corresponding to the real space from an image; an arrangement unit (154D, S507, S511) for arranging an instruction object that instructs the user's action so that the user can visually recognize the captured image and an image of the instruction object arranged in the virtual space; A display control unit (156D, S603) for displaying a composite image on a display unit (12D, 30D), and a detection unit ( 157D, S703) and an evaluation unit (158D, S705) that evaluates the motion detected by the detection unit based on the timing and position based on the pointing object placed in the virtual space.

According to the configuration of appendix B6, in the game processing for evaluating the user's action based on the timing and position of the instruction object that instructs the user's action, the instruction object is synthesized with the image captured by the user. Since the synthesized video is visually displayed on the display unit of a smartphone or a home-use TV, it is possible to guide the user to what actions should be performed with a simple configuration so that more intuitive play is possible.

1C game system, 10, 10A, 10C, 10D game device, 11 imaging unit, 11DA front camera, 11DB back camera, 12, 12D display unit, 13, 13D sensor, 14, 14C, 14D storage unit, 15, 15C, 15D CPU, 16, 16C, 16D communication section, 17, 17D sound output section, 18D video output section, 20C HMD, 21C imaging section, 22C display section, 23C sensor, 24C storage section, 25C CPU, 26C communication section, 27C sound output 150, 150A, 150C, 150D control unit 151, 151D image acquisition unit 152, 152D virtual space generation unit 153A user image detection unit 153D user detection unit 154, 154A, 154D object placement unit 155 line of sight direction detection unit 156, 156D display control unit 157, 157D motion detection unit 158, 158D evaluation unit

Claims (9)

A computer that executes processing of a game that can be played using a video output device that can be worn on the user's head to output a video that can be visually recognized by the user and that can visually recognize the real space,
a step of obtaining a captured image of the real space;
generating a virtual space corresponding to the real space from the captured image;
detecting an image corresponding to the user from the captured image;
arranging an instruction object for instructing an action of the user at a position in the virtual space based on a reference position corresponding to the user so that the user can see the instruction object;
displaying the virtual space in which at least the pointing object is placed in association with the real space;
detecting motion of at least part of the user's body from the captured image;
evaluating the detected motion based on timing and position based on the pointing object placed in the virtual space;
and
The reference position includes a second reference position in the virtual space of the detected image corresponding to the user,
game program. The reference position includes a first reference position in the virtual space corresponding to the position of the user wearing the video output device,
The first reference position is based on the position of the video output device in the virtual space,
The game program according to claim 1. In the placing step,
restricting a position where the pointing object is placed to a part of the virtual space according to the orientation of the user wearing the video output device;
3. A game program according to claim 2. In the placing step,
reducing the visibility of the pointing object placed at the position based on the first reference position when placing the pointing object at the position based on the second reference position in the virtual space; or not placing said pointing object at a position based on
4. The game program according to claim 2 or 3 . the detected image corresponding to the user is an image of the user reflected in a mirror present facing the user;
In the placing step,
when arranging the pointing object at a position based on the second reference position in the virtual space, reversing a front-rear orientation with respect to the second reference position;
A game program according to any one of claims 1 to 4 . In the placing step,
moving the pointing object arranged at a predetermined position in the virtual space toward a predetermined judgment position;
In the evaluating step,
evaluating the detected motion based on the timing at which the pointing object moving in the virtual space reaches the determination position and the determination position;
A game program according to any one of claims 1 to 5 . The content of the action to be instructed to the user differs depending on the type of the instruction object.
A game program according to any one of claims 1 to 6 . A game processing method executed by a computer that executes processing of a game that can be played using a video output device that can be worn on the user's head to output a video that can be visually recognized by the user and that can visually recognize the real space. and
a step of obtaining a captured image of the real space;
generating a virtual space corresponding to the real space from the captured image;
detecting an image corresponding to the user from the captured image;
arranging an instruction object for instructing an action of the user at a position in the virtual space based on a reference position corresponding to the user so that the user can see the instruction object;
displaying the virtual space in which at least the pointing object is placed in association with the real space;
detecting motion of at least part of the user's body from the captured image;
evaluating the detected motion based on timing and position based on the pointing object placed in the virtual space;
including
The reference position includes a second reference position in the virtual space of the detected image corresponding to the user,
Game processing method. A game device that, by being worn on the user's head, outputs a video that can be visually recognized by the user and executes processing of a game that can be played using a video output device that allows the user to visually recognize the real space,
an acquisition unit that acquires a captured video image of the real space;
a generation unit that generates a virtual space corresponding to the real space from the captured image acquired by the acquisition unit;
an image detection unit that detects an image corresponding to the user from the captured image;
an arrangement unit that arranges an instruction object that instructs the user's action at a position based on a reference position corresponding to the user in the virtual space generated by the generation unit so that the user can see the instruction object;
a display control unit that displays the virtual space in which at least the pointing object is arranged in association with the real space;
a detection unit that detects motion of at least part of the user's body from the captured image acquired by the acquisition unit;
an evaluation unit that evaluates the motion detected by the detection unit based on timing and position based on the pointing object placed in the virtual space;
with
The reference position includes a second reference position in the virtual space of the detected image corresponding to the user,
game device.

Images