JP7144796B2 - Simulation system and program - Google Patents

Description

The present invention relates to a simulation system, a program, and the like.

Conventionally, systems that realize mixed reality (MR), augmented reality (AR), virtual reality (VR), and the like are known. For example, Patent Literature 1 discloses an image processing device that synthesizes and displays an image of a virtual object observed from the user's viewpoint position on a real space image observed from the user's viewpoint position. In this image processing device, a real space area is specified by a user's operation, a virtual object is created based on the specified real space area, and the created virtual object is converted into an observed real space image. Synthesize to

Japanese Patent Application Laid-Open No. 2005-157611

If an MR or AR system such as that disclosed in Patent Document 1 is used, a virtual space image can be synthesized with a real space image. However, no proposals have been made for arranging placement objects, such as guide objects, in a corresponding virtual space so that the user moving in the real space can easily see them. In addition, no technique has been proposed for effectively utilizing information on the real space in arranging objects to be arranged.

According to some aspects of the present invention, it is possible to provide a simulation system, a program, and the like that make it possible to effectively utilize real space information and display arranged objects in a preferred display mode.

According to one aspect of the present invention, real space information obtained by performing recognition processing of a real space around a user, an information acquisition unit that acquires user information including position information of the user, and a virtual space generation unit for generating a virtual space corresponding to the real space; setting a reference point in the virtual space based on the user information and the real space information; and placing objects based on the reference point. and a display processing unit that performs processing for displaying an image including the image of the placed object on a display unit, wherein the object processing unit performs a user movement in the virtual space corresponding to the user. A simulation system that sets the display mode of the placement object according to the positional relationship between the body and the placement object. The present invention also relates to a program that causes a computer to function as the above units, or a computer-readable information storage medium that stores the program.

In one aspect of the present invention, real space information is acquired by performing recognition processing of the real space around the user, and a virtual space corresponding to the real space is generated based on the real space information. A reference point is set based on the real space information, and the placed object is arranged in the virtual space based on the reference point. Then, the display mode of the placed object is set according to the positional relationship between the user moving body in the virtual space and the placed object. Therefore, it is possible to set a virtual space corresponding to the real space, place the placed object, and set the display mode of the placed object according to the positional relationship between the user's moving body and the placed object in the virtual space. Become. Therefore, it is possible to provide a simulation system or the like that makes effective use of real space information and enables display of arranged objects in a suitable display mode.

In one aspect of the present invention, the object processing unit arranges a first arranged object and a second arranged object as the arranged objects, and arranges the first arranged object in the direction of the reference point side of the user's moving object. When the placement object and the second placement object are positioned, the first placement object is set in a display state, the second placement object is set in a non-display state, and the reference point of the user's moving object is set. When the first placed object is positioned in the direction opposite to the side direction and the second placed object is positioned in the direction of the reference point side of the user moving body, the first placed object A non-display state may be set, and the second placed object may be set in a display state.

By doing so, it is possible to limit the number of displayed arranged objects, and it is possible to set a display mode of arranged objects that is easy for the user to see.

In one aspect of the present invention, a priority order regarding display is set for the objects in the virtual space. When the objects are arranged so as to overlap each other, the display mode of the arranged objects may be set based on the order of priority.

By doing so, it is possible to set the display mode of the placed object reflecting the priority of display of the placed object and other objects.

Further, in one aspect of the present invention, the object processing unit may set a display mode or a layout mode of the placed object according to a line-of-sight direction of the user's moving body.

By doing so, it is possible to set the placed object in an appropriate display mode and layout mode according to the line-of-sight direction of the user's moving body.

In one aspect of the invention, the object processing section may set a display mode or a layout mode of the placed object according to the position of the reference point.

By doing so, it is possible to set the placed object in an appropriate display mode or layout mode according to the position of the reference point.

Further, in one aspect of the present invention, the object processing section may set the reference point according to the user's situation information, the user's mobile object's situation information, or the game situation information.

In this way, it is possible to set an appropriate reference point according to the user's situation, the user's mobile object's situation, or the game situation.

In one aspect of the present invention, the information acquisition unit acquires the environment information of the real space, and the object processing unit sets the reference point or displays the arranged object according to the environment information. Alternatively, the arrangement mode may be set.

By doing so, it is possible to set an appropriate reference point and an appropriate display mode or layout mode of the placed object, reflecting the environment of the real space.

In one aspect of the present invention, the object processing unit may set the display mode or layout mode of the placed object according to the user situation information, the user moving body situation information, or the game situation information. good.

In this way, it is possible to set an appropriate display mode or layout mode of placed objects according to the user's situation, the user's mobile object's situation, or the game situation.

In one aspect of the present invention, the placement object may be a placement object that guides movement of the user, or a placement object that warns, notifies, or instructs the user.

In this way, by using the arranged objects, it is possible to realize a suitable guide display for the user in the real space, and to issue warnings, notifications, or instructions to the user regarding various situations.

Further, in one aspect of the present invention, the object processing unit may set the reference point for the target of the user's behavior, and arrange the placement object based on the reference point set for the target. good.

In this way, it is possible to arrange the arrangement object and set the display mode based on the reference point set for the target of the user's action.

1 is a block diagram showing a configuration example of a simulation system according to this embodiment; FIG. An example of an HMD used in this embodiment. An example of an HMD used in this embodiment. 4A and 4B are explanatory diagrams of real space information acquisition processing by real space recognition processing. Explanatory drawing of the display area of HMD. Explanatory diagram of a play field in real space. 7A and 7B are explanatory diagrams of the attraction game according to the present embodiment. 8A and 8B are explanatory diagrams of the attraction game according to the present embodiment. 9A and 9B are explanatory diagrams of hit processing according to the present embodiment. FIG. 4 is an explanatory diagram of hit processing according to the embodiment; 11A and 11B are explanatory diagrams of hit processing according to the present embodiment. 12(A) and 12(B) are explanatory diagrams of the attraction game of the present embodiment. 13(A) and 13(B) are explanatory diagrams of the attraction game of the present embodiment. FIG. 4 is an explanatory diagram of a ride-type attraction game played by a user riding on a ride housing; 15(A) and 15(B) are explanatory diagrams of the attraction game of the present embodiment. Explanatory drawing of the attraction game of this embodiment. 17A and 17B are explanatory diagrams of hit processing according to the present embodiment. 18(A) to 18(C) are explanatory diagrams of the hit volume setting and user movement detection processing. 19A and 19B are explanatory diagrams of hit processing according to the present embodiment. 20A and 20B are explanatory diagrams of processing for characters. 21A and 21B are explanatory diagrams of setting processing of the display mode of the placed object according to the present embodiment. 22A and 22B are explanatory diagrams of setting processing of the display mode of the placed object according to the present embodiment. FIG. 7 is an explanatory diagram of setting processing of the display mode of the placed object according to the embodiment; 24A and 24B are explanatory diagrams of setting processing of the display mode of placed objects based on priority. 25(A) and 25(B) are explanatory diagrams of setting processing of a display mode or an arrangement mode of a placed object according to the line-of-sight direction of a user character. FIG. 11 is an explanatory diagram of setting processing of a display mode and an arrangement mode of a placed object according to the position of a reference point; 5 is a flowchart showing an example of a reference point setting process based on environment information, user, user character, character situation information, and game situation information; 5 is a flow chart showing an example of processing for setting a display mode or an arrangement mode of placed objects based on situation information of a user or a user character and game situation information; 29(A) and 29(B) are explanatory diagrams of setting processing of the display mode or layout mode of the placed object based on the user's situation information. 30(A) and 30(B) are explanatory diagrams of setting processing of a display mode or a placement mode of placed objects based on game situation information. 4 is a flowchart showing a detailed processing example of the embodiment;

The present embodiment will be described below. It should be noted that the embodiments described below do not unduly limit the content of the present invention described in the claims. Moreover, not all the configurations described in the present embodiment are essential constituent elements of the present invention.

1. Simulation System FIG. 1 is a block diagram showing a configuration example of a simulation system (simulator, game system, image generation system) according to the present embodiment. The simulation system of this embodiment realizes MR (Mixed Reality), AR (Augmented Reality), etc., and is applicable to various systems such as a game system that provides game content. The simulation system of this embodiment is not limited to the configuration of FIG. 1, and various modifications such as omitting some of its constituent elements (each part) or adding other constituent elements are possible.

The operation unit 160 is used by a user (player) to input various operation information (input information). The operation unit 160 can be realized by various operation devices such as operation buttons, direction keys, joysticks, steering wheels, pedals, levers, or voice input devices.

The detection unit 162 detects the motion of the user, and can be realized by a motion sensor such as an acceleration sensor or a gyro sensor. For example, the detection unit 162 is attached to a user's body part or the like to detect movement of the user's body part or the like. For example, a motion sensor such as an acceleration sensor detects a change in acceleration or the like when a part of the user moves to detect the movement of the user. Note that the movement of the user may be detected using a camera. As an example, a leap motion sensor that detects the movement of a fingertip or the like with two infrared LEDs and two cameras (stereo cameras) may be used. Alternatively, the motion of the user may be detected by posture detection using Kinect (registered trademark).

The image capturing unit 164 (camera) captures an image of a subject, and is realized by an image sensor such as a CCD or CMOS sensor and an optical system including a focus lens and the like. The imaging unit 164 may be a depth camera realized by an infrared camera or the like. For example, a light coding method may be adopted in which an IR projector that posts an infrared pattern is provided in the imaging unit 164, the projected infrared pattern is read by an infrared camera, and depth information is obtained from pattern distortion.

The storage unit 170 stores various information. The storage unit 170 functions as a work area for the processing unit 100, the communication unit 196, and the like. A game program and game data necessary for executing the game program are held in this storage unit 170 . The function of the storage unit 170 can be realized by a semiconductor memory (DRAM, VRAM), HDD (hard disk drive), SSD, optical disk device, or the like. The storage unit 170 includes a real space information storage unit 171 , an object information storage unit 172 and a drawing buffer 178 .

The information storage medium 180 (computer-readable medium) stores programs and data, and its function can be realized by an optical disk (DVD, BD, CD), HDD, semiconductor memory (ROM), or the like. . The processing unit 100 performs various processes of this embodiment based on programs (data) stored in the information storage medium 180 . That is, in the information storage medium 180, a program (a program for causing the computer to execute the processing of each unit) for functioning a computer (a device having an input device, a processing unit, a storage unit, and an output unit) as each unit of the present embodiment is stored. is stored.

A display unit 190 outputs an image generated according to this embodiment, and can be realized by an LCD, an organic EL display, a CRT, or the like. The display unit 190 is, for example, a display (display device) of an HMD 200 (head-mounted display device) shown in FIGS. 2 and 3, which will be described later. The sound output unit 192 outputs the sound generated according to this embodiment, and its function can be realized by a speaker, headphones, or the like.

An I/F (interface) unit 194 performs interface processing with the portable information storage medium 195, and its function can be realized by an ASIC for I/F processing. The portable information storage medium 195 is used by the user to store various types of information, and is a storage device that retains the storage of this information even when the power is not supplied. The portable information storage medium 195 can be implemented by an IC card (memory card), USB memory, magnetic card, or the like.

The communication unit 196 communicates with the outside (another device) via a wired or wireless network, and its function is implemented by hardware such as a communication ASIC or a communication processor, and communication firmware. It can be realized by

A program (data) for causing a computer to function as each part of the present embodiment is delivered from an information storage medium of a server (host device) to the information storage medium 180 (or storage unit 170) via the network and communication unit 196. You may Use of the information storage medium by such a server (host device) can also be included within the scope of the present invention.

The processing unit 100 (processor) receives operation information from the operation unit 160 and tracking information of the HMD 200 in FIGS. , information acquisition processing, virtual space generation processing, object processing, virtual camera control processing, game processing (simulation processing), display processing, sound processing, and the like, based on programs.

Each process (each function) of this embodiment performed by each part of the processing unit 100 can be realized by a processor (a processor including hardware). For example, each process of this embodiment can be implemented by a processor that operates based on information such as a program and a memory that stores information such as the program. In the processor, for example, the function of each section may be implemented by separate hardware, or the function of each section may be implemented by integrated hardware. For example, a processor includes hardware, which may include circuitry for processing digital signals and/or circuitry for processing analog signals. For example, the processor can be configured with one or more circuit devices (such as ICs) or one or more circuit elements (such as resistors and capacitors) mounted on a circuit board. The processor may be, for example, a CPU (Central Processing Unit). However, the processor is not limited to the CPU, and various processors such as GPU (Graphics Processing Unit) or DSP (Digital Signal Processor) can be used. Alternatively, the processor may be a hardware circuit based on ASIC. The processor may also include amplifier circuits, filter circuits, and the like that process analog signals. The memory (storage unit 170) may be a semiconductor memory such as SRAM or DRAM, or may be a register. Alternatively, it may be a magnetic storage device such as a hard disk device (HDD) or an optical storage device such as an optical disk device. For example, the memory stores computer-readable instructions, and the processes (functions) of each part of the processing unit 100 are realized by executing the instructions by the processor. The instruction here may be an instruction set that constitutes a program, or an instruction that instructs a hardware circuit of a processor to operate.

The processing unit 100 includes an information acquisition unit 102 , a virtual space generation unit 104 , an object processing unit 106 , a virtual camera control unit 112 , a game processing unit 114 , a display processing unit 120 and a sound processing unit 130 . As described above, each process of this embodiment executed by these units can be realized by a processor (or processor and memory). It should be noted that various modifications such as omitting some of these constituent elements (each section) or adding other constituent elements are possible.

The information acquisition unit 102 performs acquisition processing of various types of information. A virtual space generation unit 104 performs processing for generating a virtual space, and an object processing unit 106 performs various processing regarding character objects and placement objects. Details of the information acquisition unit 102, the virtual space generation unit 104, and the object processing unit 106 will be described later.

The virtual camera control unit 112 (program module for virtual camera control processing) controls the virtual camera. For example, processing for controlling the virtual camera is performed based on the user's operation information and tracking information input by the operation unit 160 . For example, the virtual camera control unit 112 controls a virtual camera set as the user's first-person viewpoint or third-person viewpoint. For example, by setting a virtual camera at a position corresponding to the viewpoint (first-person viewpoint) of a user moving body in virtual space corresponding to a user in real space, and setting the viewpoint position and line-of-sight direction of the virtual camera, It controls the position (position coordinates) and orientation (rotation angle around the rotation axis). Alternatively, a virtual camera is set at a viewpoint (third-person viewpoint) position that follows the user's moving body, and the position and orientation of the virtual camera are controlled by setting the viewpoint position and line-of-sight direction of the virtual camera.

For example, the virtual camera control unit 112 controls the virtual camera to follow changes in the user's viewpoint based on the tracking information of the user's viewpoint information obtained by viewpoint tracking. For example, in the present embodiment, tracking information (viewpoint tracking information) of viewpoint information, which is at least one of the user's viewpoint position and line-of-sight direction, is acquired. This tracking information can be obtained by performing tracking processing of the HMD 200, for example. Then, the virtual camera control unit 112 changes the viewpoint position and line-of-sight direction of the virtual camera based on the acquired tracking information (at least one of the user's viewpoint position and line-of-sight direction). For example, the virtual camera control unit 112 controls the virtual camera so that the viewpoint position and line-of-sight direction (position, posture) of the virtual camera in the virtual space change according to changes in the user's viewpoint position and line-of-sight direction in the real space. Configure your camera. By doing so, it is possible to control the virtual camera so as to follow changes in the user's viewpoint based on the tracking information of the user's viewpoint information.

The game processing unit 114 (game processing program module) performs various game processing for the user to play the game. In other words, the game processing unit 114 (simulation processing unit) executes various simulation processes for the user to experience MR (Mixed Reality), AR (Augmented Reality), or VR (Virtual Reality). do. Game processing includes, for example, processing for starting a game when a game start condition is satisfied, processing for progressing a started game, processing for ending a game when a game end condition is satisfied, or calculating a game result. processing, etc.

The display processing unit 120 performs display processing of virtual space images (game images, simulation images). For example, drawing processing is performed based on the results of various processes (game processing, simulation processing) performed by the processing unit 100 to generate an image and display it on the display unit 190 . Specifically, geometry processing such as coordinate transformation (world coordinate transformation, camera coordinate transformation), clipping processing, perspective transformation, or light source processing is performed. coordinates, texture coordinates, color data, normal vectors or alpha values, etc.) are created. Then, based on this drawing data (primitive surface data), an object (one or a plurality of primitive surfaces) after perspective transformation (after geometry processing) is stored in the drawing buffer 178 (frame buffer, work buffer, etc.) in units of pixels. draw to a memory buffer). As a result, an image viewed from a virtual camera (given viewpoints: first and second viewpoints for left eye and right eye) is generated in the virtual space. The drawing processing performed by the display processing unit 120 can be realized by vertex shader processing, pixel shader processing, or the like.

The sound processing unit 130 (sound processing program module) performs sound processing based on the results of various processes performed by the processing unit 100 . Specifically, it generates game sounds such as music (music, BGM), sound effects, or voices, and causes the sound output unit 192 to output the game sounds.

The simulation system of this embodiment includes an information acquisition unit 102, a virtual space generation unit 104, an object processing unit 106, and a display processing unit 120, as shown in FIG.

The information acquisition unit 102 (a program module for information acquisition processing) acquires real space information obtained by performing recognition processing of the real space around the user. The information acquisition unit 102 also acquires user information including user position information. Real space recognition processing can be realized by spatial mapping, for example. For example, space mapping is realized by performing real space recognition processing using various cameras provided as the imaging unit 164 . Taking FIG. 2, which will be described later, as an example, real space recognition processing is performed using an RGB camera 246, a depth camera 247, and environment recognition cameras 248 and 249 provided in the HMD 200. FIG. The real space information is 3D map information of the real space obtained by real space recognition processing (space mapping), and is information obtained by meshing the scanned real space. Real space information obtained by scanning the real space is stored in the real space information storage unit 171 . Real space information is composed of, for example, a plurality of spatial data. Each spatial data of the plurality of spatial data is data corresponding to each region when the real space is divided into a plurality of regions and scanned. An ID (surface ID) is assigned to each space data and stored in the real space information storage unit 171 . For example, a polygon mesh is generated based on the space data to which this ID is assigned, and it becomes possible to mesh and display the scanned real space. It also enables interaction between real space maps and virtual space objects.

The information acquisition unit 102 also acquires user information including position information of the user in real space. The user information can include the user's orientation information in real space. As the positional information of the user, for example, viewpoint positional information of the user can be used. User's line-of-sight direction information can be used as the user's direction information. The user's position information (viewpoint position information) and direction information (line-of-sight direction information) can be acquired by head tracking processing of the HMD 200 .

The virtual space generation unit 104 (program module for virtual space generation processing) performs processing for generating a virtual space corresponding to the real space based on the real space information acquired by the information acquisition unit 102 . The virtual space generation unit 104 performs setting processing of a virtual space that is a three-dimensional space (game space). For example, map information obtained by real space recognition processing (space mapping) is set as virtual space information. For example, virtual space information is set by a plurality of spatial data corresponding to a plurality of areas obtained by dividing the real space. The virtual space information is stored in the object information storage unit 172 as object information.

A virtual space generation unit 104 (virtual space setting unit) performs setting processing of a virtual space (object space) in which an object is arranged. For example, various objects (polygons, free-form surfaces or sub (Objects composed of primitive surfaces such as division surfaces) are placed and set in the virtual space. That is, the position and rotation angle (synonymous with orientation and direction) of the object in the world coordinate system are determined, and at that position (X, Y, Z) at that rotation angle (rotation angle around the X, Y, Z axes) Place objects. Specifically, in the object information storage unit 172 of the storage unit 170, object information such as the position, rotation angle, movement speed, movement direction, etc. of an object (part object) in the virtual space is associated with an object number. stored. The virtual space generation unit 104 performs, for example, a process of updating this object information for each frame.

The object processing unit 106 (program module for object processing) performs various types of processing for character objects and placement objects. Specifically, the object processing unit 106 sets a reference point in the virtual space based on the user information and the real space information. For example, a reference point is set in the virtual space based on the user's position information and the real space information. This reference point setting process is performed by the reference point setting unit 108 . Then, the object processing unit 106 performs processing for arranging the arranged object in the virtual space based on the reference point. For example, a process of arranging an arrangement object at an arrangement point set corresponding to a reference point is performed. The display processing unit 120 (a program module for display processing) performs processing for displaying an image including an image of an arranged object on the display unit 190 . For example, a process of displaying an image (virtual space image) including an image of a placed object on the display device of the HMD 200 is performed.

In this embodiment, the object processing unit 106 performs processing for setting the display mode of the placed object according to the positional relationship between the user's moving body in the virtual space corresponding to the user and the placed object. For example, the distance relationship between the user's moving body and the placed object, the directional relationship between the direction in which the user's moving body faces and the placed object, etc. are determined as the positional relationship. Then, the object processing unit 106 displays the placed object in the first display mode when the user's moving body and the placed object have the first positional relationship (first distance relationship, first direction relationship). set as When the user's moving object and the placed object have a second positional relationship (second distance relationship, second direction relationship), the placed object is displayed in a second display mode different from the first display mode. set to The display mode of the placed object includes the color, luminance (brightness), translucency, texture, display content, etc. of the placed object.

For example, when the distance between the user's moving body and the placed object is short (when the distance is equal to or less than the threshold value), the object processing unit 106 sets the placed object to the first color, the first brightness, or the first translucency. map the first texture, or set the display content of the placed object to the first display content. On the other hand, when the distance between the user's moving object and the placed object is long (longer than the threshold distance), the placed object is set to a second color, a second brightness, or a second translucency, A second texture is mapped or the display content of the placement object is set to the second display content.

In addition, when the direction of the user's moving object (line-of-sight direction) is the first directional relationship in which the direction of the user's moving object is directed toward the placed object, the object processing unit 106 sets the placed object to the first color, the first brightness, or the Set to a first translucency, map a first texture, or set the display content of the placed object to a first display content. On the other hand, if the direction of the user's moving body (line-of-sight direction) is in the second direction relationship in which the placed object faces in a direction different from that of the placed object, the placed object is given the second color, the second brightness, or the second color. Set translucency, map a second texture, or set the display content of the placed object to the second display content.

Here, the first color is, for example, a color (a conspicuous color) with higher visibility or distinguishability than the second color, the first luminance is a luminance brighter than the second luminance, and the first The translucency of is higher in translucency (α value) than the second translucency and is closer to opaque. The first texture is a texture with higher visibility or distinguishability than the second texture.

Further, the display content of the placement object is instruction content or the like represented by characters, symbols, graphics, or the like drawn on the placement object. When the placed object is a placed object for guiding the user (for guidance display), the display content is the content of a guide, warning, or the like instructing the user. The first display content is the content instructing the user for guidance, warning, etc. when the user moving body and the placed object have the first positional relationship (first distance relationship, first direction relationship). is. The second display content is content instructing the user for guidance, warning, etc. when the user's moving body and the placed object are in the second positional relationship (second distance relationship, second direction relationship). be.

Also, the object processing unit 106 arranges a first arrangement object and a second arrangement object as arrangement objects. Note that the number of placed objects is not limited to two, and may be three or more. Then, when the first placed object and the second placed object are positioned in the direction of the reference point side of the user's moving object, the object processing unit 106 sets the first placed object to the display state, and sets the second placed object to the display state. Sets the placed object to the hidden state. The display state is, for example, an opaque state, and the non-display state is, for example, a transparent state. When the first placed object is positioned in the opposite direction to the reference point side of the user's moving body and the second placed object is positioned in the direction of the reference point side of the user's moving body, the object processing unit 106 Then, the first placed object is set to be hidden and the second placed object is set to be displayed. That is, the first placed object that was in the display state is changed to the non-display state, and the second placed object that was in the non-display state is changed to the display state. For example, by gradually making the first placed object in the display state transparent, it changes to the non-display state, which is the transparent state. Also, by gradually increasing the translucency (α value) of the second placed object that has been in a non-display state, it is changed to a display state that is an opaque state. In other words, when the distance between the user's moving object and the reference point is in the first distance relationship, the first placed object is set in the display state, and the second placed object is set in the non-display state. set to On the other hand, if the distance between the user's moving body and the reference point is the second distance relationship, which is closer than the first distance relationship, the first placed object is set to a non-display state, and the second placed object is placed. to be displayed.

In addition, display priorities are set for objects in the virtual space. This priority information is stored, for example, in the storage unit 170 (object information storage unit 172). Then, the object processing unit 106, when the placed object overlaps another object in the virtual space as viewed from the user's moving body (when the placed object overlaps with the user's moving body in the line-of-sight direction), the priority order is Set the display mode of the placed object based on. Here, the other object may be the second placed object, or may be an object other than the placed object. An object that is not a layout object is an object that is different in type from a layout object, such as an object that is displayed when an event occurs.

For example, it is assumed that other objects (second placed objects or objects other than placed objects) in the virtual space are set to have higher display priority than placed objects. Suppose that the arranged object and another object overlap each other when viewed from the user's moving body (in the line-of-sight direction of the user's moving body). In this case, other objects with higher display priority are set to the display state, and arranged objects with lower display priority are set to the non-display state. For example, even if the placed object is placed closer to the user's moving body than the other objects, the other objects are set to the displayed state, and the placed object is set to the non-display state. On the other hand, it is assumed that the display priority of the placed object in the virtual space is higher than that of other objects in the virtual space. Assume that the placed object and other objects are placed overlapping each other when viewed from the user's moving body. In this case, a placed object with a high display priority is set to a display state, and other objects with a low display priority are set to a non-display state.

The object processing unit 106 also sets the display mode or layout mode of the placed object according to the line-of-sight direction of the user's moving body. The line-of-sight direction of the user's moving body corresponds to, for example, the line-of-sight direction of the user, and can be said to be, for example, the direction in which the user's moving body faces. The arrangement mode of the arrangement object is the arrangement position, the arrangement direction, the arrangement number, or the like of the arrangement object. For example, the object processing unit 106 displays the arranged objects depending on whether the line-of-sight direction of the user moving body (the user's line-of-sight direction) faces the reference point or faces in a different direction from the reference point. The mode or arrangement mode is made different. For example, the color, brightness, translucency, texture, or display contents of the placed objects are varied, or the placement position, placement direction, or number of placements, which are the mode of placement of the placed objects, are varied.

The user's moving body corresponds to the user in the real space, and may be a display object on which the image is displayed or a virtual object on which the image is not displayed. For example, when a first-person viewpoint image is displayed as an image of the virtual space, the user moving body may not be displayed, or only a part of the user moving body (for example, hand, chest, face, etc.) may be displayed. may be displayed. When the user's moving body is hidden, for example, virtual cameras (left-eye and right-eye virtual cameras) in the virtual space corresponding to the user's viewpoint in the real space can be regarded as the user's moving body (avatar). can.

The object processing unit 106 also sets the display mode or layout mode of the placed object according to the position of the reference point. For example, assume that a first reference point and a second reference point are set as reference points. For example, the first reference point is a reference point positioned in the line-of-sight direction of the user's moving body, and the second reference point is a reference point positioned in a direction different from the line-of-sight direction of the user's moving body. In this case, the object processing unit 106 makes the display mode or layout mode of the placed object different between the first reference point and the second reference point. For example, the color, brightness, translucency, texture, or display contents of the placed objects are varied, or the placement position, placement direction, or number of placements, which are the mode of placement of the placed objects, are varied.

The object processing unit 106 also sets a reference point according to the user's situation information, the user's mobile object situation information, or the game situation information. For example, based on these pieces of information, the setting positions and the number of setting of the reference points are determined, or the appearance timing of the reference points is determined. For example, the setting position or number of reference points is changed, or the appearance timing of the reference points is changed. The user's situation information is, for example, the user's age group, physique, or gender information. Alternatively, the user's situation information is the user's game play situation. For example, the user's game level (beginner, intermediate, advanced), information on the number of times the user has played the game, information on the frequency of game play, information on the user's game results, and the like. The information of the user moving body is information such as the status (level, experience value) and ability (attack power, defense power, etc.) of the user moving body, and is, for example, information represented by game parameters of the user moving body. The game status information includes the progress of the game, the status of the stage where the game is played, the status of the map, and the like.

The information acquisition unit 102 also acquires environment information of the real space. Environmental information is information about the environment such as the brightness and size of the real space. For example, it is brightness information and size information in the play field where the user is positioned. Then, the object processing unit 106 sets the reference point according to the environment information. Alternatively, the display mode or layout mode of the placed object is set according to the environment information. For example, depending on the environmental information, the setting positions and the number of setting of the reference points are determined, or the appearance timing of the reference points is determined. For example, the setting position or number of reference points is changed, or the appearance timing of the reference points is changed. Alternatively, depending on the environment information, the color, brightness, translucency, texture, or display contents of the placed objects are changed, or the position, direction, or number of placed objects are changed.

The object processing unit 106 also sets the display mode or layout mode of the placed object according to the user's situation information, the user's mobile object's situation information, or the game situation information. For example, the color, brightness, translucency, texture, or display content of a placed object can be changed, or the position, direction, or placement of a placed object can be changed according to user situation information, user moving body situation information, or game situation information. change the number.

A placement object is a placement object that guides the movement of the user, or a placement object that warns, notifies, or instructs the user. A placement object that guides the movement of the user is an object that indicates in which direction the user should move in real space, for example. A placement object that warns a user is an object that warns that a situation unfavorable to the user will occur when such a situation occurs. The placement object that notifies the user is an object that notifies the user of various situations or information. A placement object that instructs the user is an object that instructs the user to perform a given action or the like. However, the arranged object of this embodiment is not limited to such a type of arranged object, and various modifications are possible.

The object processing unit 106 also sets a reference point for the user's action target, and arranges the placement object based on the reference point set for the target. The target of the user's behavior is the object or destination (target location) of the user's behavior. For example, the target is an object of the user's game play (for example, an item such as a treasure chest to be searched) or a destination (destination on the playfield or map). In this embodiment, a reference point is set for such a target, and a placement object is arranged based on the set reference point. In this way, a placement object for guiding the action of the user toward the target, issuing a warning about the target, reporting the status of the target, or instructing the action of the user toward the target is set to the target. can be set based on the established reference point.

Also, in this embodiment, the object processing unit 106 sets the hit volume based on the position of the user's moving object in the virtual space corresponding to the user. For example, based on the position of the user's moving object, the position of the hit volume is determined and arranged. The hit volume may be set at the position of the user's part or the position of the possession specified by the position of the user's moving body. Then, when the user performs a given input, the object processing unit 106 performs processing on the character according to the positional relationship between the hit volume and the character. For example, the positional relationship between the hit volume and the character when the user performs a given input is determined, and the character is processed. Whether or not the user has made a given input can be detected using the operation unit 160 or the detection unit 162, for example. For example, when the user performs a given operation using the operation unit 160, it is determined that the user has performed a given input. Alternatively, when the detection unit 162 detects a given movement of the user (movement of a part such as the user's hand), it is determined that the user has performed a given input. A given motion of the user may be detected by capturing an image with the imaging unit 164 .

Then, the object processing unit 106 determines the positional relationship between the hit volume and the character at the timing when the user performs a given input. For example, hit determination processing (collision determination processing) is performed to determine whether or not the position of the character is within the hit volume. Alternatively, a hit determination process may be performed to determine whether or not the hit volume set based on the position of the user's moving body and the second hit volume set based on the position of the character intersect. These hit determination processes are performed by the hit processing unit 109 . Then, when the user performs a given input (at the timing of the given input), if the hit volume and the character are in a given positional relationship, the character is processed according to the positional relationship. against For example, when it is determined that the hit volume and the character hit, the character is hit. Specifically, it performs erasing processing, destruction processing, display mode changing processing, or notification processing using sound or vibration, which will be described later.

Note that the hit volume is, for example, the hit volume set for the user's body part or the user's belongings. For example, a hit volume that includes the user's body part or a hit volume that includes the user's belongings is set. Then, by performing hit determination processing between this hit volume and the character, it is possible to determine whether or not the user's body part or belonging has hit the character. The hit volume is, for example, a three-dimensional shape, but it may be a two-dimensional hit area.

Alternatively, the object processing unit 106 sets a hit volume between the user's moving object in the virtual space corresponding to the user and the reference point. For example, the hit volume is set in an area (intermediate area) between the position of the user's moving body and the position of the reference point. For example, the hit volume is set on a line connecting the position of the user's moving body and the position of the reference point. Then, when the user performs a given input, the object processing unit 106 performs processing on the character according to the positional relationship between the hit volume and the character. Specifically, whether or not the user has performed a given input is detected using, for example, the operation unit 160 and the detection unit 162 as described above. Then, the positional relationship between the hit volume and the character is determined at the timing when the user performs a given input. For example, a hit determination process is performed to determine whether or not the position of the character is within the hit volume. Alternatively, hit determination processing (collision determination process) may be performed. Then, when the hit volume and the character are in a given positional relationship when the user performs a given input, the character is processed according to the positional relationship. For example, hit processing is performed on a character. More specifically, character disappearance processing, destruction processing, display mode change processing, or notification processing using sound or vibration, which will be described later, are performed.

The object processing unit 106 includes a moving body processing unit 110 . The moving body processing unit 110 performs various processes on a moving body that moves within the virtual space. For example, it performs a process of moving a moving body in a virtual space (object space, game space) and a process of moving the moving body. The moving object is a user moving object, character, or the like corresponding to the user. For example, the moving object processing unit 110 processes a moving object ( Model object) is moved in the virtual space, and control processing is performed to move the moving object (motion, animation). Specifically, a simulation that sequentially obtains movement information (position, rotation angle, speed, or acceleration) and motion information (position or rotation angle of a part object) for each frame (for example, 1/60 second) of a moving object. process. Note that the frame is a unit of time for movement/motion processing (simulation processing) of a moving body and image generation processing.

When the user's moving body and the reference point are in the first distance relationship, the object processing unit 106 sets the hit volume based on the position of the user's moving body, and when the user performs a given input, the object processing unit 106 sets the hit volume. , the character is processed according to the positional relationship between the hit volume and the character. On the other hand, when the user's moving body and the reference point are in the second distance relationship, a hit volume is set between the user's moving body and the reference point, and when the user performs a given input, a hit volume is generated. The character is processed according to the positional relationship between the volume and the character. Here, the first distance relationship is a relationship in which the distance between the user's moving body and the reference point is shorter than that in the second distance relationship. In this case, the hit volume is set based on the position of the user's moving object, and hit determination processing and the like are performed. On the other hand, the second distance relationship is a relationship in which the distance between the user's moving object and the reference point is longer than that in the first distance relationship. In this case, a hit volume is set between the user's moving object and the reference point, and hit determination processing and the like are performed.

The object processing unit 106 also determines whether or not the user has made a given input based on the information from the detection unit 162 that detects the movement of the user. For example, based on the motion sensor (acceleration sensor, etc.) of the detection unit 162, the motion, vibration, or acceleration of the user's body part is detected, and it is determined whether or not the user has performed a given input. For example, it is determined whether or not the user has performed a tapping action. Specifically, the timing at which the acceleration of the user's body part changes abruptly is determined as the timing at which the user performs a given input. Then, the character is processed according to the positional relationship between the hit volume and the character at the timing when the user performs a given input.

The object processing unit 106 also performs character disappearance processing, character destruction processing, character display mode change processing, or character notification processing as processing for the character. Character disappearance processing is processing to hide a character so that it cannot be visually seen in the virtual space. The character destruction process is a process of displaying an image of a character in a destroyed state or a process of generating a destruction effect. The process of changing the display mode of the character is a process of changing the color, luminance (brightness), translucency, texture, or the like of the character. The character notification process is a process of notifying the user of the occurrence of an attack hit event or the like against the character using sound, vibration, or the like. If the attack on the character is unsuccessful, a presentation process may be performed to indicate that the attack was unsuccessful.

Note that the information acquisition unit 102 acquires the user's position information in the real space based on the viewpoint tracking information. The viewpoint tracking information can be obtained by performing tracking processing of the HMD 200, for example. For example, the information acquisition unit 102 acquires position information of the HMD 200 as position information of the user wearing the HMD 200 . As an example, the tracking information includes change information of the viewpoint position from the user's initial viewpoint position (change value of the coordinates of the viewpoint position), and change information of the line-of-sight direction from the user's initial line-of-sight direction (rotation axis of the line-of-sight direction). change in rotation angle). Based on change information of viewpoint information included in such tracking information, the viewpoint position corresponding to the user's position and the line-of-sight direction corresponding to the user's direction can be identified. Note that the position information of the user may be obtained not by the tracking process of the HMD 200 but by the process of directly tracking the user or a part such as the user's head. The information acquisition unit 102 may also acquire direction information and posture information of the user. For example, based on tracking information obtained by viewpoint tracking processing, direction information indicating the direction in which the user faces in real space is obtained. The information acquisition unit 102 also acquires posture information, which is motion information of the user. Posture information is information that identifies the movement of a user's hand, head, foot, or other part. For example, the information acquisition unit 102 acquires the user's posture information through a process called Kinect (registered trademark). Note that the position information acquired by the information acquisition unit 102 may be relative position information.

The display processing unit 120 (image generation unit) also generates an image displayed on the display unit 190 as a virtual space image. The virtual space image is, for example, an AR or MR image. The virtual space image may be a VR image. An AR image is an image that is displayed so as to be superimposed on a display object that constitutes a landscape in a real space. An MR image is an image generated by the MR technique, which mixes real and virtual spaces to create a new space in which the real and the virtual interact in real time. It is a concept that includes augmented reality and augmented virtuality. The VR image is, for example, an image that is generated so that the VR space expands all around the visual field of the user wearing the HMD 200 described later.

Also, the display processing unit 120 generates an image viewed from a virtual camera (given viewpoint) in the virtual space as a display image of the HMD 200 . For example, drawing processing of an image seen from a virtual camera set to the viewpoint (first-person viewpoint) of a character (user character, other user character) is performed. Alternatively, drawing processing of an image seen from a virtual camera set to a viewpoint (third-person viewpoint) that follows the character is performed. The image to be generated is desirably an image for stereoscopic viewing, such as an image for the left eye and an image for the right eye.

Further, in the present embodiment, virtual reality simulation processing is performed as game processing of a game played by a user. The virtual reality simulation process is a simulation process for simulating an event in the real space in a virtual space, and is a process for allowing the user to virtually experience the event. For example, the virtual user corresponding to the user in the real space and the user's mobile object such as the user's mobile object are moved in the virtual space, and processing is performed to allow the user to experience changes in the environment and surroundings accompanying the movement.

Note that the processing of the simulation system of the present embodiment in FIG. 1 may be realized by a processor and memory built into the HMD 200. For example, a processor built into the HMD 200 performs each process of this embodiment, and displays an image (virtual space image) generated by the process of this embodiment on the display unit 190 of the HMD 200 . Alternatively, the processing of the simulation system of this embodiment may be implemented by a home game device or commercial game device, or may be implemented by distributed processing between a home game device or commercial game device and a server system. . Alternatively, the simulation system of this embodiment may be realized by a processing device such as a PC installed in a facility, a processing device worn by a user, or distributed processing of these processing devices. For example, a user wears a jacket, and a processing device such as a backpack PC is attached to the back side of the jacket. Then, a processing device such as a backpack PC worn by the user executes each process of the present embodiment and generates an image (virtual space image) displayed on the display unit 190 . Alternatively, each processing of the present embodiment is executed or an image displayed on the display unit 190 is generated by distributed processing between the processing device worn by the user and the management processing device installed in the facility. Alternatively, the processing of the simulation system of this embodiment may be realized by a portable communication terminal that can be connected to a network, or by a portable communication terminal and a server system. For example, it may be realized by distributed processing between the mobile communication terminal and the server system.

2. HMD
Next, the HMD 200 used in this embodiment and the tracking process using the HMD 200 will be described.

FIG. 2 is an example of the HMD 200 used in this embodiment. The HMD 200 in FIG. 2 is a see-through (transmissive) glasses type HMD. Specifically, it is an optical see-through HMD. This see-through type HMD 200 enables display of virtual space images of MR and AR, for example.

The HMD 200 of FIG. 2 has a temple portion 240 and a goggle portion 242 . A speaker is built in the temple part 240 . The goggle section 242 is provided with a display device 243 and a holographic optical element 244 . The display device 243 is provided with a display unit composed of a microdisplay, a mirror, a prism, and the like. The display device 243 is provided with a display unit for the left eye and a display unit for the right eye, thereby realizing stereoscopic vision. Further, by providing a pair of microdisplays in each display unit for the left eye and the right eye, the position of the virtual image can be varied.

The display light from the display device 243 is guided to the front of the user US while being refracted inside the light guide plate of the holographic optical element 244 . The holographic optical element 244 refracts the display light toward the eyeball and sends the display light to the eyeball. This makes it appear as if there is a virtual image (reproduced image of the hologram) in front of the eyes. The light guide plate of the holographic optical element 244 is also called a combiner, and the combiner, which is a half mirror, makes it possible to see the real and virtual images of the outside world overlapped, realizing MR and AR.

The goggles section 242 is also provided with an RGB camera 246 , a depth camera 247 , and environment recognition cameras 248 and 249 . By using the RGB camera 246, it is possible to photograph the front direction of the user US. Depth information (depth information) in the front direction can be obtained by using the depth camera 247 . For example, the goggles section 242 is provided with an emitter (not shown) for a depth camera. An IR projector, which is an emitter, projects an infrared pattern onto an object in real space, the projected infrared pattern is read by an infrared camera, which is a depth camera 247, and depth information of the object in real space is obtained from the distortion of the pattern. get. Furthermore, by using the environment recognition cameras 248 and 249, the environment around the user US can be recognized. The goggles section 242 also incorporates an inertial measurement unit (IMU) configured by an acceleration sensor and a gyro sensor. The position and direction of the head of the user US are detected based on an image captured by the camera provided in the goggles section 242 and measurement information from the inertial measurement unit, etc., thereby realizing head tracking. It is also possible to acquire position information (viewpoint position information) and direction information (line-of-sight direction information) of the user US. By using the RGB camera 246, the depth camera 247, and the environment recognition cameras 248 and 249, real space recognition processing called space mapping is realized. By scanning the surroundings of the user in the real space using these cameras, recognition processing of the real space around the user is realized, and real space information is acquired. By using these cameras, it is also possible to obtain position information (relative position information), direction information (relative direction information), or posture information (motion information) of other users around the user US. For example, by acquiring posture information through processing such as Kinect (registered trademark), it becomes possible to detect what actions other users have taken. Alternatively, the motion of the user's own body part may be acquired as the user's posture information (motion information).

Another example of the HMD 200 is shown in FIG. The HMD 200 in FIG. 3 is a type of HMD that is worn so as to cover the entire visual field of the user. The display unit of the HMD in FIG. 3 is implemented by, for example, an organic EL display (OEL) or a liquid crystal display (LCD). For example, the display unit of the HMD 200 is provided with a first display or first display area set in front of the left eye of the user and a second display or second display area set in front of the right eye of the user. , and stereoscopic display is possible. When stereoscopic display is performed, for example, a left-eye image and a right-eye image having different parallaxes are generated, the left-eye image is displayed on the first display, and the right-eye image is displayed on the second display. do. Alternatively, the image for the left eye is displayed in the first display area of one display, and the image for the right eye is displayed in the second display area. The HMD 200 is also provided with two eyepieces (fisheye lenses) for the left eye and the right eye, which expresses a VR space that extends all around the user's visual field. Then, correction processing for correcting distortion caused by an optical system such as an eyepiece is performed on the image for the left eye and the image for the right eye.

Further, the HMD 200 in FIG. 3 is provided with an RGB camera 216, a depth camera 217, and environment recognition cameras 218 and 219. As shown in FIG. By using these cameras, a video see-through type HMD 200 is realized, and MR and AR can be realized. Further, by using these cameras, similar to the HMD 200 in FIG. 2, real space recognition processing, acquisition of position information, direction information, and posture information (motion information) of other users, acquisition of user posture information, and the like can be performed. can be realized.

In the HMD 200 of FIG. 3, the tracking process for acquiring the user's position information and direction information may be implemented by an inertial measurement unit (IMU) built in the HMD 200 and configured by an acceleration sensor and a gyro sensor. Alternatively, the HMD 200 is provided with a plurality of light receiving elements, and a measuring device (base station) having light emitting elements is provided around the user. Then, light such as laser light emitted from light-emitting elements (LEDs) of the measuring device (a plurality of measuring devices) is received by the light-receiving element of the HMD 200, so that tracking processing of the HMD 200 is performed, and user position information (viewpoint position information) and direction information (line-of-sight information) may be acquired. Alternatively, the HMD 200 is provided with a plurality of light emitting elements, and a measuring device having cameras (first and second cameras) around the user is provided. Then, the position of the head of the user US in the depth direction and the like are detected by capturing an image of the light from the light emitting element of the HMD 200 with the camera of the device for measurement. Further, the rotation angle (line of sight) of the head of the user US may be detected based on the detection information of the inertial measurement unit (motion sensor) provided in the HMD 200, and the tracking process of the HMD 200 may be realized.

Various modifications can be made to the tracking process for acquiring the user's position information, direction information, and the like. For example, the inertial measurement unit provided in the HMD 200 may be used to implement the tracking process by the HMD 200 alone, or the tracking process may be implemented by providing an external measuring device as described above. Alternatively, various viewpoint tracking processes such as known eye tracking, face tracking, or head tracking may be employed.

As the HMD 200, a type called a so-called smartphone VR or VR goggles may be used. In this type of HMD 200, the smart phone is housed in the goggle section of the HMD 200 so that the display section of the smart phone faces the user's eyes. A left-eye eyepiece lens and a right-eye eyepiece lens are built in the inside of the goggle section (VR goggles). The user views the image for the left eye and the image for the right eye displayed on the display unit of the smartphone through the eyepiece lens for the left eye and the eyepiece lens for the right eye, respectively, to appreciate the VR stereoscopic image. be able to. Tracking processing that identifies the user's viewpoint position and line-of-sight direction can be realized based on an inertial measurement unit (acceleration sensor, gyro sensor) or the like built into the smartphone. Real space recognition processing may be performed using a goggles unit or a camera provided in a smartphone.

3. Method of this Embodiment Next, the method of this embodiment will be described in detail. In the following description, the case where the technique of the present embodiment is applied to an attraction game in which a user wears an HMD and exterminates characters such as mosquitoes will be mainly described as an example. However, the method of the present embodiment can be applied to various games (virtual experience games, fighting games, competition games, RPGs, sports games, horror experience games, vehicle simulation games such as trains and airplanes, puzzle games, communication games, and music games. etc.) and can be applied to other than games. In the following description, a user moving object in virtual space corresponding to a user in real space is referred to as a user character. This user character may be a displayed object, or may be a virtual one that is not displayed as an object.

3.1 Description of Game First, the attraction game realized by this embodiment will be described. In this embodiment, real space information is acquired by performing recognition processing of the real space around the user, and a virtual space corresponding to the real space is generated based on the acquired real space information. For example, in FIG. 4A, objects such as a desk DK and a wall WL exist in real space. Real space information is obtained by performing recognition processing in which the real space is scanned by a camera provided in the HMD 200 shown in FIGS. 2 and 3 . Specifically, as shown in FIG. 4B, the real space information is acquired as wire frame data obtained by meshing the real space. This real space recognition processing is executed, for example, in real time, and the real space around the user is sequentially scanned to obtain real space information. For example, space data associated with IDs are sequentially acquired and accumulated and stored as real space information in the real space information storage unit 171 in FIG. This real space information also has depth information (depth value), and by using this depth information, for example, hidden surface removal processing between an object in virtual space and an object in real space is possible. Further, by generating a virtual space based on real space information, interaction between virtual objects (characters, placed objects) and real space objects (desk DK, wall WL) becomes possible. For example, hit processing (collision processing) between a virtual object and an object in the real space becomes possible, making it possible to realize unprecedented types of MR and AR (hereinafter simply referred to as MR).

FIG. 5 shows an image seen by the user in the optical see-through HMD 200 of FIG. In areas B1, B2, B3, and B4 in FIG. 5, scenery such as background objects in the real space appears to the user's eyes. A virtual space image generated by the display device 243 in FIG. 2 is displayed in the display area ARV. When the user shakes his or her face left and right, the display area ARV moves left and right as shown by DR1 and DR2. The display device 243 displays images of virtual space objects such as characters and placed objects. This makes it possible to realize MR in which the scenery of the real space of B1, B2, and B3 and the virtual space image of the display area ARV are mixed. Although FIG. 5 shows an example in which the display area ARV is narrow and the viewing angle is small, by improving various devices of the HMD 200, it is possible to widen the display area ARV and further expand the viewing angle. be.

FIG. 6 is an example of a play field FL used in the attraction game of this embodiment. In the attraction game of this embodiment, for example, stores in a shopping mall are installed in the play field FL of an indoor theme park. A user, for example, wears the HMD 200 in FIG. 2 on his or her head and searches the play field FL with friends or other companions. Specifically, the player starts at location P1, moves to location P2, exterminates a mosquito (character) appearing at location P3, and returns to location P1. After that, a ride-type attraction game as described in FIG. 14 is played.

In this embodiment, the camera provided in the HMD 200 shown in FIG. 2 is used to perform real space recognition processing, thereby generating a virtual space corresponding to the play field FL in the real space. In this case, as the user wearing the HMD 200 moves on the play field FL, the real space around the moving user is sequentially scanned and accumulated as real space information for updating.

When the user moves to the place P2 in FIG. 6, a poster PS serving as a marker is pasted on the wall in front of the user as shown in FIG. 7(A). By recognizing this poster PS (marker) with the camera of the HMD 200, an image IMA with a picture of a mosquito and a guide GDA are displayed. That is, the image IMA and the guide GDA are displayed in the display area ARV of FIG. The guide GDA points toward the location P3 in FIG. 6, and when the user directs his/her line of sight toward the location P3, the user sees a landscape as shown in FIG. 7(B). Then, as shown in FIG. 7B, a reference point RP is set at a location corresponding to the pillar PIL. For example, in the present embodiment, real space information is acquired through real space recognition processing, and the reference point RP is set based on user information such as user position information and the real space information. Specifically, recognition processing of the pillar PIL in the real space is performed by the camera of the HMD 200, and a virtual object corresponding to the pillar PIL in the real space is generated in the virtual space. For example, a virtual pillar object of mesh polygons is generated like the desk in FIG. 4B. Then, a reference point RP is set at a location corresponding to this pillar object. For example, when it is determined that the user's position is close to the pillar object based on the user's position information, the reference point RP is set for the pillar object. By doing so, it becomes possible to set the reference point RP at a location corresponding to the pillar PIL in the real space. It should be noted that there are cases where accurate coordinates cannot be acquired in real space recognition processing by the camera of the HMD 200 . For this reason, the absolute coordinates of the positions of the markers on the poster PS shown in FIG. 7A, for example, may be used to set accurate coordinates.

In this embodiment, as shown in FIG. 8A, an object of a mosquito character CH appears based on the set reference point RP. For example, a mosquito character CH is arranged and made to appear at an arrangement point (placement location) specified by the reference point RP. As a result, the image of the mosquito character CH is displayed in the display area ARV described with reference to FIG. 5, and it is possible to realize MR in which the mosquito appears as if it appeared at the position of the pillar PIL in the real space. That is, it is possible to display an image in which the landscape (pillar) in the real space and the image (mosquito) in the virtual space are mixed. In this embodiment, the reference point RP is set for the virtual pillar object generated by the real space recognition process. Therefore, it is possible to make the mosquito character CH appear from the point corresponding to the pillar PIL in the real space without using an AR marker or the like. In addition, in FIG. 8A, a guide GD1 with the words "Coming from the front" is also displayed. This guide GD1 is an arranged object in the virtual space, and the arrangement point of the arranged object of this guide GD1 is also set based on the reference point RP.

In FIG. 8B, when the user performs an action of tapping with both hands as described later with reference to FIGS. 9A to 10, an attack effect AG (air cannon) is displayed, and a mosquito character Hit CH to exterminate mosquitoes. When the mosquitoes are successfully exterminated, a guide GD2 is displayed with the words "I did it, let's go back". Also displayed is a guide GD3 for instructing the direction in which the user returns. The placement positions of the placement objects of these guides GD2 and GD3 are also set based on the reference point RP.

In FIGS. 9A to 10, the user (US) wears the HMD 200 of FIG. 2 on his head. In addition, the user holds belongings BL and BR in his/her hands HL and HR. The belongings BL and BR are realized by, for example, large hand-shaped paper fans. An arm sensor SE is attached to the hand HL. The arm sensor SE is an armband type motion sensor (gesture sensor), and incorporates an acceleration sensor, a 3-axis gyro sensor, a myoelectric potential sensor, and the like. By attaching the arm sensor SE to the hand HL, it becomes possible to detect the movement of the arm and the opening and closing of the palm. As shown in FIGS. 9(A) and 9(B), when the user performs a snapping action with the hands HL and HR, the movement of this action is detected by the acceleration sensor of the arm sensor SE. This arm sensor SE corresponds to the detection unit 162 in FIG. 1, and the action of snapping with the hands HL and HR is a given input performed by the user in this embodiment. By this action, the attack effect AG is displayed, and the mosquito character CH is exterminated, extinguished or destroyed as shown in FIG.

In the case of FIGS. 9A to 10, a user character USC (in a broad sense, a user moving body ) and the character CH, and the hit determination process (collision determination process) for the character CH is performed using this hit volume HV. That is, when the distance between the user character USC and the character CH is long, the hit determination process is performed by the method shown in FIGS. 19A and 19B.

On the other hand, in FIGS. 11A and 11B, the hit volume HV is set based on the position of the user character USC, as will be described later with reference to FIGS. 17A and 17B. Hit determination processing for the character CH is performed using the volume HV. That is, when the distance between the user character USC and the character CH is short, the hit determination process is performed by the method shown in FIGS. 17A and 17B. In this case, for example, the belonging objects corresponding to the belongings BL and BR may appear in the virtual space, and the size of the belonging objects may be increased each time the user clapping the hands HL and HR. As a result, even the mosquito character CH, which is located relatively far away, can be defeated by hitting the possessed object.

After exterminating the mosquitoes appearing at location P3 in FIG. 6 as described in FIGS. 8A to 11B, the user returns to location P1, which is the first starting point. In this case, in this embodiment, as shown in FIGS. 12A and 13B, placement objects OB1 to OB4, which serve as guides for guiding and moving the user from location P3 to location P1, are placed in the virtual space. to be placed. Specifically, a reference point RP is set for the location P1, and the placed objects OB1 to OB4 are arranged based on the set reference point RP. In this case, in the present embodiment, real space recognition processing is performed as described with reference to FIGS. 4A and 4B, and real space information is acquired. Then, the reference point RP is set based on the user information such as the user's position information and the acquired real space information. Specifically, real space recognition processing is performed by the camera of the HMD 200, and virtual objects corresponding to objects (for example, an entrance, a wall, and a door) placed at the location P1 are generated in the virtual space. A reference point RP is set at the location of this virtual object. By doing so, the reference point RP can be appropriately set with respect to the location P1, and the placement object OB1 serving as a guide for guiding the user is placed between the location P1 where the reference point RP is set and the user. ~ OB4 can be placed appropriately.

In this embodiment, the display modes of the placed objects OB1 to OB4 are changed according to the positional relationship between the user character (user moving body) in the virtual space corresponding to the user and the placed objects OB1 to OB4.

For example, in FIG. 12A, a placement object OB1 placed at a position far from the reference point RP is displayed in a display state. That is, the image of the placed object OB1 is displayed in the display area ARV in FIG. For example, the text "this is it" is written on this placement object OB1.

After that, when the user moves toward and approaches the reference point RP of the location P1, the next placed object OB2 is set in the display state and displayed as shown in FIG. 12(B). The placed object OB2 is an object placed at a position closer to the reference point RP than the placed object OB1. For example, characters such as "this way, this way" are written on this placement object OB2. In this case, the placement object OB1 changes from an opaque state to a translucent state, and then changes to a non-display state (transparent state).

When the user moves closer to the reference point RP of the location P1, the next placed object OB3 is set to the display state and displayed as shown in FIG. 13(A). Placement object OB3 is an object placed at a position closer to reference point RP than placement objects OB1 and OB2. This placement object OB3 is written with, for example, the characters "It's about time." In this case, the placement object OB2 changes from an opaque state to a translucent state, and then changes its display mode to a non-display state.

When the user moves further toward the reference point RP of the location P1 and comes close to the location of the reference point RP, the next placed object OB4 is set to be displayed as shown in FIG. 13B. be done. Placement object OB4 is an object placed at a position closer to reference point RP than placement objects OB1, OB2, and OB3. For example, a character "Welcome home" is written on this placement object OB4. In this case, the placement object OB3 changes from an opaque state to a translucent state, and then changes its display mode to a non-display state.

As described above, in this embodiment, by using the placed objects OB1 to OB4 set based on the reference point RP, the user is successfully guided to the location P1 of the reference point RP and moved.

The user who has returned to the place P1 plays a ride-type attraction game whose entrance is provided at the place P1. FIG. 14 shows the play field FLR of this ride type attraction. A course CS is set in the play field FLR. The users US1 and US2 board the ride enclosures RD1 and RD2, and the ride enclosures RD1 and RD2 move along the course CS. Mosquitoes appear and attack the user in the areas M1 to M14 of the play field FLR. The user performs the action of clapping his hands as described in FIGS. 9(A) to 11(B) to exterminate the attacking mosquitoes. Also, at a place on the far side of the play field FLR, an event EV1 occurs in which the object OBC on the ceiling collapses and falls. This ceiling object OBC is represented by a virtual space object, and a virtual space image of the falling ceiling is displayed on the HMD 200 . In areas M13 and M14, an event EV2 occurs in which a large number of mosquitoes attack, and finally an event EV3 occurs in which a giant mosquito boss BSF appears. This mosquito boss BSF is an object in the real space, and an effect is performed in which an effect image in the virtual space is superimposed and displayed on this boss BSF.

FIGS. 15A and 15B show an example of a mosquito appearance event on the playfield FLR. In FIGS. 15A and 15B, holes HE1, HE2, and HE3 appear in the real space, and mosquito characters CH1, CH2, and CH3 pop out from the holes HE1, HE2, and HE3. is realized. The user can then perform the action of clapping his hands as described in FIGS. 9A to 11B to exterminate the attacking mosquitoes. Here, the images of the holes HE1, HE2, and HE3 are virtual space images, and the images of the characters CH1, CH2, and CH3 are also virtual space images, and these virtual space images are displayed superimposed on the objects in the real space. MR is realized.

In this embodiment, the objects of the holes HE1 to HE3 and the objects of the characters CH1 to CH3 are arranged based on the reference point RP. For example, real space recognition processing as shown in FIGS. 4A and 4B is performed on real space objects arranged in the play field FLR of FIG. As a result, a virtual space is generated in which virtual objects corresponding to the objects in the real space of FIGS. 15A and 15B are arranged, and a reference point RP is set in the generated virtual space. Then, when the user character corresponding to the user who moves on the ride housing approaches the position of the reference point RP, as shown in FIG. , and the characters CH1 to CH3 also appear from the holes HE1 to HE3. By doing so, it is possible to generate MR images as shown in FIGS. 15A and 15B without using an AR marker or the like.

In FIG. 16, the guide GD is displayed. This guide GD is a placement object that indicates the direction of the existing position of the character CH. For example, when the display area ARV of the virtual space image of the HMD 200 is narrow (when the viewing angle is small) as shown in FIG. cannot be displayed. In this regard, if the guide GD as shown in FIG. 16 is displayed, for example, when the user looks to the right, the display area ARV of the HMD 200 shown in FIG. It will be displayed in the area ARV. By arranging and displaying the placement object that serves as the guide GD in this way, the user can defeat the mosquito character CH by performing an action of clapping his hands while directing his gaze toward the character CH. .

Note that the process of notifying the existing position of the character CH is not limited to the process of using a display object such as a guide GD as shown in FIG. For example, the position of the character CH may be notified to the user using three-dimensional sound from a sound output unit (headphones, speakers) provided in the HMD 200 or the like. For example, when the character CH exists in the right direction, the sound generated by the character CH (for example, the sound of "boom") can be heard from the right direction by performing three-dimensional sound processing and outputting it from the sound output unit. good. By doing so, even when the HMD 200 having a narrow display area ARV and a small viewing angle as shown in FIG. 5 is used, the existing position of the character CH can be appropriately notified to the user.

3.2 Setting of Reference Point and Hit Volume As described above, in this embodiment, as described with reference to FIGS. Get real space information. It also acquires user information including user location information. A virtual space corresponding to the real space is generated based on the acquired real space information. For example, a virtual space is generated in which a polygon mesh object as shown in FIG. 4B is arranged. 8(A), 8(B), 12(A) to 13(B), 15(A), and 15(B), for example, user information (location information) and actual Based on the space information, a reference point RP is set in the virtual space, and based on the reference point RP, the character object and the arranged objects are arranged in the virtual space, and an image including the image of the character and the arranged object is displayed on the display unit 190. to display. For example, a virtual space image including images of characters and placed objects is displayed on the display unit 190 (display area ARV) of the HMD 200 .

A hit volume is set based on the position of a user character (a user moving body in a broad sense) in a virtual space corresponding to the user, and when the user performs a given input, the positional relationship between the hit volume and the character is determined. In response, the character is processed.

For example, FIG. 17(A). In FIG. 17(B), based on real space information acquired by performing real space recognition processing (scanning processing, mapping processing), a virtual image corresponding to a background object (pillar, wall, door, etc.) in the real space is displayed. A spatial background object MRS (for example, an object composed of mesh polygons) is generated. Also, a reference point RP is set based on real space information and the like. By generating the background object MRS corresponding to the background object in the real space, it becomes possible to set the reference point RP at the corresponding location of the background object in the real space. Then, the character CH is arranged based on this reference point RP. For example, the character CH is caused to appear from an appearance point specified based on the reference point RP.

The reference point RP is set based on the real space information, and is a point or place for setting the placement positions of the character (CH) and the placement objects (OB1 to OB4). The reference point RP may be the intentional position (appearance position) of the character or the placed object itself. Since the reference point RP is set based on the real space information, it reflects the position, shape, etc. of the object in the real space. In addition, by setting the reference point RP based on user information including the user's position information and direction information, for example, the user's movement, facing direction, approach, etc. are detected, the reference point is set, and characters and placed objects appear. It becomes possible to let For example, the reference point RP can be used as a pseudo marker in MR.

Further, in this embodiment, a hit volume HV (hit area) is set based on the position PS of the user character USC corresponding to the user in the real space. For example, in FIGS. 17A and 17B, the hit volume HV is set in front of the position PS of the user character USC. Then, when the user performs a given input, the positional relationship between the hit volume HV and the character CH is determined. Taking FIGS. 9A to 11B as an example, it is determined that the user has performed a given input when the user performs an action of clapping their hands. Then, the positional relationship between the hit volume HV and the character CH is determined at the timing when the given input by the action of clapping hands is performed.

For example, when the positional relationship between the hit volume HV and the character CH is as shown in FIG. 17A, the user performs an action of clapping his hands. In this case, since the position of the character CH is not within the hit volume HV, it is determined that the attack by the user's hand clapping did not hit the character CH.

On the other hand, when the positional relationship between the hit volume HV and the character CH is as shown in FIG. In this case, since the position of the character CH is within the hit volume HV, it is determined that the attack by the user's hand clapping hits the character CH. Then, as the processing for the character CH, the character CH is erased, destroyed, or the like. By doing so, as shown in FIGS. 11(A) and 11(B), the character CH, which is approaching directly to the user, is attacked, destroyed or destroyed. processing becomes possible.

Note that in FIGS. 17A and 17B, the second hit volume may be set based on the position of the character CH. Then, the hit determination process may be realized by performing the intersection determination process between the hit volume HV set based on the position PS of the user character USC and the second hit volume set based on the position of the character CH. .

As described above, in the present embodiment, a virtual space corresponding to the real space is generated based on real space information acquired based on real space recognition processing, and the reference point RP is set. This makes it possible to set the reference point RP at a position corresponding to the real space. Further, by setting the reference point RP using user information including the position information of the user, for example, when a user character corresponding to the user approaches, the reference point RP is set and specified by the reference point RP. The character CH can be arranged and made to appear at a place where the character CH appears. In this embodiment, the positional relationship between the character CH and the hit volume HV set based on the position of the user character USC is determined, and the character CH is processed. Therefore, it is possible to realize an interaction between the user and the virtual character CH appearing based on the reference point RP, and it is possible to realize hit determination processing for determining whether or not the user's attack hit. Therefore, it is possible to set a virtual space corresponding to the real space, arrange the character CH, and realize interaction with the character CH based on the user's input, thereby realizing an interactive game using real space information. becomes possible. In other words, it is possible to realize a simulation system that can effectively utilize real space information and realize favorable interaction between a user and a character.

Also, in this embodiment, as the hit volume HV, a hit volume set for a user's body part or a user's belongings can be used.

For example, in FIG. 18A, the hit volume HV is set for the user's hand HL. For example, a hit volume HV that includes the user's hand HL is set. When the user moves the hand HL, the hit volume HV moves accordingly. The movement of the user's hand HL can be detected using an arm sensor SE (detector in a broad sense) worn by the user. For example, the motion of the hand HL is detected using an acceleration sensor or a 3-axis gyro sensor built in the arm sensor SE, and the hit volume HV is moved so as to follow the motion of the hand HL.

Also, in FIG. 18B, a hit volume HV is set for the user's belongings BL. For example, a hit volume HV that includes the user's belongings BL is set. When the user moves the item BL, the hit volume HV moves accordingly. The movement of the possessed item BL is detected using an arm sensor SE or the like, and the hit volume HV is moved so as to follow the movement of the possessed item BL. In this case, a motion detection sensor such as an arm sensor SE may be attached to the property BL to detect the motion of the property BL. In addition, in FIG. 18B, the belonging BL has a shape of a large hand, but various shapes can be adopted as the belonging BL. For example, it may be a possession BL shaped like a sword, a stick, an ax or a gun.

If the hit volume is set for the user's body part or belongings in this way, when the user moves that body part or belongings in real space, the hit volume will move accordingly. Therefore, it is possible to perform hit determination processing and the like using the hit volume that moves in conjunction with the movement of the user's body part or belongings, thereby improving the user's sense of virtual reality.

Further, in this embodiment, it is determined whether or not the user has made a given input based on information from the detection unit 162 that detects the user's movement. For example, as shown in FIGS. 9A to 11B, the detection unit 162 is used to detect whether or not the user has performed a given input by an action such as clapping. For example, the detection unit 162 can be realized by the arm sensor SE shown in FIGS. 18(A) and 18(B). Specifically, it is detected using an acceleration sensor built into the arm sensor SE. FIG. 18C shows the detection result of the acceleration sensor when the user performs a given input. When the user performs an action of clapping his hands, the moving hand stops due to sudden acceleration, so the acceleration sensor obtains a detection result (actually a negative direction acceleration) as indicated by A1 in FIG. 18(C). . Based on this detection result, it is determined that the user performed a given input by clapping his hands. Then, the positional relationship between the hit volume HV and the character CH may be determined at the timing at which it is determined that a given input has been made.

By doing so, it is possible to detect the actual movement of the user in the real space, determine whether or not the user has made a given input, and implement hit determination processing using the hit volume or the like. Become. As a result, the user can attack the character by actually moving the body part or belongings, thereby further improving the user's sense of virtual reality.

Note that the processing for detecting whether or not the user has performed a given input is not limited to processing using such an acceleration sensor. For example, motion sensors other than acceleration sensors (gyro sensors, etc.) can be used to detect movements when the user performs input, and external devices such as external cameras can be used to detect the movements of the user's body parts and belongings. It may be detected to detect whether the user has made a given input. Alternatively, a user's operation input using the operation unit 160 such as a game controller may be detected to detect whether or not the user has performed a given input.

Further, in this embodiment, a hit volume is set between a user character in a virtual space corresponding to the user and a reference point, and when the user performs a given input, the positional relationship between the hit volume and the character changes. In response, the character is processed.

For example, in FIG. 19A, a hit volume HV is set between the user character USC and the reference point RP. For example, the hit volume is set at the middle point of the line connecting the user character USC and the reference point RP. When the user character USC is directing the line of sight toward the reference point RP, the hit volume HV is set on the extension of the line of sight, for example. Then, when the user performs a given input, the positional relationship between the hit volume HV and the character CH is determined. For example, as shown in FIGS. 9(A) to 11(B), the positional relationship between the hit volume HV and the character CH is determined at the timing when the given input by the action of clapping hands is performed.

Assume that the user performs an action of clapping when the positional relationship between the hit volume HV and the character CH is as shown in FIG. 19A. In this case, since the position of the character CH is not within the hit volume HV, it is determined that the attack by the user's hand clapping did not hit the character CH.

On the other hand, it is assumed that the user performs an action of clapping his hands when the positional relationship between the hit volume HV and the character CH is as shown in FIG. 19B. In this case, since the position of the character CH is within the hit volume HV, it is determined that the attack by the user's hand clapping hits the character CH. Then, as the processing for the character CH, the character CH is erased, destroyed, or the like. By doing so, as shown in FIGS. 9(A) to 11, it is possible to attack the character CH at a position distant from the user to eliminate or destroy the character CH. Become.

Note that in FIGS. 19A and 19B, a second hit volume is set based on the position of the character CH, and the hit volume HV set between the user character USC and the reference point RP is The hit determination process may be realized by performing the intersection determination process with the second hit volume set based on the position. Also, when the character CH exceeds the range of the hit volume HV from the position shown in FIG. good too. Then, a performance process may be performed using an effect indicating that the user character USC is damaged, hit points of the user character USC are reduced, or that an attack has been received.

According to the method of FIGS. 19A and 19B, even if the reference point RP or the position of the character CH arranged based on the reference point RP is far from the user character USC, the character CH It is possible to interact with In other words, by setting the hit volume HV between the reference point RP and the user character USC, it is possible to perform hit processing on the character CH located far away, and favorable interaction between the user and the character in MR. Become.

Further, in this embodiment, when the user character USC and the reference point RP are in the first distance relationship, the hit volume HV is set based on the position of the user character USC, and when the user performs a given input, the hit volume HV is set. Next, the character CH is processed according to the positional relationship between the hit volume HV and the character CH.

For example, as shown in FIGS. 17(A) and 17(B), when there is a first distance relationship in which the distance L1 between the user character USC and the reference point RP is short, a hit is performed based on the position of the user character USC. A volume HV is set and a hit determination process is performed. By doing so, when the character CH appears at a place close to the user character USC, it is possible to realize an interaction such as the attack of the user character USC hitting the character CH. For example, as shown in FIGS. 11(A) and 11(B), the character CH in the immediate vicinity of the user is attacked by actually hitting the character CH with the user's belongings BL, BR or the hands HL, HR. It is possible to process such as extinguishing or destroying. That is, it is possible to perform image representation of MR such that the character CH, which is an object in the virtual space, is hit with the belongings BL, BR and the hands HL, HR, which are objects in the real space.

It should be noted that a part object such as a user's hand, a part object corresponding to a belonging, or a belonging object may be displayed in the virtual space. For example, a part object or a possession object is displayed as part of the user character. Then, for example, each time the user performs an action such as clapping hands, the size of the part object or the possessed object may be gradually increased. By doing so, even when a large number of characters such as mosquitoes are generated (for example, event EV2 in FIG. 14), it is possible to perform a game effect such as attacking these large numbers of characters with enlarged hands. become.

On the other hand, when the user character USC and the reference point RP are in the second distance relationship, a hit volume HV is set between the user character USC and the reference point RP, and when the user performs a given input, Next, the character CH is processed according to the positional relationship between the hit volume HV and the character CH.

For example, as shown in FIGS. 19A and 19B, when there is a second distance relationship in which the distance L2 between the user character USC and the reference point RP is long, the distance between the user character USC and the reference point RP is A hit volume HV is set between and a hit determination process is performed. In this way, even when the character CH appears at a place far away from the user character USC as shown in FIGS. 9A to 10, the attack of the user character USC hits the character CH. It will be possible to realize interactions such as For example, as shown in FIGS. 9A to 10, an attack effect AG generated by the action of clapping the user's hands hits the character CH located at a distant distance, thereby erasing or destroying the character CH. etc. can be processed.

In the present embodiment, character disappearing processing, character destruction processing, character display mode changing processing, or character notification processing are performed as processing for the character.

For example, in FIG. 20A, as the processing for the character CH, the character CH is destroyed, destroyed, or changed in display mode. For example, when the character CH is positioned within the hit volume HV as shown in FIGS. is performed by the user, the disappearance processing, destruction processing, or change processing of the display mode of the character CH is performed. The character CH disappearing process is a process of hiding the character CH as a virtual space image. For example, remove the character CH from the list of displayed objects. Destruction processing of the character CH can be realized, for example, by switching the object of the character CH to an object representing the destroyed state. Alternatively, the display processing of an image effect representing destruction may be performed as the destruction processing of the character CH. The process of changing the display mode of the character CH is, for example, a process of changing the color, luminance, translucency, texture, or the like of the character CH. For example, the display mode is changed to indicate that the character CH has been attacked by the user. Alternatively, as the processing for the character CH, notification processing using an effect sound or the like as shown in FIG. 20B may be performed. For example, a sound effect is output to inform that the character CH has been attacked. In this case, three-dimensional sound processing is performed so that the effect sound can be heard from the position where the character CH exists. Alternatively, as the processing for the character CH, notification processing by vibration using a vibration device may be performed. For example, when it is determined that the attack hits the character CH, the vibration device incorporated in the game controller held by the user is vibrated. Alternatively, the play field FL of FIG. 6 or the play field FLR of FIG. 14 is provided with a sensory device such as a vibrating device or an air cannon, and the sensory device is used to perform a process of notifying that the character CH has been attacked. may

Further, for example, when the user's attack fails, an effect process or notification process may be performed to notify the failure of the attack. For example, an effect image representing the failure of an attack is displayed, or an effect sound is output. For example, when the character CH is positioned within the hit volume HV as shown in FIGS. 17(B) and 19(B), if the user does not clap his/her hands, an effect is given to indicate that such an attack has failed. Perform processing and notification processing. Alternatively, in FIG. 19(B), when the character CH exceeds the range of the hit volume HV and approaches the user character USC, effect processing or notification processing is performed to notify that the attack has failed. good too.

3.3 Setting of Arranged Objects As described above, in the present embodiment, real space information obtained by performing recognition processing of the real space around the user and user information including the user's position information are acquired, and the real space A virtual space corresponding to the real space is generated based on the information. For example, as shown in FIGS. 21A to 22B, a virtual space background object MRS corresponding to a real space background object is generated and arranged in the virtual space. A reference point RP is set in the virtual space based on user information including user position information and real space information, and a placement object is arranged based on the reference point RP. In FIGS. 21A to 22B, placement objects OB1, OB2, OB3, and OB4 are placed based on the reference point RP. For example, a plurality of placed objects OB1 to OB4 are placed between the reference point RP and the user character USC. Then, an image including the image of the arranged object is displayed on the display unit 190 . For example, a virtual space image including an image of an arranged object is displayed on the display unit 190 (display area ARV) of the HMD 200 .

In this embodiment, the display mode of the placed object is set according to the positional relationship between the user character (user moving body) in the virtual space corresponding to the user and the placed object. The display mode of the arranged object is the color, brightness, translucency, texture, display content, or the like of the arranged object. The display contents of the layout object are instruction contents represented by characters, symbols, graphics, or the like drawn on the layout object. For example, the display content is the content of a guide or warning presented to the user. For example, the display contents are "here you go", "here you go", "I'll be there soon", "Welcome back", etc. written in the placement objects OB1-OB4 described in FIGS. 12A-13B. This is the content of the guide to the user.

For example, in FIG. 21A, it is determined that the user character USC and the placement object OB1 are in a close positional relationship. For example, it is determined that the distance between the user character USC and the placed object OB1 is equal to or less than a given threshold distance. In this case, the display mode of the placed object OB1 is set to the display state. For example, the translucency (α value) of the placement object OB1 is set to a high value, and the display state is set to an opaque state. Further, the display content of the placement object OB1 is set to a guide display that says "this is it". As a result, it becomes possible to display the placed object OB1 in a display mode as shown in FIG. 12(A).

On the other hand, in FIG. 21B, it is determined that the user character USC and the placed object OB2 are in a close positional relationship. For example, it is determined that the distance between the user character USC and the placed object OB2 is equal to or less than a given threshold distance. In this case, the display mode of the placed object OB2 is set to the display state. For example, the translucency of the placement object OB2 is set to a high value, and the opaque display state is set. The display content of the placement object OB2 is set to guide display with the display content of "this way, this way". As a result, it becomes possible to display the placed object OB2 in a display mode as shown in FIG. 12(B). Note that the display mode of the placed object OB1 is changed to a non-display state. For example, the translucency (α value) of the placed object OB2 is gradually changed to a lower value to gradually change to a non-display state, which is a transparent state.

In FIG. 22(A), it is determined that the user character USC and the placed object OB3 are in a close positional relationship. In this case, the display mode of the placed object OB3 is set to the display state. Further, the display content of the placement object OB3 is set to guide display with the display content of "It's about time." As a result, it is possible to display the placed object OB3 in a display mode as shown in FIG. 13(A). Note that the display mode of the placed object OB2 is changed to a non-display state.

In FIG. 22(B), it is determined that the user character USC and the placement object OB4 are in a close positional relationship. In this case, the display mode of the placed object OB4 is set to the display state. The display content of the placed object OB4 is set to "Welcome back". As a result, it is possible to display the placed object OB4 in a display mode as shown in FIG. 13B. Note that the display mode of the placed object OB3 is changed to a non-display state.

As described above, in this embodiment, the display mode of the placed object is set according to the positional relationship between the user character and the placed object. By doing so, it is possible to display the placed object in an appropriate display mode according to the positional relationship between the user character and the placed object.

For example, when the user character USC approaches the placed object OB1 as shown in FIG. 21A, the placed object OB1 is set in a display state, or a guide is displayed with the display content "This way!". By doing so, it is possible to notify the user that the reference point RP exists in the direction of the placement object OB1. For example, the reference point RP is set with respect to the target TG of the guide to the user. The target TG is the guide's destination or object. For example, in this embodiment, it is necessary to move the user who exterminated the mosquitoes at the place P3 in FIG. 6 to the place P1 where the entrance of the ride-type attraction in FIG. 14 is installed. In FIG. 21A, this place P1 is set as the target TG, which is the destination. By displaying the placement object OB1 in the display modes shown in FIGS. 21A and 12A, it is possible to appropriately guide the user to move in the direction of the target TG, which is the destination. become.

As described above, in this embodiment, the reference point RP is set for the target TG of the user's action, and the placement object is arranged based on the reference point RP set for the target TG. For example, the placement object of this embodiment is a placement object that guides the movement of the user, or a placement object that warns, notifies, or instructs the user. When the placement object is a placement object that guides the movement of the user, setting the reference point RP for the target TG guides the user's behavior such as moving with respect to the target TG, for example. becomes possible. Further, when the placed object is a placed object that warns the user, it is possible to warn the user about the target by using the placed object. Further, when the placed object is a placed object that notifies the user, the situation of the target TG can be notified by using the placed object. In addition, when the placement object is a placement object that instructs the user, it is possible to instruct the user's behavior with respect to the target by using the placement object.

Also, as shown in FIG. 21B, when the user character USC approaches the placed object OB2, the placed object OB2 is set in a display state, or a guide display with the display contents "this way, this way" is displayed. By doing so, it is possible to appropriately inform the user who has passed the placed object OB1 that the reference point RP exists in the direction of the placed object OB2. That is, by displaying the placed object OB2 in the display modes shown in FIGS. 21B and 12B, the user is appropriately guided to move in the direction of the destination or target TG. becomes possible.

Similarly, when the user character USC approaches the placed objects OB3 and OB4 as shown in FIGS. ” and “Welcome back” are displayed as guides. By doing so, it is possible to appropriately inform the user who has passed the placed object OB2 that the reference point RP exists in the direction of the placed objects OB3 and OB4.

As described above, according to the present embodiment, it is possible for a user moving in the real space to easily arrange the placed objects serving as the guide display in the virtual space generated corresponding to the real space. It is possible to smoothly guide the vehicle to its destination.

Also, in this embodiment, a first placement object and a second placement object are placed as placement objects. Then, when the first placed object and the second placed object are positioned in the direction of the reference point of the user character, the first placed object is set to the displayed state, and the second placed object is set to the non-displayed state. set. On the other hand, when the first placed object is positioned in the direction opposite to the direction of the user character's reference point side, and the second placed object is positioned in the direction of the user character's reference point side, the first placed object is set to the hidden state, and the second placed object is set to the displayed state.

For example, in FIGS. 21A and 21B, placed objects OB1 and OB2 (first and second placed objects) are placed. Then, as shown in FIG. 21A, when the placed objects OB1 and OB2 are positioned in the direction DR on the side of the reference point RP of the user character USC, the placed object OB1 is set to be displayed and the placed object OB2 is hidden. state. The placed objects OB3 and OB4 are also set to the non-display state. In this way, as shown in FIG. 12A, only the placed object OB1 is displayed in the field of view of the user, and the placed object OB2 and the like are not displayed. become. For example, when the display area ARV of the HMD 200 is narrow and the viewing angle is small as shown in FIG. 5, if a large number of placed objects are displayed in the display area ARV, the display becomes difficult for the user to see, causing user confusion. There is a risk. In this regard, by displaying only the placement object OB1 and not displaying the other placement objects OB2 and the like, it is possible to realize guide display that is easy for the user to see even in a narrow display area ARV.

On the other hand, as shown in FIG. 21B, the placed object OB1 is positioned in the direction opposite to the direction DR of the user character USC on the side of the reference point RP, and the placed object OB2 is positioned in the direction DR of the user character USC on the side of the reference point RP. When positioned, the placed object OB1 is set to the non-display state, and the placed object OB2 is set to the displayed state. That is, the placed object OB1 through which the user character USC has passed is set to a non-display state. Specifically, by gradually decreasing the translucency of the placement object OB1, the placement object OB1 becomes gradually transparent. Then, the placed object OB2, which was in the non-display state in FIG. 21A, is set to the display state. Specifically, by gradually increasing the translucency of the placed object OB2, it becomes gradually opaque. By doing so, it is possible to appropriately display the placed object OB2, which is the guide display following the placed object OB1, to the user as shown in FIG. 12B.

Similarly, in the cases of FIGS. 22A and 22B as well, when the placed objects OB3 and OB4 are positioned in the direction DR on the side of the reference point RP of the user character USC, the placed object OB3 is set to the display state. and the placed object OB4 is set to the non-display state. On the other hand, when the placed object OB3 is positioned in the direction opposite to the direction DR on the side of the reference point RP of the user character USC, and the placed object OB4 is positioned in the direction DR on the side of the reference point RP of the user character USC, the placed object OB3 is set to the non-display state, and the placed object OB4 is set to the display state.

By doing so, it is possible to set the display mode of the placed object that provides a guide display that is easier for the user to see.

Note that the processing for setting the display mode of the placed object according to the positional relationship between the user character and the placed object is not limited to the processing described with reference to FIGS. is. For example, in FIGS. 21(A) to 22(B), as the display mode of the placed object, the translucency and display contents of the placed object are changed. For example, for a placed object in the direction of the reference point RP (TG) (placed object in the direction to guide the user), the translucency is increased (made opaque) or guided toward the reference point RP (TG). It is set in the display contents. On the other hand, for example, as the display mode of the arranged object, the color, brightness or texture of the arranged object may be changed. For example, for an arranged object in the direction of the reference point RP, a color with high visibility or discrimination is set, brightness is increased, or texture with high visibility or discrimination is mapped. On the other hand, an arranged object in a direction different from the direction of the reference point RP is set to a color with low visibility or low identifiability, lowered in brightness, or mapped with a low visibility or low identifiability texture.

Also, various modifications can be made to the setting of the display contents of the arranged objects. For example, in FIG. 23, arrow symbols (graphics) are drawn for the placement objects OB1 to OB4, and the direction of the arrow indicates the direction toward the reference point RP (TG). In FIG. 23, the closer the placed object is to the reference point RP, the larger the arrow. For example, the size of the arrow drawn on the placed object OB4 closest to the reference point RP is the largest, and the size of the arrow drawn on the placed object OB1 farthest from the reference point RP is the smallest. In this way, the change in the display content, which is the display mode of the placed object, may be realized by changing the size or shape of the symbol or figure.

In addition, in the present embodiment, display priorities are set for objects in the virtual space. Then, when the placed object and another object are placed so as to overlap each other as viewed from the user character, the display mode of the placed object is set based on the order of priority. Here, the other object may be the second placed object, or may be an object of a different type than the placed object.

For example, in FIG. 24(A), an arrangement object OB is arranged and displayed, in which the characters "this way" are written and an arrow indicating the direction of the reference point (destination) is drawn. For example, a placement object OB is displayed in the display area ARV of the HMD 200 in FIG. Then, in FIG. 24B, an event that the ceiling collapses has occurred. For example, in the ride-type attraction game of FIG. 14, an event EV1 occurs in which the object OBC on the ceiling collapses. In FIG. 24A, fragments of the ceiling are displayed as objects OBC1 to OBC5 due to the collapse of the ceiling. These objects OBC1 to OBC5 are objects generated by events and other objects. In this case, the display priority of the objects OBC1 to OBC5 is set to a higher priority than the placement object OB, which is the guide display. In FIG. 24B, the fragment object OBC1 and the placement object OB are arranged to overlap each other when viewed from the viewpoint of the user character (user's viewpoint). In this case, the display mode of the placed object OB is set based on the order of priority. For example, in FIG. 24B, an object OBC1 (other objects in a broad sense) with a high display priority is set to the display state, while an arranged object OB with a low display priority is set to the non-display state. set to Alternatively, the placement object OB may be set in a translucent state. Alternatively, the arranged object OB may be set to a color with low visibility or low recognizability, the luminance may be lowered, or a texture with low visibility or low identifiability may be mapped.

In this way, the important object OBC1 can be preferentially displayed to the user in the event of FIG. 24(B). For example, even when the depth value (depth value) is set so that the placed object OB is closer to the front than the object OBC1, the object OBC1 is preferentially displayed. For example, when the display area ARV of the HMD 200 is narrow as shown in FIG. 5, if a large number of arranged objects are displayed, the display becomes extremely difficult for the user to see. In this regard, if the object OBC1 is preferentially displayed as shown in FIG. 24B, such a situation can be prevented from occurring.

It should be noted that the setting of the priority order for the arranged objects may be performed for a plurality of arranged objects having different display contents (guide contents). In this case, the display mode of the placed objects may be set so that the placed objects having the display contents with the higher priority are displayed with higher priority to the user. For example, it is assumed that the display content of the second placed object, which is another object, has a higher priority for the user than the display content of the placed object. For example, assume that a second placed object, which is another object, is a placed object for warning the user (for example, warning that there is a risk of colliding with an object). It is also assumed that the placement object is a placement object for guiding the user. In this case, the display content (warning) of the second placed object has a higher priority than the display content (guide) of the placed object. At this time, when the second placed object and the placed object overlap each other from the point of view of the user character (user's point of view), the second placed object is preferentially displayed. Set the display mode. For example, while setting the second placed object to the display state, the placed object is set to the non-display state. By doing so, for example, information having a high degree of importance can be displayed to the user with higher priority. Note that, based on the set priority, for example, the second placed object, which is another object, may be set to the non-display state, while the placed object may be set to the displayed state.

Further, in this embodiment, the display mode or layout mode of the placed object is set according to the line-of-sight direction of the user character. For example, the color, brightness, translucency, texture, or display content of the placed object are set according to the line-of-sight direction of the user character. Alternatively, the arrangement position, the arrangement direction, or the number of arranged objects are set according to the line-of-sight direction of the user character.

For example, in FIG. 25A, the line-of-sight direction VD of the user character USC faces the reference point RP. The line-of-sight direction VD of the user character USC corresponds to the line-of-sight direction of the user in the real space. For example, when the user in the real space changes the line-of-sight direction, the line-of-sight direction VD (the line-of-sight direction of the virtual camera) of the user character USC in the virtual space changes. Then, when the line-of-sight direction VD of the user character USC faces the reference point RP, the placed objects OB1 to OB3 are set in the display mode or layout mode as shown in FIG. 25(A). For example, the display contents (display mode) of the layout objects OB1 to OB3 are drawn as arrows pointing upward on the paper surface. That is, the display contents of the placed objects OB1 to OB3 are set to indicate the direction from the user character USC toward the reference point RP. Also, the placement positions (placement modes) of the placement objects OB1 to OB3 are set, for example, in an area between the user character USC and the reference point RP. For example, the layout positions of the layout objects OB1 to OB3 are set on a line connecting the user character USC and the reference point RP.

On the other hand, in FIG. 25B, the line-of-sight direction VD of the user character USC faces in a different direction from the reference point RP. When the line-of-sight direction VD of the user character USC faces in a direction different from that of the reference point RP, the placed objects OB1 to OB3 are set in the display mode or layout mode as shown in FIG. 25(B). For example, the display contents of the placed objects OB1 to OB3 are drawn as arrows pointing diagonally upward to the right on the page. That is, the display contents of the placed objects OB1 to OB3 are set to indicate the direction from each placed object toward the reference point RP. Also, the line-of-sight direction VD of the user character is oriented in a direction different from the reference point RP, and the layout positions of the placed objects OB1 to OB3 are set on a line along the line-of-sight direction VD.

By doing so, it is possible to set the placed objects OB1 to OB3 in an appropriate display mode or layout mode according to the line-of-sight direction VD of the user character USC. For example, when the line-of-sight direction VD of the user character USC is directed toward the reference point RP, the arranged objects OB1 to OB3 are set in the display mode or arrangement mode as shown in FIG. 25(A). By setting such a display mode or arrangement mode, it becomes possible to appropriately guide and guide the user in the direction of the reference point RP set on the target TG. Also, when the line-of-sight direction VD of the user character USC faces in a direction different from that of the reference point RP, the arranged objects OB1 to OB3 are set in the display mode or arrangement mode as shown in FIG. 25(B). By setting such a display mode or arrangement mode, it becomes possible to guide and guide a user who is moving in a direction different from the reference point RP to move in the direction of the reference point RP.

Further, in this embodiment, the display mode or layout mode of the placed object is set according to the position of the reference point. For example, depending on the position of the reference point, the color, brightness, translucency, texture, or display content of the arranged object are set. Alternatively, the layout position, layout direction, or number of layout objects are set according to the position of the reference point.

For example, in FIG. 26, reference points RP2 and RP3 are set in addition to the reference point RP. The reference point RP is set with respect to the target TG, which is the destination or object to which the user is to be guided. On the other hand, the reference points RP2 and RP3 are set at locations PLA and PLB different from the target TG. Placement objects OB1, OB2, and OB3 set in a display mode or a placement mode as shown in FIG. 26 are placed with respect to the reference point RP set on the target TG. On the other hand, placed objects OB21, OB22, and OB23 whose display modes or placement modes are different from those of placed objects OB1, OB2, and OB3 are placed with respect to reference point RP2 set at a location PLA different from target TG. Placed objects OB31, OB32, and OB33 having different display modes or placement modes from those of placed objects OB1, OB2, and OB3 are laid out with respect to a reference point RP3 set at a place PLB different from the target TG. That is, the display mode or layout mode of the placed object differs depending on the reference points RP, RP2, and RP3.

For example, with respect to the reference point RP set on the target TG, placement objects OB1, OB2, and OB3 drawn with arrows pointing upward on the paper are arranged on a line connecting the reference point RP and the user character USC.

On the other hand, with respect to the reference point RP2 set at a location PLA different from the target TG, the arranged objects OB21, OB22, and OB23 with arrows pointing diagonally upward to the right on the paper plane are aligned with the reference point RP2 and the user character USC. placed on the line connecting The arrows of the placement objects OB21, OB22, and OB23 point toward the reference point RP set on the target TG.

With respect to the reference point RP3 set at a location PLB different from the target TG, arranged objects OB31, OB32, and OB33 with arrows pointing diagonally upward to the left on the paper surface connect the reference point RP3 and the user character USC. placed on a line. The arrows of the placement objects OB31, OB32, and OB33 point toward the reference point RP set on the target TG.

By setting the display mode or layout mode of the placed object according to the position of the reference point in this way, it is possible to set the placed object in a display mode or layout mode suitable for each reference point. Therefore, it is possible for the user to set the layout of the layout objects that are easier to see and suitable for use as a guide display.

Further, in this embodiment, the reference point is set according to the user's situation information, the user character's situation information, or the game situation information. Furthermore, in this embodiment, environment information of the real space is acquired. Then, according to the environment information, the setting of the reference point or the setting of the display mode or layout mode of the placed object is performed. That is, the setting of the reference point is variably changed based on various information.

FIG. 27 shows a flowchart of the reference point setting process. First, environmental information such as brightness information and area information of the real space is acquired (step S1). The brightness information and the size information of the real space are, for example, the brightness information and the size information of the play field FL in FIG. 6 and the play field FLR in FIG. For example, the brightness information is brightness information due to lighting or the like in the play fields FL and FLR. For example, if the lighting is weak, it is determined to be dark, and if the lighting is strong, it is determined to be bright. The size information is information that serves as an index of the size of the play fields FL and FLR. For example, the recognition range of the real space recognition processing described with reference to FIGS. 4(A) and 4(B), the accuracy and reliability of recognition, etc. change according to the width and brightness of the play fields FL and FLR.

Next, the user's situation information such as the user's age group, physique, and game play situation is acquired (step S2). The age group of the user is, for example, information as to whether the user is a child or an adult. Alternatively, it may be information as to whether the user is young or old. The physique is information such as the height of the user, for example. Alternatively, information on the user's weight may be used. The game play status includes, for example, the user's game play proficiency level (beginner, intermediate, or advanced), the user's equipment status, the user's past game results, the user's number of times the user has played the game, the user's game play frequency, the user's statuses in the game, or situations related to the user's friends.

Next, the situation information of the user character such as the user character's status, level, game parameters, etc. is obtained (step S3). The user character's status is the user character's attack power, defense power, durability, equipment, or the like. The level of the user character represents the level of the experience value of the user character in the game. The game parameters of the user character are various parameters of the user character used during game processing.

Next, character status information and type information are obtained (step S4). The character situation information is information such as the character's status, level, and game parameters. The character type information is information about what type the character belongs to. For example, when an insect character other than a mosquito, an animal character, or a monster character appears as a character, the type information is information indicating the type of insect, animal, or monster.

Next, game situation information is obtained (step S5). The game situation information includes the progress of the game, the occurrence of events in the game, the degree of difficulty of the game, the clearing status of the game stage, the status of the map of the game, and the like.

Then, a reference point is set based on the acquired real space environment information, user situation information, user character situation information, character situation information or type information, or game situation information (step S6). Alternatively, the display mode or layout mode of the placed object is set according to the environment information.

For example, if the environment such as lighting in the real space is dark, the recognition range of the real space recognition processing in FIGS. Therefore, when it is determined that the environment is dark, the reference point is set at a position close to the user character, or the interval between the reference points is narrowed. For example, a sub reference point is set between the main reference point and the user character. Alternatively, instead of using real space information acquired in real time by real space recognition processing, real space information acquired by past recognition processing and accumulated in the real space information storage unit 171 is used to determine the reference point. set.

Also, the setting position of the reference point is changed according to the age group and physique of the user. For example, if the user is a child or a short user, the reference point is set at a low position. On the other hand, for an adult user or a tall user, the reference point is set at a higher position than for a child or a short user. Alternatively, depending on whether the user is a beginner or an advanced player, or the number and frequency of games played by the user, the setting positions, the number of settings, the setting interval, or the setting mode of the reference points are changed. For example, if the user is a beginner, the reference point is set so that the difficulty level becomes lower, and if the user is an advanced user, the reference point is set so that the difficulty level becomes higher. Also, for a user who has played the game many times or has a high playing frequency, as a benefit, a reference point is set so that the user's game play becomes more advantageous.

In addition, the set position, number of set points, set interval, or setting mode of the reference points are changed according to the user character's status such as offensive strength, defensive strength, equipment or endurance, level, or various game parameters. For example, depending on whether the ability of the user character is high or low, or whether the user is wearing predetermined equipment, the setting of the reference point is changed. Alternatively, the setting position, setting number, setting interval, or setting mode of the reference points are changed according to the situation information and type information of the character. For example, the setting of the reference point is changed according to the character's status such as offensive power, defensive power, equipment or endurance. Alternatively, depending on whether the character that appears is of the first type (for example, a first type of insect, animal, or monster) or of the second type, the set position of the reference point or the like is changed.

Also, depending on the game situation, the setting position, the number of setting, the setting interval, or the setting mode of the reference points are changed. For example, the setting of the reference point is changed according to the progress of the game or the occurrence of events. For example, in FIG. 14, when the collapse event of the ceiling object OBC occurs, it is not preferable to set the reference point on the back side of the fragment of the ceiling object OBC, so the reference point is set on the front side of the fragment. . Alternatively, the set position, the number of set points, or the set intervals of the reference points are changed according to the difficulty level of the game. For example, as the game becomes more difficult, the setting of the reference point is changed so that it becomes more difficult for the user to clear the game.

By doing so, it is possible to set an appropriate reference point according to the environment of the real space, the situation of the user, the situation of the user character, the situation or type of the character, or the game situation.

Note that in this embodiment, the display mode or layout mode of the placed object may be set according to the environment information. For example, the display mode of the placed objects is changed or the placement mode of the placed objects is changed according to the brightness information of the lighting in the play field in the real space, the size information, and the like. For example, if the real space is a very bright environment, and the visibility and discrimination power of the placed object decreases, the brighter the brightness in the real space, the higher the visibility and discrimination power of the placed object. display mode change processing. Alternatively, the placement position, placement direction, or number of placement objects is changed according to the size of the play field in the real space.

Further, for example, in real space, it is assumed that a wall or the like exists near the user, and it is detected based on environment information that the user may collide with the wall. In this case, when the user approaches the wall in the real space, a reference point is set at the position of the wall object in the virtual space generated corresponding to the wall. You may place a placement object that gives a warning.

Alternatively, a reference point is set for the object located at the point where the notification event occurs, and based on the set reference point, various messages are notified to the user, information about the game situation, etc. You can place objects. Alternatively, a reference point is set for an object located at the point where an instruction event to the user occurs, and based on the set reference point, the user is instructed to perform various actions, or to perform various actions related to game play. A placement object that gives instructions may be placed.

Further, in this embodiment, the display mode or layout mode of the placed object is set according to the user's situation information, the user character's situation information, or the game situation information. For example, depending on user situation information, user character situation information, or game situation information, the color, brightness, translucency, texture, or display content of a placed object may be set, or the position, direction, or arrangement of a placed object may be set. set the number.

FIG. 28 is a flowchart of processing for setting the display mode and layout mode of placed objects according to the user, user character, and game situation information. First, the user's situation information such as the user's age group, physique, and game play situation is acquired (step S11). Next, the situation information of the user character such as the user character's status, level, game parameters, etc. is obtained (step S12). Also, character status information and type information are acquired (step S13). Furthermore, game situation information is acquired (step S14). These user situation information, user character situation information, character situation information, type information, and game situation information are the information as described above.

Then, based on the user situation information, the user character situation information, or the game situation information, the display mode or layout mode of the placed object is set (step S15). For example, based on these pieces of information, the display mode or layout mode of the placed object is changed.

For example, the display mode or layout mode of the placed objects is changed according to the age group and physique of the user. For example, as shown in FIG. 29A, if the user US is a child or short, the height LH1 of the placement position of the placement object OB is reduced. On the other hand, as shown in FIG. 29B, for an adult or tall user US, the height LH2 of the placement position of the placement object OB is increased. In this way, the user US can place the placement object OB at a position that is easier for the user US to see. In addition, the display mode or layout mode of the placed objects is changed according to whether the user is a beginner or an advanced player, or the number and frequency of games played by the user. For example, if the user is a beginner, the placement objects are arranged in a display mode or arrangement mode that is easier to see than for an advanced user. Also, for a user who has played the game a large number of times or has a high frequency of play, the arranged object is arranged in a more legible display mode or arrangement mode as a benefit. Alternatively, conversely, the arranged objects may be arranged in a more difficult-to-see display mode or arrangement mode.

Also, the display mode or layout mode of the placed objects is changed according to the user character's status such as offensive power, defensive power, equipment or endurance, level, or various game parameters. For example, depending on whether the ability of the user character is high or low, or whether the user is wearing predetermined equipment, the display mode or layout mode of the placed object is changed.

Also, the display mode or layout mode of the arranged objects is changed according to the game situation. For example, depending on whether a predetermined event has occurred in the game, the display mode or layout mode of the placed objects is changed. Alternatively, the display mode or layout mode of the arranged objects is changed according to the difficulty level of the game. For example, as the difficulty level of the game increases, the display mode or layout mode of the placed objects is changed so that it becomes more difficult for the user to clear the game.

For example, in FIG. 30A, the reference point RP is set at the location of the treasure box TR, which is the target TG. Placement objects OB1, OB2, and OB3 are placed to guide the user toward the treasure box TR. When the treasure box TR has not yet been opened and there is content in it, the placed objects OB1 to OB3 are placed in a display mode or layout mode as shown in FIG. 30(A). Then, for example, in the first dungeon search, it is assumed that the user character USC reaches the position of the treasure box TR by the guidance of the placed objects OB1 to OB3, opens the treasure box TR, and obtains its contents. In this case, as shown in FIG. 30(B), when the dungeon is searched for the second time, the display mode of the placed objects OB1 to OB3 is set to a non-display state, or a display with low visibility or identification power is displayed. mode. In this way, in FIGS. 30A and 30B, the display modes of the arranged objects OB1 to OB3 are changed according to the game situation.

By doing so, it is possible to appropriately set the placement of the placed objects according to the user's situation, the user character's situation, or the game situation.

In this embodiment, a treasure box TR is placed in the real space play field, and recognition processing of the treasure box TR in the real space is performed using the camera of the HMD 200 or the like to generate a treasure box object corresponding to the treasure box TR, which is the target TG. . Then, a reference point RP is set for the treasure chest object, and placement objects OB1 to OB3 are arranged based on the set reference point RP. In this manner, a reference point RP is set for the target TG of the user's game play (behavior in a broad sense), and the placement objects OB1 to OB3 can be arranged based on the reference point RP set for the target TG. become. The position of the target TG and the position of the reference point RP may coincide, or the reference point RP may be set at a position relatively separated from the position of the target TG by a given distance.

4. Detailed Processing Next, a detailed processing example of this embodiment will be described with reference to the flowchart of FIG.

First, recognition processing of the real space around the user is performed to acquire real space information (step S21). That is, the real space is scanned as described with reference to FIGS. 4A and 4B, and the acquired real space information is stored in the real space information storage unit 171 shown in FIG. Also, user information including the user's location information is obtained (step S22). For example, user information including user's position information, user's direction information, posture information, etc. is acquired by the tracking processing of the HMD 200 described with reference to FIGS. 2 and 3 .

Next, a virtual space corresponding to the real space is generated based on the acquired real space information (step S23). For example, a virtual object corresponding to an object (a desk, a wall, etc.) in the real space as shown in FIG. 4B is generated and arranged in the virtual space. A reference point is set based on the user information and the real space information, and the arranged object is arranged in the virtual space based on the reference point (step S24). Then, the display mode of the placed object is set according to the positional relationship between the user character and the placed object (step S25). For example, as shown in FIGS. 21(A) to 22(B), display modes such as translucency and display contents of the placed objects are set according to the positional relationship between the user character USC and the placed objects OB1 to OB4. As a result, as shown in FIGS. 12(A) to 13(B), it is possible to display the placed objects OB1 to OB4 in an easy-to-see display mode that can appropriately guide the user to the target TG.

Although the present embodiment has been described in detail as above, those skilled in the art will easily understand that many modifications are possible without substantially departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included within the scope of this invention. For example, terms (user character, arm sensor, etc.) that are described at least once in the specification or drawings together with different terms (user moving body, detection unit, etc.) with broader meanings or synonyms may be used anywhere in the specification or drawings. can also be replaced by its different term. Real space information and user information acquisition processing, virtual space generation processing, reference point setting processing, placement object placement processing, placement object display mode or placement mode setting processing, character placement processing, hit volume setting The processing, positional relationship determination processing, and the like are not limited to those described in the present embodiment, and equivalent methods, processing, and configurations are also included in the scope of the present invention. Also, the present invention can be applied to various games. The present invention can also be applied to various simulation systems such as commercial game machines, home game machines, and large-scale attraction systems in which many users participate.

RP, RP2, RP3 reference point, CH, CH1 to CH3 character,
OB, OB1-OB4, OB21-OB23, OB31-OB33 arrangement object,
TG Target, US, US1, US2 User, USC User Character,
MRS background object, PS position, HV hit volume, DK desk, WL wall,
OBC1 to OBC5 Object, VD Line of sight, TR Treasure box,
ARV display area, HL, HR hand, BL, BR possession, SE arm sensor,
FL, FLR Playfield, GD, GDA, GD1~GD3 Guide,
PS poster, IMA image, PIL pillar, RD1, RD2 ride housing,
AG attack effect, HE1-HE3 hole, CS course, OBC object,
100 processing unit, 102 information acquisition unit, 104 virtual space generation unit,
106 object processing unit, 108 reference point setting unit, 109 hit processing unit,
107 moving object processing unit, 112 virtual camera control unit, 114 game processing unit,
120 display processing unit, 130 sound processing unit, 170 storage unit, 171 real space information storage unit,
172 object information storage unit, 178 drawing buffer,
180 information storage medium, 192 sound output section, 194 I/F section,
195 portable information storage medium, 196 communication unit,
200 HMD (head mounted display), 216 RGB camera,
217 depth camera, 218, 219 environment recognition camera,
240 temple section, 242 goggles section, 243 display device,
244 holographic optics, 246 RGB camera, 247 depth camera,
248, 249 environment recognition camera,

Claims (8)

An information acquisition unit that acquires three-dimensional real space information, which is three-dimensional map information of the real space obtained by performing recognition processing for scanning the real space around the user, and user information including the position information of the user. When,
a virtual space generation unit that generates a three-dimensional virtual space corresponding to the real space around the user based on the three-dimensional real space information;
an object processing unit that sets a reference point in the virtual space based on the user information and the real space information and arranges a placed object based on the reference point;
a display processing unit that performs processing for displaying an image including the image of the arranged object on a display unit;
including
The object processing unit
setting a display mode of the placed object according to a positional relationship between the user moving object in the virtual space corresponding to the user and the placed object ;
The virtual space generation unit
generating a virtual object in the virtual space based on the three-dimensional real space information;
The object processing unit
A simulation system , wherein the reference point is set at a location corresponding to the virtual object when it is determined that the position of the user is close to the virtual object based on the user information . In claim 1 ,
The object processing unit
placing a first placement object and a second placement object as the placement objects;
When the first placed object and the second placed object are positioned in the direction of the reference point side of the user's moving object, the first placed object is set to a display state, and the second placed object is placed. set the object to invisible,
When the first placed object is positioned in the direction opposite to the direction of the reference point side of the user's moving body, and the second placed object is positioned in the direction of the reference point side of the user's moving body and setting the first placed object in a non-display state and setting the second placed object in a displayed state. In claim 1 or 2 ,
A display priority is set for objects in the virtual space,
The object processing unit
A simulation characterized in that, when the arranged object and another object in the virtual space are arranged so as to overlap each other as seen from the user's moving body, a display mode of the arranged object is set based on the order of priority. system. In any one of claims 1 to 3 ,
The object processing unit
A simulation system, wherein a display mode or an arrangement mode of the placed object is set according to a line-of-sight direction of the user's moving object. In any one of claims 1 to 4 ,
The information acquisition unit
Acquiring environmental information of the real space;
The object processing unit
A simulation system, wherein the setting of the reference point or the setting of the display mode or layout mode of the placed object is performed according to the environment information. In any one of claims 1 to 5 ,
The simulation system, wherein the placement object is a placement object that guides movement of the user, or a placement object that warns, notifies, or instructs the user. In any one of claims 1 to 6 ,
The object processing unit
A simulation system, wherein the reference point is set for the target of the user's behavior, and the placement object is arranged based on the reference point set for the target. An information acquisition unit that acquires three-dimensional real space information, which is three-dimensional map information of the real space obtained by performing recognition processing for scanning the real space around the user, and user information including the position information of the user. When,
a virtual space generation unit that generates a three-dimensional virtual space corresponding to the real space around the user based on the three-dimensional real space information;
an object processing unit that sets a reference point in the virtual space based on the user information and the real space information and arranges a placed object based on the reference point;
A display processing unit that performs processing for displaying an image including the image of the arranged object on the display unit,
make your computer work
The object processing unit
setting a display mode of the placed object according to a positional relationship between the user moving object in the virtual space corresponding to the user and the placed object;
The virtual space generation unit
generating a virtual object in the virtual space based on the three-dimensional real space information;
The object processing unit
A program for setting the reference point at a location corresponding to the virtual object when the user's position is determined to be close to the virtual object based on the user information .

Images