JP7332823B1 - program - Google Patents

Abstract

A program capable of motivating a user who is having a simulated experience in a virtual space to actually go to a real space is provided. A program causes a computer to arrange virtual space definition means for defining a virtual space corresponding to the real space, and the number of objects based on the detection result of a detection device installed in the real space in the virtual space. arranging means; camera moving means for moving the virtual camera within the virtual space based on a movement operation performed on the first user terminal; It functions as a generation means and an image display means for displaying a virtual space image on the first user terminal. [Selection drawing] Fig. 10

Description

The present invention relates to programs.

Conventionally, content corresponding to a predetermined place in the real world (for example, town, commercial facility, store, event venue, theme park, park, etc.) is not in the place (hereinafter referred to as "real space"). There is known a so-called VR technology that is provided to users (for example, Patent Literature 1).

More specifically, an image cut out from a virtual space corresponding to the real space is displayed on the user terminal, and the viewpoint of the displayed image is moved based on a movement operation on the user terminal. Thereby, the user operating the user terminal can have an experience as if he or she were in the real space without actually going to the real space.

JP 2022-122810 A

However, there is a problem that it is difficult to motivate users who have simulated experiences in the virtual space to actually go to the real space. As the VR technology develops and the feeling of immersion in the virtual space increases, the above-mentioned problems become more conspicuous.

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and its object is to provide a program capable of motivating a user who is having a simulated experience in a virtual space to actually go to the real space. That's what it is.

In order to solve the above problems, a program according to the present invention comprises a computer, a virtual space defining means for defining a virtual space corresponding to the real space, and a number of objects based on the detection result of a detection device installed in the real space. in the virtual space ; image generating means for generating a virtual space image obtained by imaging the virtual space with a virtual camera; and image display means for displaying the virtual space image on the first user terminal. and the image generation means generates a message registered in advance in association with the attribute detected by the detection device based on the selection operation of the object performed on the first user terminal, Output through the selected object .

According to the present invention, it is possible to motivate a user who is having a simulated experience in the virtual space to actually go to the real space.

It is a figure which shows the outline|summary of the system which concerns on this embodiment. 3 is a hardware configuration diagram of a server; FIG. 2 is a hardware configuration diagram of an HMD set as an example of a user terminal; FIG. FIG. 4 is a hardware configuration diagram of a tablet terminal that is another example of a user terminal; 1 is a diagram conceptually representing one aspect of a virtual space; FIG. It is a figure showing the YZ cross section which looked at the field-of-view area from the X direction in virtual space. It is a figure showing the XZ cross section which looked at the field-of-view area from the Y direction in virtual space. 3 is a functional block diagram of a server and user terminals; FIG. 6 is a flow chart showing processing of the server according to the first embodiment; 8 is a flow chart showing processing of the first user terminal according to the first embodiment; 4 is a flowchart of virtual space image generation and display processing; It is a conceptual diagram of the virtual space corresponding to steps S21 and S22. It is a figure which shows the conceptual diagram and virtual space image of the virtual space corresponding to step S47. It is a figure which shows the conceptual diagram and virtual space image of the virtual space corresponding to step S48. It is a figure which shows the conceptual diagram and virtual space image of the virtual space corresponding to step S45. FIG. 10 is a diagram showing a conceptual diagram of a virtual space and a virtual space image corresponding to the case of No in step S43; It is an example of a message table that holds attributes of people and messages in association with each other. 10 is a flow chart showing processing of the second user terminal according to the second embodiment; It is a figure which shows a status setting screen. 10 is a flow chart showing processing of the first user terminal according to the second embodiment; FIG. 10 is a diagram showing a virtual space image when another person's avatar in an operable state is selected; FIG. 10 is a diagram showing a virtual space image when a non-operable third party avatar and a mob character are selected;

A system 1 according to an embodiment will be described below with reference to the drawings. It should be noted that the embodiments of the present invention described below are merely examples of embodying the present invention, and the scope of the present invention is not limited to the scope of the description of the embodiments. Therefore, the present invention can be implemented by adding various changes to the embodiments.

[Overview of System 1]
FIG. 1 is a diagram showing an outline of a system 1 according to this embodiment. As shown in FIG. 1, the system 1 mainly includes a server 10 and user terminals 20A, 20B, and 20C (hereinafter collectively referred to as "user terminals 20"). Although three user terminals 20 are illustrated in FIG. 1, examples of the user terminals 20 included in the system 1 are not limited to this. The server 10 and the user terminal 20 are connected via the communication network 2 so as to be able to communicate with each other. A specific example of the communication network 2 is not particularly limited, but is configured by, for example, the Internet, a mobile communication system (eg, 4G, 5G, etc.), a wireless network such as Wi-Fi (registered trademark), or a combination thereof.

A system 1 according to this embodiment is a system for allowing users of a plurality of user terminals 20A to 20C to experience a common virtual space provided by a server 10. FIG. More specifically, the user terminal 20 arranges a user character associated with the user (hereinafter sometimes referred to as an “avatar”) in the virtual space, and an image of the virtual space viewed from a preset viewpoint. is displayed on the user terminal 20 .

In addition, the user terminal 20 operates the avatar in the virtual space in conjunction with the user's operation on the operation device. The actions of the avatar include, for example, moving within the virtual space, moving parts of the body, changing posture, changing facial expressions, speaking, and moving objects placed within the virtual space. including things.

Furthermore, the user terminal 20 transmits avatar data indicating changes in the avatar's actions and status to the server 10 via the communication network 2 . The server 10 transmits the avatar data received from the user terminal 20 to other user terminals 20 through the communication network 2 . Based on the avatar data received from the server 10, the user terminal 20 updates the motion and status of the corresponding avatar in the virtual space.

Further, the system 1 according to this embodiment provides a virtual space corresponding to the real space. In other words, the virtual space according to this embodiment is a space imitating the real space. However, the virtual space provided by the system 1 does not have to match the real space completely, and may be deformed as appropriate. The real space refers to, for example, a town, commercial facilities, stores, theme parks, event venues, parks, stations, etc. that exist in reality.

Furthermore, the system 1 (server 10 ) according to this embodiment is configured to be able to communicate with the detection device 3 installed in the physical space through the communication network 2 . The detection device 3 is a device that detects a person in the physical space. Then, the detection device 3 transmits detection data indicating the detection result to the server 10 through the communication network 2 . In addition, the type of the detection device 3 connected to the system 1 is not limited to one type, and multiple types may be combined.

As an example, the detection device 3 may be a camera (fixed-point camera) that captures an image of the physical space from above. In this case, the detection device 3 transmits a bird's-eye view image of the physical space to the server 10 as detection data. As another example, the detection device 3 may be a camera (for example, a device that measures the surface temperature) that captures the face of a person entering a commercial facility, store, theme park, or event venue. The detection device 3 in this case transmits an image of a person's face to the server 10 as detection data. As still another example, the detection device 3 may be a ticket gate for entering a theme park, an event venue, or a train station. In this case, the detection device 3 uses the purchase information of the ticket inserted into the ticket gate (for example, whether it is an adult or a child, the type of ticket, the purchase price, etc.) and the time when the ticket was inserted into the ticket gate as detection data. Send to However, specific examples of detection data are not limited to these.

Then, the server 10 according to this embodiment analyzes the detection data received from the detection device 3, and identifies the number, positions, and attributes of people present in the physical space. The attributes of a person refer to, for example, sex, age (adult or child), posture (eg, standing, walking, running), and the like. Then, the server 10 arranges a non-user character (hereinafter referred to as a "mob character") corresponding to the specified person in the virtual space. More specifically, the server 10 transmits mob-chara data indicating the shape and arrangement of the mob-chara to the user terminal 20 via the communication network 2 . Then, based on the mob character data received from the server 10, the user terminal 20 arranges mob characters within the virtual section. As a result, the mob character is displayed on the user terminal 20 .

As described above, avatars and mob characters are placed in the virtual space according to this embodiment. Avatars and mob characters are common in that they are shaped like humans. The avatar's appearance (eg, face shape, hairstyle, clothing, etc.) is determined by the user of the user terminal 20 . Also, the avatar operates in the virtual space in conjunction with the user's operation through the user terminal 20 . On the other hand, the appearance of the mob character is determined by the server 10 . As an example, the server 10 may select a mob character that matches the attributes detected by the detection device 3 from among pre-registered mob character candidates. As another example, the server 10 may select a mob character that matches the attributes detected by the detection device 3 from a log of avatars who have been in the virtual space in the past. Also, the mob character may operate in a manner determined by the server 10, or may be stopped.

[Configuration of server 10]
FIG. 2 is a hardware configuration diagram of the server 10. As shown in FIG. The server 10 is implemented by, for example, a workstation or a general-purpose computer such as a personal computer. As shown in FIG. 2 , the server 10 mainly includes a processor 11 , memory 12 , storage 13 , input/output interface 14 and communication interface 15 . Each component of server 10 is connected to communication bus 19 .

The processor 11 realizes the processing described later by executing a series of instructions included in the server program 13P stored in the memory 12 or the storage 13. FIG. The processor 11 is implemented as, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processor Unit), an FPGA (Field-Programmable Gate Array), and other devices.

The memory 12 temporarily holds the server program 13P and data. The server program 13P is loaded from the storage 13, for example. The data includes data input to server 10 and data generated by processor 11 . For example, the memory 12 is implemented as a RAM (Random Access Memory) or other volatile memory.

The storage 13 permanently holds the server program 13P and data. The storage 13 is implemented as, for example, a ROM (Read-Only Memory), hard disk device, flash memory, or other non-volatile memory device. Also, the storage 13 may be implemented as a removable storage device such as a memory card. As still another example, the storage 13 may be connected to the server 10 as an external storage device instead of being built in the server 10 . According to such a configuration, for example, in a scene where a plurality of user terminals 20 are used like an amusement facility, it is possible to collectively update the server program 13P and data.

The input/output interface 14 is an interface for connecting external devices such as a monitor, an input device (for example, a keyboard, a pointing device), an external storage device, a speaker, a camera, a microphone, and a sensor to the server 10 . Processor 11 communicates with external devices through input/output interface 14 . The input/output interface 14 is implemented using, for example, a USB (Universal Serial Bus), DVI (Digital Visual Interface), HDMI (High-Definition Multimedia Interface, registered trademark), and other terminals.

The communication interface 15 communicates with other devices connected to the communication network 2 (eg, the detection device 3, the user terminal 20). The communication interface 15 is implemented as, for example, a wired communication interface such as LAN (Local Area Network), or a wireless communication interface such as Wi-Fi (Wireless Fidelity), Bluetooth (registered trademark), NFC (Near Field Communication).

[Configuration of user terminal 20]
FIG. 3 is a hardware configuration diagram of an HMD (Head Mounted Display) set, which is an example of the user terminal 20. As shown in FIG. FIG. 4 is a hardware configuration diagram of a tablet terminal that is another example of the user terminal 20. As shown in FIG. The user terminal 20 is implemented as, for example, an HMD set shown in FIG. 3, a tablet terminal shown in FIG. 4, a smart phone, a feature phone, a laptop computer, a desktop computer, or the like.

As shown in FIG. 3 , the user terminal 20 implemented as an HMD set includes a computer 26 having a processor 21 , memory 22 , storage 23 , input/output interface 24 and communication interface 25 . Each component of computer 26 is connected to communication bus 29 . The basic configuration of the processor 21, memory 22, storage 23, input/output interface 24, communication interface 25, and communication bus 29 is the same as the processor 11, memory 12, storage 13, input/output interface 14, communication interface 15, Common with the communication bus 19 . The storage 23 also holds a terminal program 23P.

Also, the user terminal 20 implemented as an HMD set includes an HMD 30 , a motion sensor 41 , and an operation device 42 as external devices of the computer 26 . The HMD 30 , motion sensor 41 and operation device 42 are connected to the processor 21 through the input/output interface 24 .

The HMD 30 is worn on the user's head and provides the user with a virtual space. More specifically, the HMD 30 can include both a so-called head-mounted display having a monitor and a head-mounted device on which a smartphone or other terminal having a monitor can be attached. The HMD 30 mainly includes a monitor 31 (display device), a gaze sensor 32 , cameras 33 and 34 , a microphone 35 and a speaker 36 .

As an example, the monitor 31 is implemented as a non-transmissive display device. The monitor 31 is arranged, for example, on the body of the HMD 30 so as to be positioned in front of both eyes of the user. The non-transmissive monitor 31 is implemented as, for example, a liquid crystal monitor or an organic EL (Electro Luminescence) monitor.

As another example, the monitor 31 is implemented as a transmissive display device. The HMD 30 in this case is not a closed type that covers the user's eyes, but an open type such as a glasses type. The monitor 31 may include a configuration for simultaneously displaying a portion of the image forming the virtual space and the real space. As an example, the transmissive monitor 31 may display an image of the real space captured by the camera mounted on the HMD 30 . As another example, the transmissive monitor 31 may be configured so that the transmissivity can be adjusted. The transmissive monitor 31 may have a high transmittance in a portion of the display area so that the real space can be directly viewed.

Further, the monitor 31 may employ the following configuration in order to allow the user to visually recognize the three-dimensional image. As an example, the monitor 31 may include a sub-monitor for displaying images for the right eye and a sub-monitor for displaying images for the left eye. As another example, the monitor 31 may be configured to integrally display the image for the right eye and the image for the left eye. The monitor 31 in this case includes a high speed shutter. The high speed shutter operates to alternately display the right eye image and the left eye image so that the image is perceived by only one eye.

Gaze sensor 32 detects the direction in which the user's right and left eyes are directed. That is, the gaze sensor 32 detects the line of sight of the user. The gaze sensor 32 is realized by, for example, a sensor having an eye tracking function. The gaze sensor 32 preferably includes a right eye sensor and a left eye sensor. The gaze sensor 32 irradiates the user's right eye and left eye with infrared light, and detects the rotation angle of each eyeball by receiving reflected light from the cornea and iris of the irradiated light. Then, the gaze sensor 32 identifies the line of sight of the user based on each detected rotation angle.

The camera 33 captures an image of the upper part of the user's face (more specifically, the user's eyes, eyebrows, etc.) wearing the HMD 30 . The camera 34 images the lower part of the user's face (more specifically, the user's nose, mouth, etc.) wearing the HMD 30 . For example, of the housing of the HMD 30, the camera 33 is attached to the side facing the user, and the camera 34 is attached to the side opposite to the side facing the user. Note that the HMD 30 may include a single camera that captures the entire face of the user instead of the two cameras 33 and 34 .

The microphone 35 converts the user's speech into an audio signal (electrical signal) and outputs it to the computer 26 . The speaker 36 converts the audio signal output from the computer 26 into sound and outputs it to the user. Note that the HMD 30 may include earphones instead of the speaker 36 .

The motion sensor 41 has a position tracking function for detecting motion of the HMD 30 . As an example, the motion sensor 41 may read a plurality of infrared rays emitted by the HMD 30 to detect the position and tilt of the HMD 30 within the physical space. As another example, motion sensor 41 may be implemented with a camera. The motion sensor 41 in this case analyzes the image information of the HMD 30 output from the camera and detects the position and tilt of the HMD 30 . As still another example, the motion sensor 41 may be realized with an angular velocity sensor, a geomagnetic sensor, or an acceleration sensor.

The operating device 42 is connected to the computer 26 by wire or wirelessly. The operating device 42 accepts command inputs (operations) to the computer 26 by the user. As an example, the operating device 42 may be a so-called controller that is held and operated by the user. As another example, the operating device 42 may be configured to be attachable to a user's body or a part of clothing, and may detect the user's movement using a motion sensor. However, specific examples of the operation device 42 are not limited to these, and may be a keyboard, pointing device, touch panel, or the like.

As shown in FIG. 4, a user terminal 20 implemented as a tablet terminal includes a processor 21, a memory 22, a storage 23, a communication interface 25, a monitor 31, cameras 33 and 34, a microphone 35, and a speaker. 36 , a motion sensor 41 , and an operating device 42 . Each component of the tablet terminal is connected to the communication bus 29 . Since the basic configuration of the processor 21, memory 22, storage 23, communication interface 25, monitor 31, cameras 33 and 34, microphone 35, speaker 36, motion sensor 41, and operation device 42 is common to the case of the HMD set, The configuration specific to the tablet terminal will be described below.

The monitor 31 is provided on the surface of a flat plate-shaped housing. The camera 33 is a so-called in-camera that is attached to the surface of the flat-plate housing and captures the face of the user viewing the monitor 31 . The camera 34 is a so-called out-camera that is attached to the rear surface of the flat housing (the surface opposite to the monitor 31) and captures an image of the surroundings. The motion sensor 41 detects the motion of the housing (for example, rotation about three mutually orthogonal axes). The operation device 42 suitable for the tablet terminal is, for example, a touch panel that is superimposed on the monitor 31 and receives various touch operations (eg, tap, slide, flick, pinch-in, pinch-out, etc.) by the user.

[Overview of virtual space 90]
FIG. 5 is a diagram conceptually showing one aspect of the virtual space 90. As shown in FIG. FIG. 6 is a diagram showing a YZ cross section of the visual field area 94 viewed from the X direction in the virtual space 90. As shown in FIG. FIG. 7 is a diagram showing an XZ cross section of the visual field area 94 viewed from the Y direction in the virtual space 90. As shown in FIG.

As shown in FIG. 5, the virtual space 90 has a spherical structure covering the entire 360-degree direction of the center C. As shown in FIG. In FIG. 5, the celestial sphere in the upper half of the virtual space 90 is illustrated so as not to complicate the explanation. Each mesh is defined in the virtual space 90 . The position of each mesh is defined in advance as coordinate values in the XYZ coordinate system, which is the global coordinate system defined in the virtual space 90 . Each partial image forming a panorama image 91 (still image, moving image, etc.) that can be developed in the virtual space 90 is associated with each corresponding mesh in the virtual space 90 .

For example, in the virtual space 90, an XYZ coordinate system with the center C as the origin is defined. The XYZ coordinate system is parallel to the real coordinate system, for example. The horizontal direction, vertical direction (vertical direction), and front-rear direction in the XYZ coordinate system are defined as the X-axis, Y-axis, and Z-axis, respectively. Therefore, the X-axis (horizontal direction) of the XYZ coordinate system is parallel to the x-axis of the real coordinate system, the Y-axis (vertical direction) of the XYZ coordinate system is parallel to the y-axis of the real coordinate system, and the The Z-axis (front-rear direction) is parallel to the z-axis of the real coordinate system.

A virtual camera 92 associated with the user terminal 20 is arranged in the virtual space 90 . The position of the virtual camera 92 within the virtual space 90 corresponds to the user's viewpoint within the virtual space 90 . Also, the orientation of the virtual camera 92 corresponds to the user's line of sight (reference line of sight 93 ) in the virtual space 90 . The processor 21 then defines a field of view area 94 (angle of view of the virtual camera 92 ) in the virtual space 90 based on the position and orientation of the virtual camera 92 .

As shown in FIG. 6, viewing area 94 includes area 95 in the YZ cross section. A region 95 is a range of a polar angle α around the reference line of sight 93 in a vertical section (YZ section) including the reference line of sight 93 in the virtual space 90 . As shown in FIG. 7, viewing area 94 includes area 96 in the XZ cross section. A region 96 is a range of an azimuth angle β centered on the reference line of sight 93 in a horizontal section (XZ section) including the reference line of sight 93 in the virtual space 90 .

The processor 21 generates (extracts) a partial image included in the field of view area 94 of the panorama image 91 developed in the virtual space 90 as a virtual space image 97 captured by the virtual camera 92 . The processor 21 then causes the monitor 31 to display the generated virtual space image 97 . That is, the field of view area 94 corresponds to the user's field of view within the virtual space 90 . Further, the field of view area 94 moves in accordance with changes in the position and orientation of the virtual camera 92 within the virtual space 90, and the virtual space image 97 displayed on the monitor 31 is updated. That is, the user's field of view moves.

For example, the processor 21 moves the virtual camera 92 within the virtual space 90 in conjunction with the user's operation received by the operation device 42 . In addition, the processor 21 changes the direction of the virtual camera 92 (that is, the reference line of sight 93) in conjunction with the motion of the user terminal 20 detected by the motion sensor 41 (for example, rotation about three mutually orthogonal axes). . Further, the processor 21 causes the monitor 31 to display a virtual space image 97 captured by the virtual camera 92 after the position and orientation have changed.

[Functional block diagram of server 10 and user terminal 20]
FIG. 8 is a functional block diagram of the server 10 and the user terminal 20. As shown in FIG. As shown in FIG. 8, the server program 13P loaded into the memory 12 causes the server 10 to function as avatar synchronization means 110, person extraction means 120, and mob character generation means . The terminal program 23P loaded into the memory 22 causes the user terminal 20 (computer 26) to function as virtual space definition means 210, placement means 220, camera movement means 230, image generation means 240, and image display means 250.

The avatar synchronization means 110 transmits (relays) avatar data received from one of the user terminals 20 through the communication network 2 to the other user terminals 20 . Thereby, all user terminals 20A to 20C hold the same avatar data. As a result, the same avatar is placed at the same position in the virtual space 90 defined by each of the user terminals 20A-20C. That is, avatars are synchronized in all user terminals 20A-20C.

The person extraction means 120 receives detection data from the detection device 3 through the communication network 2 . Next, the person extraction means 120 extracts persons in the physical space from the detection data received from the detection device 3 . Also, the person extracting means 120 may extract attributes of persons in the physical space from the detection data received from the detection device 3 . As an example, when the detection device 3 is a camera, the person extraction means 120 uses a well-known person recognition technology to extract the number and positions of people in the physical space, their sex, age, and posture from image data or video data. Extract etc. As another example, when the detection device 3 is a ticket gate, the person extraction means 120 extracts the number and age of people in the physical space from the ticket inserted in the ticket gate. However, the specific method by which the person extracting means 120 extracts the person and attributes is not limited to the above example, and other well-known methods can be adopted.

The mob character generating means 130 generates mob character data representing mob characters corresponding to the people extracted by the person extracting means 120 . The mob character data indicates the appearance of the mob character (eg, body shape, facial expression, hairstyle, clothes, etc.), the position of the mob character in the virtual space 90, the posture of the mob character, and the like. However, the position of the person in the real space and the position of the mob character in the virtual space 90 do not have to match exactly. Then, the mob character generating means 130 transmits the generated mob character data to all the user terminals 20A to 20C. More specifically, the mob character generating means 130 generates as many mob characters as the number of people extracted by the person extracting means 120 . Also, the mob character generating means 130 generates a mob character having a mode corresponding to the attribute of the person extracted by the person extracting means 120 . For example, if the extracted gender is female, a female mob character is generated, if the extracted age is a child, a child mob character is generated, and if the extracted posture is sitting, a sitting mob character is generated. .

The virtual space defining means 210 defines a virtual space 90 corresponding to the real space. More specifically, virtual space definition means 210 develops virtual space data representing virtual space 90 in memory 22 . The virtual space data indicates, for example, the panoramic image 91 and the shapes and positions of objects (eg, buildings, plants) arranged in the virtual space 90 . The virtual space data may be downloaded from the server 10 and stored in the storage 23 in advance, or may be downloaded from the server 10 when the virtual space 90 is defined. Also, the virtual space defining means 210 may define a virtual space 90 selected from among the plurality of virtual spaces 90 through the operation device 42 . Since the specific processing for defining the virtual space 90 is already well known, detailed description thereof will be omitted.

The placement means 220 places the character in the virtual space 90 defined by the virtual space definition means 210 . Here, a case where the terminal program 23P is executed on the user terminal 20A will be described. The characters arranged in the virtual space 90 by the arrangement means 220 are the avatar of the user of the user terminal 20A (first user terminal) (hereinafter referred to as "personal avatar") and the other user terminals 20B and 20C (second user terminals). user terminal) user's avatar (hereinafter referred to as "other person's avatar") and mob characters.

First, the arrangement means 220 arranges the principal avatar in the virtual space 90 based on the avatar data pre-stored in the storage 23 . The arranging means 220 then transmits the avatar data of the principal avatar to the server 10 via the communication network 2 . Further, when receiving a user's operation to instruct the movement of the principal avatar through the operating device 42, the arranging unit 220 causes the principal avatar to move in the virtual space 90 according to the operation. The arranging means 220 then updates the avatar data so as to indicate the state after the action of the principal avatar, and transmits the updated avatar data to the server 10 via the communication network 2 .

Also, the arranging means 220 receives avatar data of other avatars from the server 10 through the communication network 2 . Then, placement means 220 places another person's avatar in virtual space 90 based on the avatar data received from server 10 . Furthermore, the arranging means 220 rearranges the avatar of another person each time the avatar data is updated, so that the avatar of another person operates within the virtual space 90 .

Furthermore, the arrangement means 220 receives the mob character data generated by the mob character generation means 130 from the server 10 via the communication network 2 . Then, the placing means 220 places the mob character in the virtual space 90 based on the mob character data received from the server 10 . Furthermore, the placement means 220 moves the mob character in the virtual space 90 by rearranging the mob character each time the mob character data is updated.

The camera moving means 230 moves the virtual camera 92 within the virtual space 90 based on the motion of the user terminal 20 detected by the motion sensor 41 or the motion instruction of the avatar through the operating device 42 . Moving the user terminal 20 and instructing the action of the avatar through the operation device 42 are examples of movement operations. If the relative position between the virtual camera 92 and the person's avatar does not change, the camera moving means 230 moves the virtual camera 92 following the person's avatar moved by the placement means 220 . In addition, the camera movement means 230 may move the virtual camera 92 independently of the person's avatar according to instructions from the motion sensor 41 or the operation device 42. FIG.

The image generating means 240 generates a virtual space image 97 in which the inside of the virtual space 90 is captured by the virtual camera 92 . More specifically, the image generating means 240 extracts an image corresponding to the field of view area 94 from the panorama image 91 as a virtual space image 97 and stores the virtual space image data representing the virtual space image 97 in the memory 22. . Also, the image generating means 240 controls how the character is included in the virtual space image 97 .

As an example, when the number of mob characters included in the field of view area 94 (that is, the angle of view of the virtual camera 92) is equal to or greater than a threshold value, the image generation means 240 reduces the number of mob characters (thus thinning out) to create a virtual space. An image 97 may be generated. As another example, the image generating means 240 may include the avatars and mob characters included in the field of view area 94 in the virtual space image in different display modes. As another example, the image generating means 240 may output a pre-registered message associated with a mob character based on a mob character selection operation performed on the operation device 42 . As another example, the image generation means 240 may switch whether to include mob characters in the virtual space image 97 based on the operation performed on the operation device 42 . As still another example, the image generator 240 may switch the display mode of the avatar based on the operation performed on the operation device 42 .

The image display means 250 displays the virtual space image 97 on the monitor 31 . More specifically, the image display means 250 develops the virtual space image data generated by the image generation means 240 in the graphic memory of the monitor 31 . Each time the character in the virtual space 90 is updated, the image generating means 240 generates a new virtual space image 97, and the image display means 250 displays the newly generated virtual space image 97 on the monitor 31. . As a result, the user of the user terminal 20 can view, through the monitor 31 , an image showing the field of view of the character operating in the virtual space 90 .

[Example 1]
The operation of the system 1 according to the first embodiment will be described with reference to FIGS. 9 to 16. FIG. In the first embodiment, it is assumed that the avatar of the user of the user terminal 20A is in the virtual space 90 corresponding to the real city. The system 1 according to the first embodiment provides the user of the user terminal 20</b>A with an experience in the virtual space 90 and notifies the corresponding situation of the real space through the virtual space 90 .

FIG. 9 is a flowchart illustrating processing of the server 10 according to the first embodiment. FIG. 10 is a flowchart illustrating processing of the user terminal 20A according to the first embodiment. FIG. 11 is a flowchart of virtual space image generation and display processing. FIG. 12 is a conceptual diagram of the virtual space 90 corresponding to steps S21 and S22. FIG. 13 shows a conceptual diagram of the virtual space 90 and a virtual space image 97 corresponding to step S47. FIG. 14 shows a conceptual diagram of the virtual space 90 and a virtual space image 97 corresponding to step S48. FIG. 15 shows a conceptual diagram of the virtual space 90 and a virtual space image 97 corresponding to step S45. FIG. 16 shows a conceptual diagram of the virtual space 90 and a virtual space image 97 corresponding to No in step S43. The processing of FIG. 9 by the server 10 and the processing of FIG. 10 by the user terminal 20A are executed in parallel.

The server 10 repeatedly executes the process shown in FIG. 9 at predetermined time intervals. First, the server 10 waits until it receives avatar data from the user terminals 20A to 20C (S11) or until it receives detection data from the detection device 3 (S12) before executing the processes from step S13 onward. Avatar data is irregularly transmitted from each of the user terminals 20A to 20C. Further, detection data is transmitted from the detection device 3 at predetermined time intervals. That is, the processing of step S13 and the processing of steps S14 to S16 are executed at arbitrary timing and frequency.

When the server 10 (avatar synchronization means 110) receives avatar data from one of the user terminals 20A to 20C through the communication network 2 (S11: Yes), The received avatar data is transmitted (relayed) (S13). Thereby, all user terminals 20A to 20C hold the same avatar data. In other words, the same avatar is placed in the virtual space 90 of all user terminals 20A-20C.

Further, when the server 10 (person extracting means 120) receives the detection data from the detection device 3 through the communication network 2 (S12: Yes), it extracts the person and attributes from the received detection data (S14). Next, the server 10 (mob character generating means 130) generates a mob character corresponding to the person and attribute extracted in step S14 (S15). Further, the server 10 (mob character generating means 130) transmits the mob character data indicating the mob character generated in step S15 to all the user terminals 20A to 20C through the communication network 2 (S16).

As shown in FIG. 10, the user terminal 20A (virtual space defining means 210) defines a virtual space 90 by loading the virtual space data stored in the storage 23 into the memory 12 (S21). Next, the user terminal 20A (placement means 220 and camera movement means 230) places the person avatar 50 and the virtual camera 92 in the virtual space 90 defined in step S21, as shown in FIG. The avatar data is transmitted to the server 10 through the communication network 2 (S22). The initial position of the person avatar 50 is, for example, the center C of the virtual space 90 . Also, the position of the virtual camera 92 may be, for example, the head position of the person avatar 50 or behind the person avatar 50 .

Next, when the user terminal 20A (arrangement means 220) receives avatar data from the server 10 through the communication network 2 (S23: Yes), for example, as shown in FIG. Other avatars 51B and 51C are arranged in the space 90 (S24). In addition, the other person's avatar 51B corresponds to the avatar data transmitted from the user terminal 20B, and the other person's avatar 51C corresponds to the avatar data transmitted from the user terminal 20C. Next, the user terminal 20A (image generation means 240 and image display means 250) generates a virtual space image shown in FIG. Display processing is executed (S29).

As shown in FIG. 11, the user terminal 20A (image generating means 240) identifies the field of view area 94 (S41). Next, the user terminal 20A (image generating means 240) determines whether or not a mob character is included in the angle of view (viewing area 94) of the virtual camera 92 (S42). Since no mob character is included in the state of FIG. 12(B) (S42: No), the user terminal 20A (image generating means 240) generates a virtual space image 97 that includes only the avatar of the avatar and the mob character (S50 ). Next, the user terminal 20A (image display means 250) displays the generated virtual space image 97 on the monitor 31 (S51).

In the state of FIG. 12(B), only the other avatar 51B out of the other avatars 51B and 51C is included in the angle of view of the virtual camera 92, so a virtual space image 97 as shown in FIG. 16(B) is displayed on the monitor 31. be done. Also, a toggle switch 98 may be displayed on the monitor 31 that displays the virtual space image 97 . The toggle switch 98 is a screen object that receives a user's operation to instruct whether or not to include mob characters in the virtual space image 97 . The operation of switching the toggle switch 98 to the ON side is an example of a display operation for instructing generation of the virtual space image 97 including the mob character included in the angle of view of the virtual camera 92 . The operation of switching the toggle switch 98 to the OFF side is an example of a non-display operation for instructing generation of the virtual space image 97 that does not include the mob character included in the angle of view of the virtual camera 92 .

Further, when the user terminal 20A (arrangement means 220) receives mob character data from the server 10 through the communication network 2 (S25: Yes), for example, as shown in FIG. Mob characters 52A, 52B, 52C and 52D are arranged in the space 90 (S26). Next, the user terminal 20A (image generation means 240 and image display means 250) executes virtual space image generation and display processing in order to reflect the mob characters 52A to 52D arranged in the virtual space 90 in the virtual space image 97. (S29).

When the user terminal 20A (image generating means 240) determines that the angle of view of the virtual camera 92 includes the mob characters 52A and 52B (S42: Yes), and determines that the toggle switch 98 is on the ON side (S43: Yes ), the number of mob characters 52A and 52B included in the angle of view of the virtual camera 92 is compared with a predetermined threshold value (S44). Then, when the number of mob characters 52A and 52B included in the angle of view of the virtual camera 92 is less than the threshold (S44: No), the user terminal 20A (image generating means 240) skips the processing of step S45, and skips step S45. The processing after S46 is executed. That is, the user terminal 20A (image generating means 240) includes all the mob characters 52A and 52B included in the angle of view of the virtual camera 92 in the virtual space image 97. FIG.

Next, the user terminal 20A (image generating means 240) compares the distance from the other person's avatar 51B and the mob character 52A included in the virtual space image 97 to the virtual camera 92 with a predetermined threshold distance (S46). Then, as shown in FIG. 13A, for example, the user terminal 20A (image generating means 240) detects the distance from the other person's avatar 51B and the mob characters 52A and 52B to the virtual camera 92 when it is equal to or greater than the threshold distance indicated by the dashed-dotted line. (S46: Yes), as shown in FIG. 13B, the other person's avatar 51B and the mob characters 52A and 52B are included in the virtual space image 97 in a common display mode (S47). On the other hand, as shown in FIG. 14A, the user terminal 20A (image generating means 240), for example, when the distance from the other person avatar 51B and the mob characters 52A and 52B to the virtual camera 92 is less than the threshold distance indicated by the dashed-dotted line, (S46: No), as shown in FIG. 14B, another person's avatar 51B and mob characters 52A and 52B are included in the virtual space image 97 in different display modes (S48).

Here, the other person's avatar 51B and the mob characters 52A and 52B have different appearances. Therefore, "a common display mode" does not mean that the appearance of each character is completely matched. "Common display mode" may mean, for example, not adding information (icons) for distinguishing between avatars and mob characters. On the other hand, "different display modes" may mean adding information (icons) for distinguishing between avatars and mob characters. For example, in the example of FIG. 14B, the user name "USER-B" set by the user of the user terminal 20B is added above the head of another person's avatar 51B, while the mob characters 52A and 52B have A character string of "MOB" is added. On the other hand, in FIG. 13B, the character strings above the other person's avatar 51B and the mob characters 52A and 52B are omitted. However, specific examples of the common display mode and different display modes are not limited to the above examples.

15(A), the user terminal 20A (image generating means 240) determines the number of mob characters 52A, 52B, 52E, 52F, 52G, 52H, 52I, and 52J included in the angle of view of the virtual camera 92. is equal to or greater than the threshold (S44: Yes), the number of mob characters 52A, 52B, 52E to 52J included in the virtual space image 97 is reduced (S45). User terminal 20A (image generating means 240), for example, when the number of 52A, 52B, 52E to 52J is equal to or greater than a first threshold (eg, 5) and less than a second threshold (eg, 10), the first threshold A mob character may be included in the virtual space image 97 . Further, the user terminal 20A (image generating means 240) may include the second threshold mob characters in the virtual space image 97 when the number of mob characters is equal to or greater than the second threshold. Note that the first threshold<the second threshold. However, the method of reducing the number of mob characters is not limited to the above example. In addition, the user terminal 20A (image generating means 240) may reduce the number of mob characters, for example, preferentially in areas where the density of mob characters is high. However, which mob character is to be reduced is not limited to the above example.

Next, the user terminal 20A (image generating means 240) executes the processes of steps S46 to S48 for the other person's avatar 51B and the mob characters 52A, 52E, 52G, 52H, and 52J that were not reduced in step S45. That is, the user terminal 20A (image generating means 240) adds the character string "MOB" above the heads of the mob characters 52H and 52J whose distance from the virtual camera 92 is less than the threshold (S47), and the distance from the virtual camera 92 is the threshold. No character string is added above the heads of the other person avatar 51B and the mob characters 52A, 52E, and 52G (S48).

Next, the user terminal 20A (image generating means 240) generates a virtual space image 97 including the other person's avatar 51B and the mob character 52 in the example of FIG. 13(A), FIG. 14(A), or FIG. 14(A). (S49). Next, the user terminal 20A (image display means 250) displays the generated virtual space image 97 on the monitor 31 as shown in FIG. 13(B), FIG. 14(B), or FIG. 15(B) ( S51).

On the other hand, when the user terminal 20A (image generating means 240) determines that the toggle switch 98 is on the OFF side (S43: No) as shown in FIG. Then, the process of step S50 is executed. That is, the user terminal 20A (image generating means 240) includes the other person's avatar 51B included in the angle of view of the virtual camera 92, and does not include the mob characters 52A, 52B, 52E to 52J included in the angle of view of the virtual camera 92. Generate image 97 . Next, the user terminal 20A (image display means 250) displays the generated virtual space image 97 on the monitor 31 as shown in FIG. 16B (S51).

Returning to FIG. 10 , the user terminal 20A (arrangement means 220 and camera movement means 230) accepts an operation to instruct the action of the principal avatar 50 through the operation device 42 (S27: Yes), the user terminal 20A for the operation device 42 , the virtual camera 92 is moved following the avatar 50, and avatar data indicating the avatar 50 after the action is transmitted to the server 10 through the communication network 2 (S28). Also, the user terminal 20A (camera movement means 230) moves the virtual camera 92 following the movement of the user terminal 20A detected by the movement sensor 41. FIG. Then, the user terminal 20A (image generation means 240 and image display means 250) executes virtual space image generation and display processing in order to generate the virtual space image 97 corresponding to the angle of view after the virtual camera 92 is moved. (S29).

The user terminal 20A repeatedly executes the processes of steps S23 to S29 until an operation (for example, logout) instructing termination of the terminal program 23P is received through the operation device 42 (S30: No). The user terminal 20A irregularly receives avatar data and mob character data, and irregularly accepts user operations. That is, the processes of steps S24, S26, and S28 are performed at arbitrary timing and frequency. Then, when the user terminal 20A receives an operation for instructing termination of the terminal program 23P through the operation device 42 (S30: Yes), the processing of FIG. 10 is terminated.

[Effect of Example 1]
According to the first embodiment, by arranging the mob character 52 corresponding to the person in the real space in the virtual space 90, the user of the user terminal 20A who operates the person avatar 50 in the virtual space 90 can see the situation in the real space (for example, , the degree of crowding of people going out on the street). As a result, it is possible to give the user who is having a simulated experience in the virtual space 90 motivation to actually go to the real space.

Further, according to the first embodiment, by arranging the mob character 52 reflecting the attributes of a person in the real space in the virtual space 90, it is possible to report the situation in the real space in more detail. As a result, the user who is having a simulated experience in the virtual space 90 is more motivated to actually go to the real space.

Further, according to the first embodiment, by changing the display modes of the other avatar 51B and the mob characters 52A and 52B whose distance from the virtual camera 92 is less than the threshold (FIG. 14), the other avatar 51B and the mob characters 52A and 52B can be displayed. , can be distinguished by the user of the user terminal 20A. This makes it possible to provide appropriate virtual space images 97 to both the user who wants to know the situation in the real space and the user who wants to spend time in the virtual space 90 (in other words, want to interact with the other person's avatar 51B). .

Further, according to the first embodiment, by sharing the display mode of the other person avatar 51B and the mob characters 52A and 52B whose distance from the virtual camera 92 is equal to or greater than the threshold (FIG. 13), the information included in the virtual space image 97 is You can prevent it from becoming too much. However, regardless of the distance from the virtual camera 92, the display modes of all other avatars and mob characters included in the field of view area 94 may be changed.

Further, according to the first embodiment, when there are many people in the real space, the number of mob characters 52 is reduced and included in the virtual space image 97, so that the number of mob characters 52 in the virtual space image 97 does not become too large. can be prevented. This prevents the mob character 52 from blocking the view of the user who wants to enjoy the virtual space 90 .

Furthermore, according to the first embodiment, it is possible to switch between display and non-display of the mob character 52 by operating the toggle switch 98, so that the view of the user who wants to enjoy the virtual space 90 is prevented from being blocked by the mob character 52. can be effectively prevented.

[Example 2]
Next, operations of the system 1 according to the second embodiment will be described with reference to FIGS. 17 to 22. FIG. Note that detailed descriptions of common points with the first embodiment will be omitted, and processing specific to the second embodiment will be mainly described. Also, the first and second embodiments can be combined in part or in whole.

In the second embodiment, the avatar of the user of the user terminal 20A (first user terminal) is a customer, and the avatar of the user of the user terminal 20C (second user terminal) is a staff member in the virtual space 90 corresponding to the actual store. It is assumed that The system 1 according to the second embodiment notifies the user of the user terminal 20A of the corresponding situation in the real space through the virtual space 90, and provides communication with the staff in the real store through the virtual space 90.

FIG. 17 is an example of a message table that holds personal attributes and messages in association with each other. FIG. 18 is a flowchart illustrating processing of the user terminal 20C according to the second embodiment. FIG. 19 is a diagram showing a status setting screen. FIG. 20 is a flowchart illustrating processing of the user terminal 20A according to the second embodiment. FIG. 21 is a diagram showing a virtual space image 97 when another person's avatar 53 in the operable state is selected. FIG. 22 is a diagram showing a virtual space image 97 when the other person's avatar 53 and the mob character 54 in the inoperable state are selected.

The storage 13 of the server 10 according to the second embodiment holds the message table shown in FIG. In the message table, attributes of persons that can be extracted by the person extracting means 120 in step S14 and messages are registered in association with each other. The message table is registered in the server 10 in advance by, for example, the manager of the store or the user of the user terminal 20C. Messages registered in the message table may be in text format or in voice format.

Then, the server 10 (mob character generating means 130) extracts a message corresponding to the attribute of the generated mob character from the message table, and adds the extracted message to the mob character data (S15). Further, the server 10 (mob character generating means 130) transmits the mob character data with the message added to all the user terminals 20A to 20C through the communication network 2 (S16).

As shown in FIG. 18, the user terminal 20C (arrangement means 220) causes the monitor 31 to display the status setting screen shown in FIG. 19 (S61). The status setting screen is a screen for setting the status of the avatar of the user of the user terminal 20C. More specifically, the status setting screen accepts operations for selecting the value of the operable flag and setting the avatar's appearance (eg, facial expression, gesture) and fixed message corresponding to the value of the operable flag. The operable flag is set to a first value "ON" indicating that the avatar is in an operable state, or a second value "OFF" indicating that the avatar is in an inoperable state. can do.

Then, the user terminal 20C (arrangement means 220) waits until the operation of selecting the value of the operable flag, the operation of setting the appearance of the avatar, and the operation of inputting a fixed message are received through the operation device 42, and the subsequent processing is not performed. Wait for execution. An operation of selecting the value of the operable flag is an example of a selection operation of selecting either the operable state or the inoperable state. Note that the interface for selecting the value of the operable flag, setting the appearance of the avatar, and receiving the input of the fixed message is not limited to the example of FIG.

When the user terminal 20C (arrangement unit 220) receives an operation to select the first value "ON" as the set value of the operable flag through the operation device 42 (S62: Yes), the operable flag "ON", the operation Avatar data including the appearance of the avatar set in association with the operable flag "ON" and the standard message input in association with the operable flag "ON" are transmitted to the server 10 via the communication network 2 (S63). Further, when the user terminal 20C (arrangement unit 220) receives an operation to select the second value "OFF" as the set value of the operable flag through the operation device 42 (S64: Yes), the operable flag is set to "OFF". , the appearance of the avatar set in association with the operability flag "OFF", and the avatar data including the fixed message input in association with the operability flag "OFF" are transmitted to the server 10 through the communication network 2 (S65). ).

The user of the user terminal 20C can switch the set value of the operable flag at any timing through the status setting screen. Then, the user terminal 20C (arrangement means 220) transmits avatar data including the operable flag to the server 10 each time the set value of the operable flag (that is, the operable state and the inoperable state of the avatar) is switched. .

The server 10 (avatar synchronization means 110) transmits the avatar data received from the user terminal 20C to all other user terminals 20A and 20B through the communication network 2 (S13). Also, the user terminals 20A and 20B (arrangement means 220) arrange other avatars in the virtual space 90 according to the avatar data received from the server 10 (S24). Further, when the other person's avatar is included in the field of view area 94 (S42: Yes), the user terminals 20A and 20B (image generating means 240) generate the virtual space image 97 including the other person's avatar (S49, S50).

20 will be described below assuming that the user terminal 20A has received the mob character data including the message shown in FIG. 17 and the avatar data transmitted from the user terminal 20C in steps S63 and S65.

As an example, when the operable flag of the avatar data transmitted from the user terminal 20C is set to the first value "ON", the user terminal 20A (image generating means 240) moves to the virtual space shown in FIG. An image 97 is generated (S49). As another example, when the operable flag of the avatar data transmitted from the user terminal 20C is set to the second value "OFF", the user terminal 20A (image generating means 240) is shown in FIG. A virtual space image 97 (however, at this time, no balloon containing a fixed message is displayed) is generated (S49). Then, the user terminal 20A (image display means 250) causes the monitor 31 to display the virtual space image 97 generated in step S49 (S51).

A virtual space image 97 shown in FIGS. 21A and 22A includes a stranger avatar 53 based on the avatar data transmitted from the user terminal 20A and a mob character 54 based on the mob character data transmitted from the server 10. common point. On the other hand, the other avatars 53 shown in FIGS. 21A and 22A differ in appearance (for example, facial expressions and gestures). Also, in FIGS. 21A and 22A, the display mode of the user name "USER-C" added to the other person's avatar 53 is different (grayed out in FIG. 22A).

In other words, the user terminal 20A (image generating means 240) performs a selection operation for selecting the setting value of the operable flag (that is, the operable state and the inoperable state of the other avatar 53) on the user terminal 20C. Then, the display mode is switched to another person's avatar 53 . The display mode of the other person's avatar 53, which is switched according to the set value of the operable flag, is not limited to the examples of FIGS. 21A and 22A. Any mode is acceptable.

Next, when the user terminal 20A (image display means 250) receives an operation to select the other person avatar 53 shown in FIG. 21A or FIG. 21(B) or as shown in FIG. 22(A), the standard message included in the avatar data is output (S72). In the examples of FIGS. 21B and 22A, a standard message in text format is displayed in a speech bubble from another person's avatar 53. In the example of FIG. As another example, the user terminal 20</b>A may output a standard voice message from the speaker 36 . Also, in the example of FIG. 21A, the pointer 55 is used to select the other person's avatar 53, but the specific method of selecting the character is not limited to this.

Next, the user terminal 20A (image display means 250) determines the set value of the operable flag included in the avatar data (S73). Then, when the user terminal 20A (image display means 250) determines that the operable flag is set to the first value "ON" (S73: Yes), as shown in FIG. and a [Send] button 57 are displayed on the monitor 31. Note that the user terminal 20</b>A is not limited to inputting a message in text format through the text box 56 , and may accept inputting a message in voice format through the microphone 35 .

Next, the user terminal 20A (placement unit 220) enters a message in the text box 56 and until the operation of pressing the [Send] button 57 is accepted through the operation device 42 (S74: No), the process of step S75 is stopped. Wait for execution. Then, when the user terminal 20A (arrangement means 220) accepts the operation of pressing the [Send] button 57 (S74: Yes), the message "Please tell me the recommended products." , and the terminal ID "TERM-C" of the user terminal 20C corresponding to the other person's avatar 53 are transmitted to the server 10 via the communication network 2 (S75). Further, when receiving avatar data including a terminal ID, the server 10 (avatar synchronization means 110) transmits (relays) the avatar data only to the user terminal 20C identified by the terminal ID "TERM-C". do.

On the other hand, when the user terminal 20A (image display means 250) determines that the second value "OFF" is set in the operable flag (S73: No), it skips the processing of steps S74 and S75. That is, as shown in FIG. 22A, the text box 56 and the [Send] button 57 are not displayed on the monitor 31, and the message cannot be sent to the user terminal 20C.

Returning to FIG. 18, when avatar data including a message is received from the server 10 (S66: Yes), the user terminal 20C (image display means 250) outputs the message in association with the avatar of the user terminal 20A ( S67). Then, the user terminal 20C (arranging means 220) waits for execution of the process of step S69 until an operation of inputting a reply message is received through the operation device 42 (S68: No). Next, when a reply message is input (S68: Yes), the user terminal 20C (arranging means 220) transmits the reply message and the avatar data including the terminal ID "TERM-A" of the user terminal 20A to the communication network. 2 to the server 10 (S69). The processing of steps S67-S69 is common to steps S72 and S74-S75.

By repeating the processing of steps S72 and S74-S75 executed by the user terminal 20A and the processing of steps S67-S69 executed by the user terminal 20C, the user of the user terminal 20A and the user terminal 20C Communication is established through the virtual space 90 with the user (that is, store staff).

Returning to FIG. 20, when the user terminal 20A (image display means 250) receives an operation to select the mob character 54 shown in FIG. 21A through the operation device 42 (S76: Yes), , the message "This shop is easy to enter even with children" included in the mob character data is output (S77). In the example of FIG. 22(B), a message in text format is displayed in a speech bubble from the mob character 54 . As another example, the user terminal 20A may output a voice message from the speaker 36. FIG.

[Effect of Example 2]
According to the second embodiment, it is possible to notify the user of the user terminal 20A by the display mode of the other person's avatar 53 whether or not the store staff can operate the other person's avatar 53 . As a result, it is possible to reduce the stress of the user of the user terminal 20A due to sending a message to the other person's avatar 53 (speaking to the other person's avatar 53) and not receiving a reply. In other words, it is not necessary to station staff near the user terminal 20C in order to make the other person's avatar 53 operable at all times in the store.

Further, according to the second embodiment, it is possible to inform the user of the user terminal 20A of a message that has been registered in advance in association with the attribute through the mob-chara 54. FIG. As a result, for example, by registering a message appealing the store, the user of the user terminal 20A can be motivated to go to the actual store. Note that the message associated with the mob character 54 is not limited to being stored in advance in the storage 13 of the server 10, and may be automatically created using an AI function, or may be a word-of-mouth message with a high rating on a word-of-mouth site on the Internet. may be quoted.

[Other embodiments]
In addition, in the present invention, if the user of the user terminal 20A can estimate the crowd of people in the real space from the number of mob characters placed in the virtual space 90, the people detected by the detection device 3 and the virtual It is not necessary to have a one-to-one correspondence with the mob characters placed in the space 90 . More specifically, the number of people detected by the detection device 3 and the number of mob characters placed in the virtual space 90 need not be the same as long as there is a positive correlation. That is, the arranging means 220 may arrange the number of mob characters in the virtual space 90 based on the detection result of the detection device 3 (for example, the number of people, density, etc.). More specifically, the placement unit 220 may place a mob character corresponding to the person detected by the detection device 3 in the virtual space 90 .

Also, the objects arranged in the virtual space 90 by the arrangement means 220 are not limited to mob characters (non-user characters) corresponding to people entering and exiting the real space. Other examples of objects may correspond to objects that may enter and exit the real space, such as vehicles (automobiles, ships, aircraft), animals, plants, products, and exhibits. Furthermore, the objects placed in the virtual space 90 are not limited to one type, and may be of multiple types.

As an example, by arranging an object corresponding to a car in the real space in the virtual space 90 including roads and parking lots, the user of the user terminal 20A can recognize traffic congestion and parking space availability through the virtual space 90. . As another example, by arranging objects corresponding to animals in the real space and mob characters corresponding to visitors in the virtual space 90 including zoos and parks, the user of the user terminal 20A can easily see the animals through the virtual space 90. You can recognize where you can see it.

Also, the program according to the present invention is not limited to a single program, and may be an aggregate of a plurality of programs. Moreover, the program according to the present invention is not limited to being executed by a single device, and may be executed by a plurality of devices in a shared manner. Furthermore, the division of roles between the server 10 and the user terminal 20 is not limited to the above example. That is, part of the processing of the server 10 may be executed by the user terminal 20 , and part of the processing of the user terminal 20 may be executed by the server 10 .

Furthermore, part or all of each means realized by the program can also be realized by hardware such as an integrated circuit. Furthermore, the program may be provided by being recorded on a computer-readable non-transitory recording medium. Recording media refer to, for example, hard disks, SD cards, DVDs, servers on the Internet, and the like.

DESCRIPTION OF SYMBOLS 1... System, 2... Communication network, 3... Detecting device, 10... Server, 11, 21... Processor, 12, 22... Memory, 13, 23... Storage, 13P... Server program, 14, 24... Input/output interface, 15 , 25... communication interface, 19, 29... communication bus, 20... user terminal, 23P... terminal program, 26... computer, 30... HMD, 31... monitor, 32... gaze sensor, 33, 34... camera, 35... microphone, 36... speaker, 41... motion sensor, 42... operating device, 50... person's avatar, 51, 53... other person's avatar, 52, 54... mob character, 55... pointer, 56... text box, 57... [send] button, 90... Virtual space 91 Panorama image 92 Virtual camera 93 Reference line of sight 94 Field of view area 95, 96 Area 97 Virtual space image 98 Toggle switch 110 Avatar synchronization means 120 People extraction Means 130 Mob character generation means 210 Virtual space definition means 220 Placement means 230 Camera movement means 240 Image generation means 250 Image display means

Claims (5)

the computer,
a virtual space defining means for defining a virtual space corresponding to the real space;
arranging means for arranging, in the virtual space, the number of objects based on the detection result of the detection device installed in the physical space ;
an image generating means for generating a virtual space image obtained by imaging the inside of the virtual space with a virtual camera;
functioning as image display means for displaying the virtual space image on the first user terminal ;
The image generation means generates a message registered in advance in association with the attribute detected by the detection device, based on the selection operation of the object performed on the first user terminal, to the selected object. A program that outputs through In the program according to claim 1,
A program according to claim 1, wherein the arranging means arranges the object in the virtual space in a manner corresponding to the attribute detected by the detector. In the program according to claim 1,
A program in which the placement means places a non-user character corresponding to the person detected by the detection device in the virtual space as the object. In the program according to claim 3 ,
The image generation means is
generating the virtual space image including the non-user character included in the angle of view of the virtual camera based on a display operation performed on the first user terminal;
A program for generating the virtual space image that does not include the non-user character included in the angle of view of the virtual camera, based on a non-display operation performed on the first user terminal. the computer,
a virtual space defining means for defining a virtual space corresponding to the real space;
arranging means for arranging, in the virtual space, the number of objects based on the detection result of the detection device installed in the physical space;
camera movement means for moving the virtual camera within the virtual space based on a movement operation performed on the first user terminal;
image generation means for generating a virtual space image obtained by imaging the inside of the virtual space with the virtual camera;
functioning as image display means for displaying the virtual space image on the first user terminal ;
The arranging means is
arranging a non-user character corresponding to the person detected by the detection device in the virtual space as the object;
arranging a user character operated through a second user terminal in the virtual space;
The image generation means is
Among the user character and the non-user character included in the angle of view of the virtual camera, the user character and the non-user character whose distance from the virtual camera is less than a threshold distance are displayed in the virtual space image in different display modes. including
A program for including the user character and the non-user character whose distance from the virtual camera is equal to or greater than the threshold distance in the virtual space image in a common display mode.

Images