JP2024070696A - Information processing system, information processing method, and program - Google Patents

Abstract

[Problem] To provide a system, method, and program capable of further improving the convenience of communication between some users at a specified location in a virtual space. [Solution] In a virtual space system, a content distribution server places virtual objects in the virtual space, including display objects that display information related to an event taking place in the real space or virtual space that is the subject of a conversation in a communication room. If the position in the virtual space of an avatar operated by a user is included in a first area that is a communication point associated with a communication room where communication between users takes place, a virtual space management server transmits information related to the selection of entering the communication room to a terminal used by the user, and if the position is included in a second area located outside the first area, transmits information related to the communication to the terminal. [Selected Figure] Figure 11

Description

This disclosure relates to an information processing system, an information processing method, and a program.

In recent years, systems have become widespread that allow users to communicate through avatars operated by the users in a virtual space in which 3D models are placed, as a form of real-time communication using images, audio, or text. In such systems, images of the virtual space are generated from any viewpoint (free viewpoint) in response to user operations, for example, and provided to the user. Images of the virtual space are provided using display devices such as HMDs (Head Mounted Displays), smartphones, tablet terminals, and PCs (Personal Computers), which cover the user's entire field of vision.

Regarding communication between users in a virtual space, for example, the following Patent Document 1 discloses a technology for establishing communication between communication terminals used by scheduled users when the scheduled users gather in a predetermined space and at a predetermined time in the virtual space.

International Publication No. 2013/054748

However, with the above-mentioned technology, it was necessary to determine in advance the participants, space, and time of communication. There is a demand for greater convenience in communication between certain users in virtual space.

Therefore, this disclosure proposes an information processing system, information processing method, and program that can further improve the convenience of communication between some users at a specific location in a virtual space.

According to the present disclosure, an information processing system is provided that includes a control unit that performs the following controls: placing, in a virtual space, virtual objects including a display object that displays information related to an event taking place in a real space or virtual space that is the subject of a conversation in a communication room; transmitting information related to a selection to enter the communication room to a terminal used by a user when the position of an avatar operated by a user in the virtual space is included in a first area that is a communication point associated with the communication room where communication between users takes place; and transmitting information related to the communication to the terminal when the position of the avatar in the virtual space is included in a second area located outside the first area.

The present disclosure also provides an information processing method including: a processor arranging, in a virtual space, virtual objects including a display object that displays information related to an event taking place in a real space or virtual space that is the subject of a conversation in a communication room; transmitting information related to a selection to enter the communication room to a terminal used by the user when the position of an avatar operated by a user in the virtual space is included in a first area that is a communication point associated with the communication room where communication between users takes place; and transmitting information related to the communication to the terminal when the position of the avatar in the virtual space is included in a second area located outside the first area.

According to the present disclosure, a program is provided that causes a computer to function as a control unit that performs the following operations: control to place, in a virtual space, virtual objects including a display object that displays information related to an event taking place in a real space or virtual space that is the subject of a conversation in a communication room; control to transmit information related to a selection to enter the communication room to a terminal used by the user when the position of an avatar operated by a user in the virtual space is included in a first area that is a communication point associated with the communication room where communication between users takes place; and control to transmit information related to the communication to the terminal when the position of the avatar in the virtual space is included in a second area that is located outside the first area.

FIG. 1 is a diagram illustrating an overall configuration of a virtual space providing system according to an embodiment of the present disclosure. 2 is a diagram illustrating each space included in a virtual space S. FIG. 11 is a diagram illustrating a genre-specific space included in the sports space S3 according to the present embodiment. FIG. 2 is a block diagram showing an example of the configuration of a virtual space management server 200 according to the present embodiment. FIG. FIG. 2 is a diagram showing a specific example of a service server included in a back-end system 30 according to the present embodiment. FIG. 2 is a block diagram showing an example of a configuration of a client terminal 10 according to the present embodiment. FIG. 11 is a sequence diagram showing an example of the flow of a connection process to a virtual space according to the present embodiment. FIG. 4 is a diagram showing an example of a display screen according to the present embodiment. FIG. 4 is a diagram showing an example of a display screen according to the present embodiment. FIG. 2 is a diagram showing an example of a simple map of an IP content specialization area according to the present embodiment. 1A to 1C are diagrams illustrating a mechanism for reproducing a match using bone data of players according to an embodiment of the present invention. FIG. 6 is a diagram showing a display screen inside a bar in which a talking table 600 according to the present embodiment is placed. 6 is a diagram illustrating an example of the shape of a talking table 600. FIG. 11 is a diagram illustrating a case where the state of the inside of a talking room is disclosed depending on the distance from a talking table 600 according to this embodiment. FIG. 11A and 11B are diagrams showing an example of an effect display on the talking table 600 according to the excitement in the talking room according to the present embodiment. FIG. 11 is a sequence diagram showing an example of an operational process from generation of the talking table 600 to entry according to this embodiment. 1 is a diagram illustrating details of communication connections between a client terminal 10 and each server when entering a talking room according to this embodiment. FIG. FIG. 13 is a diagram showing an example of a talking room screen according to the present embodiment. FIG. 11 is a diagram showing another example of the talking room screen according to the present embodiment. FIG. 2 is a diagram illustrating a mechanism for event linkage according to the present embodiment. FIG. 2 is a diagram showing an example of a display screen in a live stadium according to the present embodiment. FIG. 2 is a diagram showing an example of a display screen in a live stadium according to the present embodiment. FIG. 2 is a diagram showing an example of a display screen in a live stadium according to the present embodiment. FIG. 2 is a diagram showing an example of a display screen in a live stadium according to the present embodiment. FIG. 2 is a diagram showing an example of a display screen in a live stadium according to the present embodiment. FIG. 2 is a diagram showing an example of a display screen in a live stadium according to the present embodiment. FIG. 4 is a diagram illustrating an example of a camera position according to the present embodiment. FIG. 28 is a diagram showing the game replay UI in the case of the camera position 810 shown in FIG. 27: Basic. FIG. 28 is a diagram showing a game replay UI in the case where the camera position 811 shown in FIG. 27 is Bird's eye. 11 is a diagram illustrating an outline of a screen of a match reproduction UI according to the present embodiment. FIG. FIG. 11 is a diagram showing an example of a chapter list display according to the present embodiment. 13A to 13C are diagrams illustrating an example of a display of a match reproduction UI in the avatar mode according to the present embodiment. FIG. 1 is a diagram illustrating highlight playback of a match according to an embodiment of the present invention.

A preferred embodiment of the present disclosure will be described in detail below with reference to the attached drawings. Note that in this specification and drawings, components having substantially the same functional configuration are designated by the same reference numerals to avoid redundant description.

The explanation will be given in the following order:
1. Overall configuration of a virtual space providing system according to an embodiment of the present disclosure 2. Configuration examples 2-1. Configuration example of a virtual space management server 200 2-2. Configuration example of a back-end system 30 2-3. Configuration example of a client terminal 10 3. Connection to a virtual space 4. Display screen examples 5. Specific examples of a virtual space 5-1. IP content area 5-2. Talking table 5-3. Event linkage 5-4. Match replay UI
6. Supplementary Information

<<1. Overall configuration of a virtual space providing system according to an embodiment of the present disclosure>>
1 is a diagram illustrating an overall configuration of a virtual space providing system according to an embodiment of the present disclosure. The virtual space providing system according to this embodiment (an example of an information processing system) is an information processing system including one or more virtual space management servers 200 (an example of an information processing device), client terminals 10 (an example of terminals) used by each user, and a back-end system 30.

The virtual space management server 200 constructs (generates) and manages a virtual space (VR: Virtual Reality) in which virtual objects are placed, and provides virtual space information to one or more client terminals 10 that are connected for communication via a network 40. The virtual space information is, for example, information on virtual objects placed in the virtual space, map information on the virtual space, information on events held in the virtual space (hereinafter also referred to as virtual events), and position information of each user in the virtual space. Based on this information, the virtual space can be reproduced (copied) on each client terminal 10 side. Specifically, each client terminal 10 places virtual objects based on the map information of the virtual space, and draws a virtual space image from a user's viewpoint (user avatar's viewpoint). The virtual space management server 200 may generate an image from an arbitrary viewpoint (for example, a user's viewpoint) in the virtual space and transmit it to the client terminal 10.

In the virtual space, virtual objects formed by, for example, a 3DCG model are placed. The virtual space may be a space in which the user's actions are restricted to a prepared action scenario, as in a game, or a space in which the user can act freely without being restricted by an action scenario. The virtual space may also be a space in which users can interact with other users by sending and receiving voice or text in both directions. The virtual space may also be a space designed with an original worldview, or a so-called mirror world that faithfully reproduces buildings, facilities, roads, cities, streetscapes, etc. that exist in the real space. In the virtual space, users may also interact with other users, receive and give education, or engage in social activities such as work, and economic activities such as selling or purchasing goods. The virtual space may also be a space called a metaverse.

In the virtual space, a user avatar may exist, which is an avatar of the user. The user avatar is generated, for example, from a character 3D model. Each user can customize the appearance of their own character 3D model as desired. The user avatar may be operated by the user to move within the virtual space. The gestures or facial expressions of the user avatar may also be operated by the user. In this specification, the position information of the user in the virtual space may be information indicating the position of the user avatar. Furthermore, the user's viewpoint in the virtual space may be the viewpoint of the user avatar, or may be a viewpoint that includes the user avatar within the angle of view. The user's viewpoint may be arbitrarily switched by the user's operation.

A user avatar is one of the virtual objects arranged in the virtual space. Information on the user avatar of each user is also shared as virtual space information by a large number of client terminals 10 that are connected to the virtual space management server 200 for communication. In addition, each user can communicate in the virtual space by voice, text, or image. The communication may be performed via the virtual space management server 200 or via the back-end system 30. The back-end system 30 includes servers (service servers 31, 32, 33, ...) that provide various services to users who use the virtual space. As an example, there are a server that provides communication by voice, a server that provides communication by text, and a server that provides communication by images (video, still image). The client terminal 10 acquires communication channel information (an example of connection information) from the virtual space management server 200 and connects to a specific service server for communication. Then, the client terminal 10 can communicate with other client terminals 10 that have similarly acquired communication channel information and connected to a specific service server through the service server.

In addition, in the virtual space providing system according to this embodiment, in order to distribute the processing load of the virtual space management server 200, the server may be multiplexed according to the number of users of the virtual space, i.e., the number of terminals connected to the virtual space management server 200. In the example shown in FIG. 1, the virtual space system 20 is composed of multiple virtual space management servers 200 (200a, 200b, 200c, ...). For example, if it is desired to allow 1000 people to use a certain virtual space, 10 virtual space management servers 200 that can be connected to 100 terminals are prepared, and each virtual space management server 200 provides the same virtual space (the same map information and the same virtual objects). Each virtual space management server 200 manages the location information of users (the location information of 100 people) corresponding to the 100 terminals connected to it for communication. This makes it possible to provide the same virtual space to a total of 1000 users, although each 100 users separately. Although the virtual spaces are roughly synchronized in terms of the flow of time and the execution of virtual events, interactions between users can be carried out for each virtual space.

Server multiplexing according to the number of connected terminals, i.e., server increase/decrease management, may be performed by a service provided by the back-end system 30. Specifically, when the number of connected terminals to one virtual space management server 200 exceeds a predetermined number (e.g., 100), the back-end system 30 may construct a new virtual space management server 200 that provides the same virtual space (the same map information and the same virtual objects) as the one virtual space management server 200. Note that a predetermined number of virtual space management servers 200 may be constructed in advance.

The server environment is not particularly limited. For example, the virtual space management server 200 may be a physical server, or a virtual server executed on a physical server. The physical server or virtual server described here may be a server provided by a hosting service, or a server prepared by a company that provides a service offering virtual space. Furthermore, the functions of one virtual space management server 200 may be realized by multiple physical servers or multiple virtual servers. Each virtual space management server 200 manages the location information of users corresponding to multiple client terminals 10 connected to it for communication.

The client terminal 10 is a terminal used by a user. For example, the client terminal 10 is realized by a smartphone, a tablet terminal, a PC (personal computer), an HMD (head mounted display) that covers the entire field of view, a glasses-type device, a projector, or a console game machine. The client terminal 10 reproduces the virtual space based on the information of the virtual space received from the virtual space management server 200. The client terminal 10 also generates and displays an image from the user's viewpoint in the virtual space. The client terminal 10 also appropriately outputs audio within the virtual space. The client terminal 10 may receive from the virtual space management server 200 an image from the user's viewpoint generated by the virtual space management server 200 and display it.

The client terminal 10 may control virtual objects placed in a virtual space in response to user operations. For example, the client terminal 10 controls the position, gestures, facial expressions, etc. of a user avatar in response to user operations. In addition, the client terminal 10 transmits information about virtual objects that have changed in response to user operations (for example, the position, gestures, facial expressions, etc. of the user avatar) to the virtual space management server 200 in real time.

In addition, the client terminal 10 may transmit various information to the virtual space management server 200, other client terminals 10, or the back-end system 30 in response to user operations. For example, the client terminal 10 may transmit login information to the virtual space management server 200 or the back-end system 30 to request user authentication. In addition, the client terminal 10 may transmit the user's voice, input text information, selected images, etc. as communication information to the virtual space management server 200, other client terminals 10, or the back-end system 30.

(Virtual space by content)
In the present embodiment, an example of a virtual space is a space for users to interact with each other, mainly focusing on various contents such as movies, music, sports, or animation (including IP contents to be described later). FIG. 2 is a diagram for explaining each space included in the virtual space S. As shown in FIG. 2, the virtual space S provided by the virtual space system 20 according to the present embodiment includes a user personal space S1, which is a virtual space for a personal user, a public space S2, which is a virtual space in which users interact with each other regardless of content, and each virtual space in which users interact with each content. Examples of each virtual space in which users interact with each content include a sports space S3, which is a virtual space in which users interact with each other about sports, a music space S4, which is a virtual space in which users interact with each other about music, a movie space S5, which is a virtual space in which users interact with each other about movies, and an anime space S6, which is a virtual space in which users interact with each other about anime. The contents listed here are examples, and the contents according to the present disclosure are not limited to these.

Each virtual space in which users interact with each content may be a further hierarchical virtual space by genre. FIG. 3 is a diagram for explaining the genre-specific spaces included in the sports space S3 according to this embodiment. As shown in FIG. 3, there are a soccer space S3-1 in which users interact with each other particularly in soccer, a basketball space S3-2 in which users interact with each other particularly in basketball, and a tennis space S3-3 in which users interact with each other particularly in tennis. In addition, as a virtual space in a lower hierarchy of the genre-specific spaces of each sport shown in FIG. 3, a virtual space in which users (fans) of each sports team interact with each other may be considered. Note that the virtual space is not limited to being hierarchical by genre. The virtual space may be a space in which each space corresponding to various IP contents exists in parallel. For example, the virtual space according to this embodiment may be a virtual space in which spaces corresponding to a certain baseball team, a certain soccer team, and a certain anime exist in parallel.

In this way, a content-specific virtual space can be thought of as a virtual space where users who are interested in that content can interact with each other. Users can talk about the content with other users who share the same interest, and enjoy the content even more.

The virtual space system 20 may provide all or some of the virtual spaces as shown in Figures 2 and 3. For example, the virtual space system 20 may provide a user's personal space S1, a public space S2, and a virtual space for interaction between fans of a specific sports team. In this embodiment, as an example, the virtual space system 20 provides a virtual space for interaction between fans of a specific soccer team.

<<2. Configuration example>>
<2-1. Example of configuration of virtual space management server 200>
4 is a block diagram showing an example of the configuration of the virtual space management server 200 according to this embodiment. As shown in FIG. 4, the virtual space management server 200 has a communication unit 210, a control unit 220, and a storage unit 230.

(Communication unit 210)
The communication unit 210 transmits and receives data to and from an external device via a wired or wireless connection. The communication unit 210 is connected to the client terminal 10 and the back-end system 30 via, for example, a wired/wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (LTE (Long Term Evolution), 4G (fourth generation mobile communication method), or 5G (fifth generation mobile communication method)), etc.

(Control unit 220)
The control unit 220 functions as a calculation processing unit and a control unit, and controls the overall operation of the virtual space management server 200 according to various programs. The control unit 220 is realized by electronic circuits such as a CPU (Central Processing Unit) and a microprocessor. The control unit 220 may also include a ROM (Read Only Memory) that stores programs to be used, calculation parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

The control unit 220 performs appropriate processing based on the data received from the external device, controls storage in the memory unit 230, and controls transmission of data to the external device.

The control unit 220 also functions as a virtual space management unit 221, a content management unit 222, and a user management unit 223.

The virtual space management unit 221 manages the virtual space. Specifically, the virtual space management unit 221 generates, acquires, adds, and updates various information for constructing (generating) the virtual space. The various information is, for example, virtual space information such as information on 3DCG models, which are virtual objects placed in the virtual space, color settings, images, map information, virtual event information, sound effects, and BGM (Background Music).

The virtual objects include user-operable objects, user-non-operable objects, background images, and effects for presentation. The virtual objects also include information about each user avatar placed in the virtual space. The information about each user avatar includes, for example, appearance information about the user avatar, real-time user position information, real-time facial expression or gesture information about the user avatar, and a profile of the corresponding user. The information about each user avatar is managed by the user management unit 223, which will be described later.

The virtual event information includes information regarding the time, progress, and presentation of the virtual event. The virtual space management unit 221 may notify the client terminal 10 to carry out a specific virtual event at a preset time, or may notify the client terminal 10 to carry out a corresponding virtual event when a specific trigger is detected in the virtual space. The virtual event information includes information required to carry out the virtual event. The virtual event may also be content. The notification of the virtual event to the client terminal 10 may include information instructing the client terminal 10 to obtain the specific content from the backend system 30 and carry it out at a specific location in the virtual space.

The virtual space management unit 221 transmits virtual space information to the client terminal 10 as appropriate. The virtual space management unit 221 can also realize communication between users sharing a virtual space by transmitting communication information such as text, audio, or images received from a client terminal 10 to another client terminal 10.

The virtual space management unit 221 receives information on user operations from the client terminal 10 and transmits it to other client terminals 10 in real time as virtual space information. The virtual space management unit 221 also performs calculation processing for collision detection according to the information on user operations received from the client terminal 10, and transmits the calculation results to each client terminal 10 as required. This synchronizes the situation of the virtual space reproduced in each client terminal 10. The information on user operations may include operations related to the movement of the user avatar, such as the movement and jumping of the user avatar, and operations related to the expression of emotions of the user avatar, such as emotes that cause the user avatar to perform poses or gestures, and stamps displayed above the user avatar's head.

The content management unit 222 manages information related to content shared in the virtual space. For example, the content management unit 222 manages information indicating content currently being shared in a content sharing area provided in the virtual space. The content sharing area may be, for example, a virtual object of one or more large displays placed in the virtual space, or a stage, plaza, or field where 3D content, etc. is reproduced. The content itself may be transmitted from the back-end system 30 to the client terminal 10. Such content may be managed by the virtual space management unit 221 as one of the virtual event information described above.

The user management unit 223 manages information about users who use the virtual space provided by the virtual space management server 200. More specifically, the user management unit 223 manages information about users corresponding to the client terminal 10 that is connected to the communication connection. The information about the user includes, for example, the user's profile (user name, user ID, icon image, etc.), appearance information of the user avatar (image, character type, or component, etc.), status information of the user avatar, information about virtual items owned by the user, information such as virtual currency or points that the user can use in the virtual space, real-time user position information in the virtual space (i.e., position information of the user avatar), and motion information such as real-time gestures, facial expressions, posture, or operating state of the user avatar. The information about the user is transmitted from the client terminal 10. In addition, part of the information about the user may be transmitted from the back-end system 30.

(Memory unit 230)
The storage unit 230 is realized by a ROM that stores programs and calculation parameters used in the processing of the control unit 220, and a RAM that temporarily stores appropriately changing parameters, etc. The storage unit 230 stores virtual space information.

<2-2. Example of the configuration of the back-end system 30>
The back-end system 30 includes servers that provide various services. Each server may be provided by a different business operator. In addition, the services provided by each server may be appropriately customized by a business operator that provides a virtual space.

Each server in the back-end system 30 may communicate with the virtual space management server 200 to send and receive information, or may communicate with the client terminal 10 to send and receive information. By cooperating with each server, the virtual space management server 200 can provide various services to users who use the virtual space.

Figure 5 is a diagram showing a specific example of a service server included in the back-end system 30 according to this embodiment. The service servers shown in Figure 5 are merely examples, and the present disclosure is not limited thereto. Each service server is described below.

The user information management server 301 manages information about users who use the virtual space. For example, the user information management server 301 stores in a data store (storage unit) user account information, user profile, user characteristics (age, sex, country, language used, career, hobbies and preferences, content viewing history, virtual event participation history, interaction history in the virtual space, etc.), item information owned by the user, user avatar appearance information, and user avatar status information. The user information management server 301 can also authenticate users using the user account information. Details will be described later with reference to FIG. 7, but the user information management server 301 performs user authentication in response to a request from the client terminal 10, and if authentication is successful, it calls up user information, such as the user profile and user avatar appearance information, from the data store and transmits it to the client terminal 10.

The hosting server 302 multiplexes the servers according to the number of connected terminals in the virtual space system 20, i.e., manages the increase and decrease of servers. For example, when the number of connected terminals to one virtual space management server 200 exceeds a predetermined number (e.g., 100), the hosting server 302 constructs a new virtual space management server 200 that provides the same virtual space (the same map information and the same virtual objects) as that one virtual space management server 200.

When there are multiple virtual space management servers 200 that provide the same virtual space, the matching server 303 determines which virtual space management server 200 the client terminal 10 that has completed user authentication and made a connection request should be connected to. As described above, for example, when the number of connections is limited for each virtual space management server 200, the matching server 303 obtains information on the current number of connections for each virtual space management server 200 from the virtual space system 20, and instructs the client terminal 10 to connect to a virtual space management server 200 that has available connections. The matching server 303 may also determine the virtual space management server 200 to connect to based on user information in addition to the number of connections. For example, the matching server 303 may instruct users with similar user characteristics to connect to the same virtual space management server 200.

The text chat server 304 is a server that provides a mechanism for users of the virtual space to converse with each other using text (character information) as a means of communication. Specifically, the text chat server 304 creates a virtual chat room (text channel) and controls the exchange of character information between users who have entered the chat room. Chat rooms are generated in response to a request from the virtual space management server 200 or the client terminal 10. The text chat server 304 also returns session information (connection information) for connecting to the generated chat room to the virtual space management server 200 or the client terminal 10. The client terminal 10 connects to the text chat server 304 for communication based on the session information and executes the text chat.

The voice chat server 305 is a server that provides a mechanism for users of a virtual space to converse with each other using voice (audio information) as a means of communication. Specifically, the voice chat server 305 creates a virtual chat room (voice channel) and controls the exchange of audio information between users who have entered the chat room. Chat rooms are created in response to a request from the virtual space management server 200 or the client terminal 10. The voice chat server 305 also returns session information (connection information) for connecting to the created chat room to the virtual space management server 200 or the client terminal 10. The client terminal 10 connects to the voice chat server 305 for communication based on the session information and executes the voice chat.

The means of communication between users using the virtual space is not limited to the text information and audio information described above, and it is also envisioned that moving images (video) may be used. Although not shown in FIG. 5, the back-end system 30 may include a server that provides a mechanism for conversations using moving images (video), for example.

The content distribution server 306 controls the distribution of content to be viewed in the virtual space. The content distribution server 306 can obtain and distribute content from various content servers (servers that store content). In this embodiment, the virtual space is reproduced (virtual objects are placed based on map information) by an application (also called a client app) running on each client terminal 10, and a virtual space image from the user's viewpoint is generated. The content distribution server 306 communicates with the client terminal 10 and transmits to the client terminal 10 content to be incorporated in the virtual space reproduced on the client terminal 10. The content to be incorporated in the virtual space is, for example, a video displayed on a large display (virtual object) placed in the virtual space. The content to be incorporated in the virtual space is, for example, a 3D image displayed in a specified field in the virtual space.

The talking table server 307 manages talking rooms (communication rooms) associated with talking tables (virtual objects) placed at any location in the virtual space. In the talking rooms, each user can talk using one or more different types of communication means (for example, text chat and voice chat). The talking rooms are generated in response to a request from the virtual space management server 200 or the client terminal 10. The talking table server 307 also returns session information (connection information) for connecting to the generated talking room to the virtual space management server 200 or the client terminal 10. Details of the talking rooms will be described later with reference to Figures 12 to 19. The talking table is an example of a virtual object that is placed at a communication point set at any location in the virtual space as a landmark for a place where multiple users gather to interact in the virtual space. Any virtual object that serves as a landmark may be placed at the communication point, and may be, in addition to a table, a chair, a pole, a parasol, or a stage, for example.

The event-linked server 308 controls the effects in the virtual space according to the progress of an event taking place in the real space. The effects in the virtual space may include virtual events. For example, the event-linked server 308 can link the effects in the virtual space to events in the real space by executing corresponding effects in real time in the virtual space according to the progress of a match taking place in the real space. More specifically, the event-linked server 308 issues instructions to the client terminal 10 to execute the effects.

This allows a user to enjoy the experience of watching, for example, a match taking place in real space (an example of a real-space event) while getting excited in virtual space with other users who are watching the same match. Users may watch the real-space match, for example, on television, the Internet, or radio, or they may watch it in virtual space. Details of event linking will be described later with reference to Figures 20 to 26.

The trading management server 309 realizes the buying and selling of items, merchandise, tickets, etc. in the virtual space, and also stores data related to the buying and selling. For example, virtual currency may be used for buying and selling in the virtual space. The trading management server 309 may be a server that provides a billing system.

The data analysis server 310 analyzes various data, such as the behavior of each user avatar in the virtual space and the user characteristics of user avatars performing specific actions. The analysis results may be presented to businesses providing the virtual space.

<2-3. Example of configuration of client terminal 10>
Fig. 6 is a block diagram showing an example of the configuration of the client terminal 10 according to the present embodiment. As shown in Fig. 6, the client terminal 10 has a communication unit 110, a control unit 120, an operation input unit 130, a sensor 140, a display unit 150, an audio output unit 160, and a storage unit 170.

(Communication unit 110)
The communication unit 110 is connected to the virtual space management server 200 via wired or wireless communication to transmit and receive data. The communication unit 110 can communicate using, for example, a wired/wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), infrared communication, or a mobile communication network (4G (fourth generation mobile communication method), 5G (fifth generation mobile communication method)), etc.

(Control unit 120)
The control unit 120 functions as an arithmetic processing unit and a control unit, and controls the overall operation of the client terminal 10 in accordance with various programs. The control unit 120 is realized by electronic circuits such as a CPU and a microprocessor. The control unit 120 may also include a ROM for storing the programs to be used, arithmetic parameters, etc., and a RAM for temporarily storing parameters that change as appropriate.

The control unit 120 also functions as a virtual space processing unit 121 and a display control unit 122.

The virtual space processing unit 121 performs various processes to provide the user with a virtual space experience in cooperation with the virtual space management server 200 and the back-end system 30 as appropriate. Such processes may be performed by a client application downloaded in advance to the client terminal 10. The virtual space processing unit 121 performs, for example, a user registration process and a login process for using the virtual space. In the user registration process and the login process, data is sent and received as appropriate to, for example, the user information management server 301 via the communication unit 110. The virtual space processing unit 121 generates a virtual space based on the virtual space information received from the virtual space management server 200. Specifically, the virtual space processing unit 121 arranges virtual objects based on the map information received from the virtual space management server 200 and reproduces the virtual space.

The virtual space processing unit 121 may also incorporate various information acquired from the back-end system 30 into the virtual space. For example, the virtual space processing unit 121 may display video distributed from the content distribution server 306 on a large display placed in the virtual space.

The virtual space processing unit 121 also controls the position, facial expression, gestures (expression of emotion using hands or the entire body), and motion state (sitting, standing, jumping, running, etc.) of the user avatar placed in the virtual space in response to user operations. The virtual space processing unit 121 continuously transmits user operation information to the virtual space management server 200 in real time. The virtual space processing unit 121 reflects information on other user avatars continuously transmitted from the virtual space management server 200 in real time on other user avatars placed in the virtual space. The virtual space processing unit 121 may realize voice conversation between users by transmitting the user's voice acquired by the sensor 140 to the virtual space management server 200 and outputting the voice of other users received from the virtual space management server 200 from the voice output unit 160.

The virtual space processing unit 121 also generates an image (display screen) from the user's viewpoint in the virtual space. The virtual space processing unit 121 may also generate a display screen in which operation buttons for a user avatar, a setting screen, a notification screen, a menu screen, etc. are superimposed on the image from the user's viewpoint. The virtual space processing unit 121 may also display various information obtained from the backend system 30 on the display screen. For example, the virtual space processing unit 121 may display a screen of a text chat between users conducted via the text chat server 304 based on information received from the text chat server 304. The text chat screen may be updated in real time.

The virtual space processing unit 121 may also transmit the user's voice acquired by the sensor 140 to the voice chat server 305, and output the voice of other users received from the voice chat server 305 from the audio output unit 160, thereby realizing voice conversation between users via the voice chat server 305.

In this embodiment, a case where a virtual space is reproduced in the virtual space processing unit 121 will be described, but the present disclosure is not limited to this. For example, an image from a user's viewpoint in a virtual space may be generated and transmitted (streamed) by the virtual space management server 200, and the virtual space processing unit 121 may provide the user with a virtual space experience by displaying the image received from the virtual space management server 200 on the display unit 150.

The display control unit 122 controls the display of an image on the display unit 150. For example, the display control unit 122 controls the display of a display screen generated by the virtual space processing unit 121 on the display unit 150.

(Operation Input Unit 130)
The operation input unit 130 accepts an operation instruction from a user and outputs the operation content to the control unit 120. The operation input unit 130 may be, for example, a touch sensor, a pressure sensor, or a proximity sensor. The operation input unit 130 may also be a physical configuration such as a button, a switch, or a lever. The user may use the operation input unit 130 to operate a user avatar in a virtual space.

(Sensor 140)
The sensor 140 has a function of detecting (obtaining) various information on the user or the user's surroundings. The sensor 140 shown in FIG. 6 may include a number of sensors. For example, the sensor 140 may be a microphone that picks up sound. The sensor 140 may also be a camera that captures an image of the user or the user's surroundings. The sensor 140 may also be a position information measuring unit that measures the user's position (absolute position or relative position). The sensor 140 may also be various sensors (camera, acceleration sensor, angular velocity sensor, geomagnetic sensor, infrared sensor, depth sensor, and biosensor) that detect the user's facial expression, emotion, gaze direction, posture, limb movement, head direction, bioinformation, etc. The sensor 140 may also have a sensor that can detect a total of nine axes, including a three-axis gyro sensor, a three-axis acceleration sensor, and a three-axis geomagnetic sensor.

The information detected by the sensor 140 may be used as information on user operation in the virtual space processing unit 121. For example, the virtual space processing unit 121 may control a user avatar in the virtual space in response to the information detected by the sensor 140.

(Display unit 150)
The display unit 150 has a function of displaying an image under the control of the display control unit 122. For example, the display unit 150 may be a display panel such as a liquid crystal display (LCD) or an organic electroluminescence (EL) display.

The display unit 150 may also be realized by an HMD that covers the user's entire field of vision. In this case, the display unit 150 displays an image for the left eye and an image for the right eye on left and right screens fixed to the user's left and right eyes, respectively. The screen of the display unit 150 is configured, for example, with a display panel such as a liquid crystal display (LCD) or an organic electroluminescence display, or a laser scanning display such as a retinal direct imaging display. The display unit 150 may also include an imaging optical system that enlarges and projects the display screen to form an enlarged virtual image with a predetermined angle of view on the user's pupil.

(Audio output unit 160)
The audio output unit 160 outputs audio information under the control of the control unit 120. For example, when the client terminal 10 is an HMD, the audio output unit 160 may be configured as headphones worn on the user's head, or may be realized by earphones or a bone conduction speaker.

(Storage unit 170)
The storage unit 170 is realized by a ROM that stores programs and calculation parameters used in the processing of the control unit 120, and a RAM that temporarily stores parameters that change as appropriate. The storage unit 170 stores, for example, a client application that executes various processes related to the virtual space. The storage unit 170 may also store user information such as a user profile and user avatar information, and virtual space information received from the virtual space management server 200.

The configuration of the client terminal 10 has been specifically described above, but the configuration of the client terminal 10 according to the present disclosure is not limited to the example shown in FIG. 6. For example, the client terminal 10 may be realized by multiple devices. Each component included in the client terminal 10 is not limited to being integrally provided in a single housing, and each component may be connected for communication via wires or wirelessly.

The client terminal 10 may also be realized by a system configuration including an output device (corresponding at least to the display unit 150 and the audio output unit 160) realized by an HMD or the like, and a processing device (corresponding at least to the control unit 120) realized by a smartphone, tablet terminal, PC, or the like.

The client terminal 10 may also be a non-wearable device such as a smartphone, tablet terminal, or PC.

The client terminal 10 may also use information received from an external device, such as a controller held by the user, a sensor worn by the user, or a sensor placed in the user's vicinity (environment), as information regarding the user's operation.

The client terminal 10 may also communicate with an external display device such as a projector, TV device, or display, and display an image of the virtual space using the display control unit 122.

<<3. Connecting to Virtual Space>>
Next, a connection to the virtual space by the client terminal 10 will be described with reference to Fig. 7. Here, it is assumed that the user has already completed user registration.

Figure 7 is a sequence diagram showing an example of the flow of a process for connecting to a virtual space according to this embodiment. As shown in Figure 7, first, the client terminal 10 starts up a client application (step S103). The operation process of the client terminal 10 described below is executed by the client application. Note that the function of the client application includes the function of the virtual space processing unit 121 described above.

Next, the client terminal 10 requests user authentication from the user information management server 301 (step S106). Specifically, the client terminal 10 transmits the user ID, authentication information, etc. to the user information management server 301.

Next, the user information management server 301 performs user authentication in response to a request from the client terminal 10 (step S109). If the user authentication is successful, the operation proceeds to the next process. If the user authentication is unsuccessful, error processing is executed.

Next, the user information management server 301 transmits the user information stored in the data store to the client terminal 10 (step S112). The user information includes the user's profile, user avatar information (appearance information, status information), information on items owned by the user, etc.

Next, the client terminal 10 sets the received user information in the client application (step S115). This allows the user to obtain the user environment regardless of which client terminal 10 the user uses. For example, whether the user accesses from a smartphone or a home PC, the latest user environment can be reflected in the terminal being used.

Next, the client terminal 10 queries the matching server 303 about the destination server (step S118).

Next, the matching server 303 performs a matching process between the client terminal 10 and the destination server (virtual space management server 200) (step S121). Specifically, the matching server 303 determines the virtual space management server 200 with available space as the destination server according to the current number of connected terminals of each virtual space management server 200. Also, the matching server 303 may determine the virtual space management server 200 previously connected by the client terminal 10 corresponding to the user ID as the destination server according to the user ID.

Next, the matching server 303 transmits data indicating the destination server (e.g., an IP (Internet Protocol) address) to the client terminal 10 (step S124).

Next, the client terminal 10 makes a connection request to the matched virtual space management server 200 (step S127), establishes communication with the virtual space management server 200, and starts two-way communication (step S130).

By establishing communication with the virtual space management server 200, the client terminal 10 can receive virtual space information (map information, virtual objects, information on other users' avatars, etc.) from the virtual space management server 200 and reproduce the virtual space locally. The reproduction of the virtual space can be performed by arranging virtual objects based on the map information. In addition, the client terminal 10 transmits user information, information on user operations, etc. to the virtual space management server 200. Establishing communication with the virtual space management server 200 may be considered as logging in to the virtual space. The client terminal 10 stores data of the connected server, and can log in to the same virtual space again, for example, even if the client terminal 10 temporarily logs off from the virtual space.

<<4. Display screen example>>
Next, a display screen displayed on the client terminal 10 that has logged in to the virtual space will be described. When the user logs in to the virtual space, the user avatar is placed at a predetermined position in the virtual space (for example, the start area 410 shown in FIG. 10). The virtual space processing unit 121 of the client terminal 10 generates an image including the user avatar in its angle of view as an image from the user's viewpoint, and displays the image on the display unit 150.

The virtual space processing unit 121 may also include on the display screen a setting screen for the user avatar, various operation buttons for operating the user avatar, a button for displaying text chats taking place in the vicinity, a button for displaying following and followers, a button for displaying notifications to the user, a map display button, and a button for displaying owned items.

Figure 8 is a diagram showing an example of a display screen according to this embodiment. As shown in Figure 8, a display screen 510 displays a virtual space image 511 and an operation screen 512. An image of a user avatar, buttons for editing the user avatar and user profile, an area-specific menu display button 513, and an area-common menu button 514 are displayed on the operation screen 512.

The area-specific menu display button 513 is a button for displaying a menu specific to the area (virtual space) in which the user avatar is currently located. In this embodiment, an area specialized in content is assumed, and the area-specific menu is assumed to be an experience menu related to the corresponding content. The content here may be IP (Interllectual Property) content. In this embodiment, an area of a specific soccer team is assumed as an example of such an area. Here, fans of a specific soccer team interact with each other. In each place of the area, the logo of the soccer team, a goods store, a virtual stadium that reproduces a stadium existing in the real world, and the like are placed. The user can also move to an area specialized in other content. In this case, the client terminal 10 may newly obtain data of a connection destination server (providing an area (virtual space) specialized in other IP content) from the matching server 303. In addition, a single virtual space management server 200 may provide areas of multiple different IP content. In this case, the client terminal 10 can move to an area without switching the connection destination server.

The area-wide menu button 514 is a menu button that can be used in any IP content area. For example, the avatar matching button is for matching with other user avatars in the area. Other user avatars with similar user characteristics can be presented as recommended interaction partners. Also displayed are setting buttons for emotes, which are actions that the avatar can take to enhance the avatar's emotional expression, and stamps, which are small images that are temporarily displayed above the avatar's head to express the avatar's emotions.

Figure 9 is a diagram showing an example of a display screen according to this embodiment. As shown in Figure 9, a display screen 520 displays an image of a virtual space and various operation buttons. In the example shown in Figure 9, a user avatar U1 is in front of a stage on which a large display 529 is placed. The large display 529 displays, for example, a live video of the real space. The content distribution server 306 obtains this live video from an external server (live video distribution server) and distributes it to each client terminal 10 in real time.

As an example of an operation button, a controller 521 that controls the movement of a user avatar within the virtual space (area) is displayed on the left edge of the display screen 520, and a jump icon 522 is displayed on the right edge of the display screen 520.

When the user slides the cursor displayed in the center of the controller 521 up, down, left or right, the virtual space processing unit 121 controls the user avatar to walk forward, backward, left or right depending on the sliding direction. If the sliding amount or sliding speed is equal to or greater than a set value, the virtual space processing unit 121 controls the user avatar to run forward, backward, left or right. When the jump icon 522 is tapped, the user avatar jumps. Information on user operations related to such movements of the user avatar, such as movement and jumping, is transmitted in real time to the client terminal 10 of the other user via the virtual space management server 200 and is reflected in the user avatar placed in the virtual space reproduced on the client terminal 10 of the other user. In this way, the movements of the user avatar are shared with other users.

The user can also move the user's line of sight in the virtual space by dragging any location on the display screen 520 (any location other than the operation buttons) up, down, left or right with one finger or the like. The user can also pinch in or out with two fingers on any location on the display screen 520 (any location other than the operation buttons) to reduce or enlarge the image in the virtual space.

The microphone icon 523 is a button that switches the user's voice input ON and OFF by tapping. When the microphone icon 523 is in the input ON state, the user's voice is transmitted to other users via the virtual space management server 200 or the voice chat server 305. Voice chat may be conducted between other user avatars located in the vicinity of the user avatar, between specific other user avatars authorized by the user, or between participating users in a voice chat group.

The text input icon 524 is a button that accepts input of text chat. When the text input icon 524 is tapped, a text input UI (user interface) is displayed, and the user inputs text into the text input UI. The input text is displayed as a speech bubble image T1 above the head of the user avatar U1, for example, as shown in FIG. 9. A speech bubble image T2 is also displayed above the head of the other user avatar U2 that is near the user avatar U1. As shown in FIG. 9, 3D expression may be used to display the speech bubble images of the other user avatars U2 and U3 on both sides of the user avatar U1. In addition, the virtual space processing unit 121 may make the speech bubble images of the other user avatars U4 and U5 that are located away from the user avatar U1 blank so that they are not visible to the user. Note that text chat is not limited to a method via a speech bubble image displayed above the head of the user avatar. The virtual space processing unit 121 may display a peripheral chat window in response to a tap operation of the peripheral chat window display switching button 527. The surrounding chat window is a screen that displays text chat between user avatars that are in the vicinity of a user avatar (for example, the area in which the user avatar is located). This allows a user to conduct text chat with other users in the vicinity. Note that text chat can be conducted via the virtual space management server 200 or the text chat server 304.

The stamp icon 525 is a button that displays a stamp menu screen. The user can select any stamp from the stamp menu screen that is displayed by tapping the stamp icon 525, and have it displayed above the head of the user avatar U1. The emote icon 526 is a button that displays an emote menu screen. The user can select any emote from the emote menu screen that is displayed by tapping the emote icon 526, and can manipulate the facial expression, pose, gestures, etc. of the user avatar U1.

The various operation buttons have been described above with reference to FIG. 9, but the arrangement and functions of the various operation buttons are merely examples, and the present disclosure is not limited thereto.

<<5. Specific examples of virtual space>>
<5-1. IP Content Area>
An example of the virtual space according to the present embodiment is an area specialized in IP content. For example, the area may be based on the concept of interaction between fans of a particular soccer team. In this case, for example, the logo of the soccer team, a goods store, and a virtual stadium that reproduces a stadium existing in the real space are arranged in each place of the area. FIG. 10 is a diagram showing an example of a simple map of the IP content specialized area according to the present embodiment. As shown in FIG. 10, the area 400 includes a start area 410 where a user avatar is initially placed when entering the area, a goods store 420, a stadium 430, and a bar 440. The area 400 according to the present embodiment may be a mirror world in which each virtual facility (such as the goods store 420 and the stadium 430) is placed, similar to the real space. The user can enjoy the atmosphere around the actual stadium by moving within the area 400 with the user avatar.

In addition, within the stadium 430, a soccer team match is played as an example of content distribution in a virtual space. Footage of the match may be displayed on a large display placed within the stadium 430 (streaming of 2D video), or the match may be reproduced by reproducing soccer players and the ball in 3D CG and moving each soccer player and the ball based on motion data from an actual match.

Figure 11 is a diagram explaining the mechanism for recreating a match using bone data of players according to this embodiment. As shown in Figure 11, first, the virtual space management server 200a transmits designation information for the content to be distributed to each client terminal 10. The designation information for the content to be distributed is information indicating the match content to be reproduced in the stadium 430.

Each client terminal 10 requests the content distribution server 306 of the back-end system 30 to distribute match content based on the specification information received from the virtual space management server 200a.

Then, the content distribution server 306 requests the distribution of match content from the match data server 50, which is an external device. The match data server 50 is a device that stores tracking data of each player and the ball collected by the tracking system 51 during a match held in real space. The tracking data of each player is, for example, bone data. The match data server 50 may also store 3DCG of each player. The match data server 50 generates match content data after formatting the tracking data acquired from the tracking system 51 and deleting unnecessary data as appropriate. The match data server 50 transmits the match content data to the content distribution server 306 in response to a request from the content distribution server 306.

Next, the content distribution server 306 transmits the match content data to each client terminal 10. The match content data includes 3DCG (also called player object) of each player, bone data of each player in the match, 3DCG of the ball, tracking data of the ball in the match, etc. The client terminal 10 can move the player object in the stadium 430 based on the bone data based on the match content data, and reproduce the match in 3D more realistically. The user can enter the stadium 430 by operating the user avatar and enjoy the match in the stadium 430. The user can also move the user avatar in the field (here, in the soccer court) and watch the movements of each player from various angles up close. The content distribution server 306 can also synchronize the timing of the match content data reproduced in each client terminal 10 (the playback time of the match reproduced in the stadium 430). The UI when reproducing the match in the virtual space using the tracking data of the match played in the real space will be described later with reference to Figures 27 to 32.

<5-2. Talking Table>
At any location in the virtual space, a communication point is set to correspond to a communication room where users can communicate with each other. The communication point may be set by the virtual space management server 200 or by the user.

In a communication room, communication can be carried out using one or more different communication means. The one or more different communication means are, for example, communication means using information such as voice, text, or images. The communication room may be provided by the back-end system 30 (for example, the talking table server 307). A user can enter a communication room (participate in communication) by moving a user avatar to a communication point where a talking table, which is a virtual object, is displayed. When entering a communication room, the display screen transitions to the communication room screen. Furthermore, communication in a communication room can be carried out by a corresponding communication server (for example, the text chat server 304, the voice chat server 305).

A virtual object serving as a landmark may be placed at the communication point. In this embodiment, a table is used as an example of a virtual object serving as a landmark. In this specification, such a table is also referred to as a talking table.

The talking table is placed at any location (where a communication point is set) in the virtual space by an input operation to the client terminal 10 operated by the user or by the virtual space management server 200. As an example, the talking table may be placed in the bar 440 described above with reference to FIG. 10. FIG. 12 is a diagram showing a display screen inside the bar in which the talking table 600 according to this embodiment is placed. The display screen 530 shown in FIG. 12 is an example of an image of the user's line of sight when the user avatar moves inside the bar 440. As described above with reference to FIG. 10, the bar 440 is provided as a place for users (here, fans of soccer teams) to interact with each other. As shown in the display screen 530 of FIG. 12, multiple talking tables 600 are provided inside the bar 440. Each talking table 600 is associated with a talking room. The talking rooms are provided by the talking table server 307.

The shape of the talking table 600 is an example. When a user brings a user avatar close to the talking table 600, information regarding the selection of entry into the talking room associated with the talking table 600 is transmitted from the virtual space management server 200 to the client terminal 10. The user can cause the user avatar to enter the talking room (communication room) (in other words, participate in the talking table) by performing an entry request operation on the client terminal 10.

The talking table 600 is displayed at a communication point that corresponds to a communication room where users can communicate with each other. The communication point is set in a specific area in the virtual space, and when an avatar is included in the communication point, information regarding the selection of entering the room is transmitted.

13 is a diagram for explaining an example of the shape of the talking table 600. As shown in FIG. 13, for example, the talking table 600 includes a table 601 and a display 602. The display 602 is an example of a display object that displays "information related to an event taking place in a real space or a virtual space" that is a conversation target in a talking room (communication room) associated with the talking table 600. The conversation target in the talking room may be preset in the virtual space management server 200, or may be the current agenda in the talking room or a video currently being viewed in the talking room (for example, the topic window 542 shown in FIG. 18). The display content of the display 602 is information related to the conversation target in the talking room, and may change appropriately according to changes in the conversation target in the talking room. The display 602 and the table 601 are placed in the virtual space as virtual objects.

For example, when "the most recent match held in the real world" is set as the conversation subject in the talking room associated with the talking table 600, the video of the match, the interview video of the players who participated in the match, or information related to the match such as the match result is displayed on the display 602. The 2D video displayed on the display 602 may be a video streamed from the content distribution server 306. The 2D video displayed on the display 602 may be specified by the virtual space management server 200. The virtual space management server 200 may specify the 2D video to be displayed on the display 602 according to the IP content associated with the area. The same 2D video may be played on multiple talking tables 600 arranged inside the bar 440.

The talking table 600 may further include a display 603 that displays information about the talking table 600. For example, the display 603 displays the identification number of the talking table 600 and the number of people currently in the talking room. By visually checking the display 603, the user can check how many people can enter the talking room and how many people are currently in the room.

(How to enter)
By approaching the talking table 600, a user can select whether to participate (enter) as a speaker or as an audience member. "Participating as a speaker" means participating in a state in which the user can make comments in the talking room (e.g., voice chat, text chat). "Participating as an audience" means participating in a state in which the user cannot make comments in the talking room but can view the talking room.

When the user avatar approaches the talking table 600, the virtual space management server 200 transmits information regarding entry selection to the client terminal 10 so that the user can select whether or not to enter the talking room associated with the talking table 600. The information regarding entry selection is information indicating that the user can enter (participate) as either a speaker or an audience member. When the user operates the client terminal 10 to request entry (participation), the user can select whether to participate as a speaker or an audience member.

When the user avatar moves within a certain distance from the position of the talking table 600, for example, based on the information regarding the entry selection received from the virtual space management server 200, the virtual space processing unit 121 displays a selection screen that allows the user to select whether to enter the talking room as a speaker or as an audience member. Whether or not to participate as an audience member may be set when creating the talking room. When entry as a speaker or an audience member is selected, an entry request is sent from the client terminal 10 to the virtual space management server 200, and each session information (connection information) corresponding to the talking table 600 is returned from the virtual space management server 200. Based on each section information received, the client terminal 10 can communicate with the talking table server 307, the text chat server 304, the voice chat server 305, and the like.

The information regarding the entry selection may include in advance each session information (connection information) corresponding to the talking table 600. This allows the client terminal 10 to issue an entry request to the virtual space management server 200 in response to the user's entry selection operation, while also establishing a communication connection with the talking table server 307, the text chat server 304, the voice chat server 305, etc., based on each section information.

Also, when there are multiple talking tables 600, the user may not know which talking table 600 to join in order to have the desired communication. In this case, the following method, for example, may be considered.

One example is a method of varying the level at which the state of the inside of the talking room is disclosed depending on the distance between the talking table 600 and the user avatar. Figure 14 is a diagram illustrating a case in which the state of the inside of the talking room is disclosed depending on the distance from the talking table 600 according to this embodiment.

As shown in FIG. 14, when the virtual space management server 200 determines that the user avatar is located in area E1 (second area) within a first distance from the position of the talking table 600 (within a first distance centered on the communication point), the virtual space management server 200 transmits information about the communication of the talking table 600 to the client terminal 10. An example of information about the communication of the talking table 600 is audio information in the communication room associated with the talking table 600. This allows the user to understand the amount of conversation and the atmosphere in the talking room without entering the talking room. In the example shown in FIG. 14, the users of user avatars U4 and U5 can hear the conversation in the talking room. In addition, the virtual space management server 200 may instruct the client terminal 10 to increase the volume so that the conversation in the talking room can be heard as the user avatar approaches the center (talking table 600).

On the other hand, as shown in FIG. 14, when the virtual space management server 200 determines that the user avatar is located in area E2 (first area) that is within a second distance from the position of the talking table 600 that is shorter than the first distance, the virtual space management server 200 transmits information regarding the selection of entry into the talking room associated with the talking table 600 to the client terminal. The user may operate the client terminal 10 to request entry. At this time, the user may select whether to participate (enter) the talking room as a speaker or an audience member (this can also be considered a user input regarding the information regarding the selection of entry).

When a user enters a talking room, the virtual space management server 200 may transmit the conversation in the talking room (e.g., voice chat information, text chat information), video in the talking room (e.g., the appearance of the user avatar shown in FIG. 18), video shared (viewed) in the talking room, or the agenda in the talking room to the client terminal 10. The agenda may be extracted from the analysis results of the voice chat or text chat in the talking room. Such an agenda extraction may be performed by the back-end system 30. In addition, the category of the topic window 542 (see FIG. 18) picked up in the talking room described later may be extracted as the agenda. In addition, the conversation in the talking room, video in the talking room (e.g., the appearance of the user avatar shown in FIG. 18), video shared (viewed) in the talking room, or the agenda in the talking room may be transmitted to the client terminal 10 corresponding to the user avatar located in the area E1.

In the example shown in FIG. 14, it is assumed that among the avatars located in area E2, user avatar U1 and user avatar U2 have entered the talking room as speakers, and user avatar U3 has entered the talking room as an audience member. At this time, user avatar U4 and user avatar U5 located in area E1 can visually confirm the contents displayed on display 602 and understand the topic of the conversation taking place in the talking room. In addition, information related to communication in the talking room (e.g., audio information in the talking room) may be transmitted from the virtual space management server 200 to the client terminal 10 of each user who operates user avatar U4 and user avatar U5. Then, a selection screen for whether or not to enter the talking room is presented on the client terminal 10 of the user who operates the user avatar that has newly entered area E2 or the user avatar that has approached the talking table 600.

As described above, in this system, the virtual space management server 200 places in the virtual space a virtual object (talking table 600) including a display object (display 602) that displays "information related to an event taking place in real space or virtual space," which is the subject of conversation in the talking room (communication room). This allows a user operating a user avatar present in the vicinity of the virtual object to visually check the display content of the display object and understand the topic of the conversation taking place in the talking room.

In addition, when the position in the virtual space of the user avatar operated by the user is included in area E2 (first area), which is a communication point associated with a communication room where communication between users takes place, the virtual space management server 200 transmits information regarding the selection to enter the communication room to the terminal used by the user (client terminal 10).

In addition, when the position of the user avatar is included in area E1 (second area) located outside area E2 (first area), the virtual space management server 200 transmits information regarding communication to the terminal (client terminal 10) used by the user who operates the user avatar.

As a result, while operating an avatar in the virtual space, the user can visually check the display object (display 602) to understand the topic in the talking room, check the audio in the talking room depending on the distance between the user avatar operated by the user and the virtual object (talking table 600), and join the talking room by bringing the user avatar closer to the virtual object than a certain amount, and can determine their relationship with the talking room by operating the avatar.

As another example, when the user avatar approaches within a certain distance of the talking table 600, the virtual space processing unit 121 may change the disclosure level of the situation in the talking room according to the user's hobby and preference information. For example, if the conversation (topic) in the talking room matches the user's hobby and preference, the virtual space processing unit 121 may play the conversation in the talking room at a loud volume or display a more detailed topic in the tab display 605. On the other hand, if the conversation (topic) in the talking room does not match the user's hobby and preference, the virtual space processing unit 121 may play the conversation in the talking room at a low volume or not play it, or not display the topic in the tab display 605.

As another example, the virtual space processing unit 121 may highlight the talking table 600 of a topic that matches the user's hobbies and preferences. The virtual space processing unit 121 may also highlight the talking table 600 of a topic that the user has mentioned in conversations in the past.

As another example, the virtual space processing unit 121 may highlight talking tables 600 in which other users who are presumed to have similar hobbies and tastes to the user are participating.

As another example, the virtual space processing unit 121 may present an effect below the table depending on the level of excitement in the talking room. FIG. 15 is a diagram showing an example of an effect display on the talking table 600 depending on the level of excitement in the talking room according to this embodiment. The virtual space processing unit 121 may present a light effect below the table 601 included in the talking table 600, as shown in FIG. 15. The virtual space processing unit 121 may control the color, blinking, intensity, etc. of the effect depending on the level of excitement in the talking room. This allows the user to grasp the level of excitement in the talking room.

The virtual space processing unit 121 may also change the shape of the talking table 600 depending on the number of participants in the talking table 600 (the number of people entering the talking room). For example, the table may be changed to a larger size as the number of participants increases.

In addition, this system does not impose any restrictions on entry to the talking room, allowing anyone to enter freely. However, a function may be provided that allows participants in the talking room to voluntarily expel other participants.

In addition, in this system, the host of a talking room does not need to be specifically set, or the first user to enter the room may be the host. If a talking room is created at the request of a user, the creator may be the host. The host may set entry restrictions to the talking room, or entry to the talking room may require permission from the host. The host may be set by the talking table server 307 that manages the talking room. Furthermore, only the host may be given the authority to make the other participants exit. Furthermore, the host's authority may be freely transferred to other participants.

The virtual space management server 200 may also display on the display 602 a specific scene of a match taking place in the real space or virtual space as information related to an event in the real space or virtual space. The information related to an event taking place in the real space is not limited to a specific scene of a match, but may be any specific scene of an event, such as a specific scene of a live music concert held in the real space. In the case of an event taking place in a virtual space, the information may be a specific scene of an event, such as a live music concert or an e-sports tournament, held in the virtual space.

The specific scene of the event may be, for example, a highlight video of a match held in real space. The virtual space management server 200 may control the display 602 (display object) to display highlight video of a match held in real space as information related to the event in real space. Examples of highlight video of a match include goal scenes and foul scenes. The specific scene of the event may also be a highlight video of an event held in virtual space.

The virtual space management server 200 may display on the display 602 (display object) "information related to a real-space event" only information related to a team that a specific user group supports at the event. For example, highlight footage of a match featuring a specific team. This allows the user to know before entering the talking room associated with the talking table 600 including the display 602 which communication will be held about which team. In this system, for example, by displaying only information related to a specific team on the display 602, it is possible to attract fans of a specific team to the talking room.

The virtual space management server 200 may also transmit information regarding the selection to enter the talking room only to users who belong to a specific user group in the event, among the users who operate user avatars approaching the talking table 600. This allows the system to place restrictions on the user groups that can enter the talking room.

(Operation Processing)
FIG. 16 is a sequence diagram showing an example of an operational process from generation of the talking table 600 to entry according to this embodiment.

As shown in FIG. 16, first, the talking table server 307 generates a talking room (step S206) in response to a talking room generation request from the virtual space management server 200 (step S203). Then, the talking table server 307 transmits session information for the generated talking room to the virtual space management server 200 (step S209). The virtual space management server 200 associates the session information with a communication point set in the virtual space. Note that, as an example, a generation request from the virtual space management server 200 is described here, but the present disclosure is not limited to this, and a generation request may be made from the client terminal 10. In this case, the talking table server 307 transmits session information for the generated talking room to the client terminal 10. The client terminal 10 associates the session information with a communication point set in the virtual space. In addition, the client terminal 10 transmits the communication point set in the virtual space and the session information to the virtual space management server 200 to share with other users.

Next, the virtual space management server 200 sends a request to create a text chat room to the text chat server 304 (step S212). The text chat server 304 creates a text chat room (step S215) and sends session information for the text chat room to the virtual space management server 200 (step S218).

Next, the virtual space management server 200 sends a request to create a voice chat room to the voice chat server 305 (step S221). The voice chat server 305 creates a voice chat room (step S224) and sends session information for the voice chat room to the virtual space management server 200 (step S227).

Next, the virtual space management server 200 stores each session information (step S230).

Next, when the user avatar moves near the talking table 600 (within a certain distance from the talking table 600) and entry into the talking room (and the mode of entry) is selected by user operation (when the user inputs information related to the entry selection), the client terminal 10 transmits an entry request to the virtual space management server 200 (step S233). Note that when the user avatar moves near the talking table 600, information related to the selection of entry into the communication room may be transmitted from the virtual space management server 200 to the client terminal 10. The client terminal 10 may display an entry selection screen based on the information related to the entry selection.

Next, the virtual space management server 200 recognizes the user's intention to enter the room by receiving an entry request from the client terminal 10 (step S236), and transmits each session information (connection information) corresponding to the talking table 600 to the client terminal 10 (step S239). In addition, the virtual space management server 200 may control the client terminal 10 to transition to a talking room screen (an example of a communication room screen) when the user enters the talking room.

Note that, as an example, a case where a user selects to enter the room is described here, but the present disclosure is not limited thereto. As described above, when controlling the disclosure of the state inside the talking room according to the distance between the user avatar and the talking table 600, the client terminal 10 may request the virtual space management server 200 to transmit each session information before obtaining the user's intention to enter the room. In addition, since the client terminal 10 continuously transmits the user's position (the position of the user avatar) to the virtual space management server 200, when the user avatar is located within a certain distance from the talking table 600 (communication point), the virtual space management server 200 may recognize that there is an intention to enter the room and transmit each session information to the client terminal 10. Alternatively, the virtual space management server 200 may include each session information corresponding to the talking table 600 in the information regarding the above-mentioned entry selection transmitted to the client terminal 10. In the client terminal 10, after receiving each session information from the virtual space management server 200, the user may select whether or not to enter the room. This allows the client terminal 10 to control the disclosure of the state of the talking room depending on factors such as the distance between the talking table 600 and the user avatar before obtaining the user's intention to enter the talking room.

Then, the client terminal 10 displays the talking room screen and starts a communication connection with each server based on each session information (step S242). Details of the communication connection with each server are described below with reference to FIG. 17.

Figure 17 is a diagram explaining the details of the communication connection between the client terminal 10 and each server when entering a talking room in this embodiment. As shown in Figure 17, each client terminal 10 that has entered a talking room performs two-way communication with the talking table server 307 to obtain entering user information. The entering user information includes the entering user's user ID, user name, user avatar information, etc.

Each client terminal 10 communicates bidirectionally with the text chat server 304 to send and receive text chat. This allows text chat to take place within the talking room.

Each client terminal 10 communicates bidirectionally with the voice chat server 305 to send and receive voice chats. This allows voice chats to take place within the talking room.

Each client terminal 10 communicates bidirectionally with the virtual space management server 200a, sending and receiving emotes, stamps, etc. in real time. This allows the emotes and stamps of each user avatar displayed in the talking room to be controlled in real time.

Each client terminal 10 communicates with the content distribution server 306 and receives streaming distribution of content (videos, etc.). This allows the content (videos, etc.) to be viewed within the talking room. Content designation information can be received from the talking table server 307 or the virtual space management server 200a. The content may be set by the talking table server 307 or the virtual space management server 200a, or may be set by the user who has entered the room.

(Talking room screen example)
Fig. 18 is a diagram showing an example of a talking room screen according to the present embodiment. When a user joins a talking table (i.e., enters a talking room), the virtual space processing unit 121 of the client terminal 10 transitions the display screen shown in Fig. 14 to a talking room screen 540 shown in Fig. 18.

On the talking room screen 540, the avatars of users who have joined the room are displayed side-by-side. In addition, various operation buttons (stamp icon 546, emote icon 547, fixed stamp icon 548, text input button 549, microphone ON/OFF button 550, menu button 541) and a topic window 542 are displayed.

Above the head of each user avatar, a speech bubble image 544 displaying the entered text and a stamp image 545 selected by the user may be displayed. The virtual space processing unit 121 may display the speech bubble image 544 and the stamp image 545 for only a few seconds, without allowing the log to be seen. In addition, the facial expression, pose, gesture, etc. of each user avatar (applause, hurrah, greeting, fist pump, gesture of disappointment, etc.) is controlled according to the emote selected by the user.

Topic window 542 is a virtual object that displays the content being viewed in the talking room. There may be multiple topic windows 542. Each user may slide through multiple topic windows 542a-542c to select the content to pick up. In the talking room, playback of the picked topic window 542 (e.g., the window located in the middle) begins. The right to pick up (channel right) may be given to all users who have entered the room.

The categories of content displayed in the topic window 542 are, for example, news related to the IP content (here, a specific soccer team) (such as the top 5 PV (page views) for the last 3 days), videos related to the IP content that have received a lot of attention (such as videos with the most views or videos that users are excited about), highlights of the most recent match, match history, etc. In addition, it is also possible for a user who has entered the room to display private photos and videos in the topic window 542, or to share the user's local screen. The categories of content displayed in the topic window 542 may be determined by the virtual space management server 200 or the talking table server 307 depending on the location where the talking table 600 is installed and the current time.

FIG. 19 is a diagram showing another example of a talking room screen according to this embodiment. As in the talking room screen 560 of FIG. 19, only the picked topic window 561 may be displayed large, or a text chat window 562 may be displayed. The display of the text chat window 562 may be switched on and off by tapping the display switch button 564. Furthermore, the microphone in the talking room may be switched on and off by tapping the voice switch button 565.

The above describes an example of a talking room screen, but the talking room screen according to this disclosure is not limited to the example shown in the drawings.

<5-3. Event Linkage>
Next, a description will be given of the control of the presentation (including virtual events) in the virtual space in response to the development of an event taking place in the real space, which is performed in cooperation with the event linking server 308. The distributed video of the event taking place in the real space may be placed in the virtual space. In addition, the user may watch the distributed video of the event taking place in the real space outside the virtual space, such as on a television or in a window other than the window in which the virtual space is displayed. In addition, the presentation in the virtual space is determined in response to the development of the event taking place in the real space and the reactions of multiple users.

Figure 20 is a diagram explaining the mechanism of event linkage according to this embodiment. In the example shown in Figure 20, a soccer match taking place in real time in the real space is assumed as the specified event.

First, the event-linked server 308 of the back-end system 30 acquires real-time match data at any time and analyzes the progress of the match. The event-linked server 308 also identifies whether the match progress is before, during, or after the match. For example, the event-linked server 308 may acquire text of the match progress as match data from a service that live-streams the match progress in text, and automatically extract specific words. An example of a service that live-streams the match progress in text is a service that converts the comments (audio) of a commentator during a match broadcast into text and distributes it. Examples of specific words that are automatically extracted include "free kick," "foul," "missed shot," "shot blocked," "corner," "opponent's free kick," "◇◇ (opponent team) goal," "halftime begins," "player substitution," "second half begins," "yellow card," "offside," "○○ (teammate) goal," and "end of the match." The text of the match progress may also be distributed independently on the official website of each team.

Analysis of the game progress is not limited to the above example. For example, the event-linked server 308 may perform image recognition on live broadcast footage of the game in real time to distinguish plays.

Next, the virtual space management server 200a designates a corresponding predetermined performance according to the analysis results of the game development acquired from the event linkage server 308 at any time and the reactions of the multiple users, and instructs each client terminal 10 to immediately perform the corresponding performance in the virtual space. In this embodiment, as an example, a performance according to the development of the game taking place in the real space can be performed in real time in the stadium 430. As a result, in this system, users who have gathered at the stadium 430 existing in the virtual space can experience the performance in the stadium 430 according to the game development and the reactions of the multiple users with other users, thereby making the game viewing in the virtual space more exciting. In addition, the virtual space management server 200a may acquire a distribution video of the game taking place in the real space from the back-end system 30 (for example, the content distribution server 306) and transmit it to each client terminal 10, and instruct them to place it in the virtual space (specifically, in the stadium 430). In addition, each client terminal 10 may acquire a distribution video of the game designated by the virtual space management server 200a from the content distribution server 306. While watching a live broadcast of a match on an external device such as a television device or PC, a user may watch the performance in stadium 430 from the user's viewpoint in the virtual space using a client application running on client terminal 10 such as a smartphone or tablet terminal. Client terminal 10 may display the live broadcast of the match in one window displayed on display unit 150, while displaying an image of the virtual space (the performance in stadium 430) in another window displayed on display unit 150.

The virtual space management server 200a also acquires user reactions from each client terminal 10, estimates the progress of the game based on the content of the reactions, and determines the corresponding presentation. User reactions include information on user operations (such as the number and type of stamps and emotes executed). User reactions may also be information such as the number, content, frequency, and voice volume of voice chat transmissions. User reactions may also be information such as the number, content, and frequency of text chat transmissions. The virtual space management server 200a can also estimate the progress of the game from the number of voice chat or text chat transmissions, etc. The estimation of the progress of the game based on user reactions may be performed by the event linkage server 308.

Examples of performances that correspond to the progress of the game include having a DJ appear on a stage on the field before the game to liven up the venue with music, showing footage of the team practicing or traveling before the game on a large display on the field, and showing live footage of scenes that appear at the game venue. In addition, a performance announcing the starting lineup may be held using a large display on the field.

At the start of the match, there is a kick-off performance, and during the match, there are performances for fouls by players on the team or the opposing team, performances for conceding a goal (such as having a team character avatar appear to encourage the team), performances for scoring (such as playing music, setting off fireworks, making confetti and balloons appear, and showing happy team character avatars), performances for a lottery event at halftime (raining presents down), performances displaying the profiles of notable players (displayed on a large display placed on the field), performances for yellow cards for players on the team or the opposing team, and performances for offsides by players on the team or the opposing team. At the end of the match, there is an end-of-match performance, and after the match, there are performances for victory (such as having a DJ appear to liven up the venue with music) or performances for defeat.

In addition, when the number of user reactions exceeds a certain number, the virtual space management server 200a may execute a corresponding performance according to the progress of the game and the reactions of multiple users. For example, when many users (users exceeding a certain number or a certain percentage) press the "GO" stamp, the virtual space management server 200a executes a performance in which the NPC (Non Player Character) placed on the field also executes a cheering emote or a performance in which cheering music is played as a cheering performance. In addition, the virtual space management server 200a may regard the number of user reactions as excitement, and increase the size and number of balloons to be flown on the field according to the level of excitement. In addition, the virtual space management server 200a may increase the volume of the cheering music to be played on the field according to the level of excitement of the users. In addition, the virtual space management server 200a may extend the length of the cheering performance beyond the default time if the users continue to be excited. Additionally, the virtual space management server 200a may determine the presentation based on the most common reaction among the user reactions.

The virtual space management server 200a may acquire, as the user's reaction, the reaction of a user operating a user avatar that exists in the virtual space managed by the virtual space management server 200a. Such a user's reaction is, for example, information on the user's operation (the number and type of stamps and emotes executed, etc.). The user's reaction may also be information such as the number, content, frequency, and voice volume of voice chat transmissions. The user's reaction may also be information such as the number, content, and frequency of text chat transmissions.

The virtual space management server 200a may further acquire, as user reactions, reactions of users who operate user avatars that exist in other virtual spaces managed by other virtual space management servers that perform presentations according to the progress of the same match as the match that the virtual space management server 200a is targeting. Such user reactions may also be user operation information, voice chat information, or text chat information. Note that, when the virtual space management server 200a is transmitting a broadcast video of the match to the client terminal 10a, the virtual space management server 200a may further acquire reactions of users who operate user avatars that exist in other virtual spaces managed by other virtual space management servers that are transmitting the same broadcast video.

Furthermore, the user's reactions are not limited to the reactions of the user operating the user avatar, but may also be the reactions of spectators watching the game on-site in real space, or the reactions of viewers watching the game on television or the like. Such reactions may be, for example, facial expressions, movements, cheers, etc. Furthermore, such reactions are sensed by various sensors such as cameras, acceleration sensors, or microphones. The virtual space management server 200a may appropriately acquire information on the spectators' reactions or viewers' reactions in real time.

The virtual space management server 200a may also determine emotes for avatars present in the virtual space as a performance determined according to the progress of the match and the reactions of multiple users. For example, when the virtual space management server 200a identifies that a match is currently in progress and further identifies that many users are cheering to boost the momentum of the match, such as by selecting the "GO" stamp, it reflects emotes such as cheering by waving towels on multiple avatars present in the virtual space. At this time, the virtual space management server 200a may reflect emotes on avatars in a non-operated state in the virtual space, may reflect emotes on NPCs, or may reflect emotes on all avatars. The virtual space management server 200a may also reflect emotes on all avatars and all NPCs in a non-operated state, or may reflect emotes on some avatars and some NPCs in a non-operated state.

(Examples of event linkage)
The game progress can be broadly divided into pre-game, game (first half), halftime, game (second half), and post-game. Specific examples of each part of the game progress will be described below.

Before the match 45 minutes before the match, the start of the live stadium (a production event that takes place in the stadium 430 in response to the progress of the match in real time in the real space. In this embodiment, this is also called the "live stadium.") is notified to each user avatar. User avatars in the area move to the stadium 430.

The user enters the stadium 430, meets up with other user avatars (other users who are registered as friends), and waits for the match to start while talking with friends, for example, in a group chat.

A DJ appears on a stage located on the field of stadium 430 and creates a musical performance to liven up the venue. A large display may also be placed above the stage. The large display shows footage of the team's practice before the match, or footage of the team bus arriving. FIG. 21 is a diagram showing an example of a display screen in a live stadium according to this embodiment. As shown in FIG. 21, a display screen 710 shows starting lineup information and footage of the team's state before the match on a large display 711 placed on the field (pitch) in the stadium in the virtual space.

In addition, on the field, the starting lineup may be announced, and avatars of the starting players may appear on stage. This will further excite the users gathered on the field. Next, when the players enter the match venue in real space, a live broadcast of the players entering the match venue is displayed on the large display 711, and the excitement of the users gathered on the field reaches a climax.

During a match During a match, the large display 711 displays the match status, effects according to the reactions of users gathered on the field, or effects that prompt users to react. FIG. 22 is a diagram showing an example of a display screen in a live stadium according to this embodiment. In the display screen 720 shown in FIG. 22, the display windows 721 and 722 of the large display 711 and the text chat window 723 are displayed in the foreground. Such display switching can be performed appropriately by user operation. The user can use the text chat window 723 to text chat with other users in the area. In addition, the user can also perform voice chat with other users who have been registered as friends and with group members that have been generated. These text chats and voice chats can be performed using the backend system 30.

Display window 721 displays images of team character avatars appearing on a stage arranged in the field of the virtual space to liven up the venue. Display window 722 also displays cheer stamps that are the same as those selected by the majority of users in the field, or cheer stamps that encourage users to make a selection. This effect creates a sense of unity among users and livens up the venue. The virtual space processing unit 121 may play music or sound as appropriate in response to effect instructions from the virtual space management server 200. Cheering stamps include, for example, NICE, BOO, GOAL, etc.

Fig. 23 is a diagram showing an example of a display screen in a live stadium according to this embodiment. The display screen 730 shown in Fig. 23 shows an example of a conceded goal presentation. When the opposing team scores a goal, for example, a presentation in which the team characters pose in despair is implemented. Furthermore, when the user taps the "OMG" cheering stamp button 450, an "OMG" cheering stamp is displayed above the head of the user avatar U. When many users have selected the "OMG" cheering stamp, a presentation in which an OMG! cheering stamp is displayed on the large display 731 is implemented.

The user can also have the user avatar U perform the same cheering emote as the team character. FIG. 24 is a diagram showing an example of a display screen in a live stadium according to this embodiment. As shown in FIG. 24, when the SNYC (Synchronization) cheering button 742 shown on the display screen 740 is selected, the virtual space processing unit 121 causes the user avatar U to perform the same cheering emote as the team character 741. This allows users at the venue to perform cheering emotes together with the team characters and other users.

When halftime begins, for example, a lottery event is held. FIG. 25 is a diagram showing an example of a display screen in a live stadium according to this embodiment. As shown on display screen 750 in FIG. 25, for example, a large number of boxes fall from the sky, and if a user operates user avatar U to pick them up, they will be entered into a lottery to win a gift such as an autographed ball or a uniform.

When a goal is scored by a team during a match, a scoring performance is performed. In the scoring performance, for example, music is played, fireworks are set off, confetti and balloons fall, and team characters dance. FIG. 26 is a diagram showing an example of a display screen in a live stadium according to this embodiment. As shown in the display screen 760 of FIG. 26, a performance is performed on the field where fireworks are set off and confetti falls. In addition, a large display 761 displays information about the player who scored, information about the players on the opposing team, scores, cheering stamps, and images of team characters. In addition, a performance may be performed in which a giant avatar 762 of the player who scored appears on the stage. In this way, a performance is performed to liven up the venue when a team scores a point.

NPCs may be placed in the spectator seats around the field and on the field, and the virtual space processing unit 121 may execute emote effects as appropriate. For example, the virtual space processing unit 121 executes a cheering emote on the NPCs when a teammate player is attacking, shooting, or taking a free kick. The virtual space processing unit 121 also executes a joyful emote on the NPCs when a teammate player scores a goal. The virtual space processing unit 121 also executes a booing emote on the NPCs when an opposing player commits a foul. The virtual space processing unit 121 also executes a dejected emote on the NPCs when an opposing player scores a goal.

In addition, in order to create excitement in the virtual space, the virtual space management server 200 may instruct each client terminal 10 to have a user avatar that is not being operated automatically perform an emote or stamp or other effect in accordance with the overall effect.

After the match When the match ends, a final performance is executed. If the team wins the match, a victory performance (such as the DJ reappearing) may be performed to liven up the venue, and if the team loses, a performance praising the players may be performed. Also, a large display on the field may show interviews with the coach and players after the match. Also, a team character may appear on the stage and perform a victory emote or play the team's cheer song.

After that, each user may move to a place where fans can interact with each other, such as the bar 440 described with reference to Figures 10 and 12, and use the talking table 600 to talk about today's game. Video of today's game may be shared within the talking table 600.

The above describes specific effects that correspond to the progress of the game. Note that the effects according to the present disclosure are not limited to the specific examples described above, and various effects are possible. Also, while effects that correspond to the progress of the game have been used as an example here, the present invention is not limited to this, and may be effects that correspond to the progress of, for example, a live music concert, lecture, presentation, etc. taking place in real space.

<5-4. Match Replay UI>
Next, a UI for reproducing a match in a virtual space using tracking data of a match played in a real space will be described. As described above with reference to Fig. 11, the client terminal 10 reproduces a match by arranging each player object and a ball in the stadium 430 (see Fig. 10) in the virtual space based on the match content data received from the content distribution server 306, moving each player object according to the bone data of each player tracked during the match, and moving the virtual ball based on the ball tracking data.

(About camera position)
When displaying the reproduced game on the display unit 150, the client terminal 10 may display an image from an arbitrary camera position among a number of camera positions. The arbitrary camera position may be selected by the user or may be automatically selected by the client terminal 10.

Figure 27 is a diagram illustrating an example of a camera position according to this embodiment. As shown in Figure 27, examples include camera position 810, camera position 811, camera position 812, and camera position 813.

Camera position 810: In Basic, virtual camera C is placed on an extension line connecting the goal and the ball. It follows the ball. Camera position 811: In Bird's eye, virtual camera C is placed in a position above and behind the ball, where a wide angle of view can be obtained. These camera positions and camera orientations may be made controllable by the user.

Each of the camera positions described above is a position from which the player can be viewed from above, but is not limited to this, and the client terminal 10 may set the position (viewpoint) of a specific player as the camera position and provide a subjective image of the player. For example, as shown in the lower part of FIG. 27, at camera position 812: GK, the goalkeeper's viewpoint is set as the camera position, and the view seen from the goalkeeper's perspective is provided. Also, at camera position 813: Shooter, the viewpoint of the player who will shoot at the end of each scene (also called chapter) is set as the camera position, and the view seen from that player is provided.

Figure 28 is a diagram showing the game reproduction UI in the case of the camera position 810: Basic shown in Figure 27. As shown in Figure 28, the game reproduction UI 820 displays an image captured by a virtual camera C placed on an extension line connecting the goal and the ball. The screen overview of the game reproduction UI 820 shown in Figure 28 will be described later.

Figure 29 is a diagram showing the game reproduction UI when camera position 811: Bird's eye shown in Figure 27 is used. As shown in Figure 29, the game reproduction UI 830 displays an image captured by virtual camera C placed in a position that provides a wide angle of view above and behind the ball. Note that the screen overview of the game reproduction UI 830 shown in Figure 29 is the same as that of the game reproduction UI 820 shown in Figure 28.

(Overview of the Match Replay UI Screen) With reference to FIG. 28, a screen overview of the match replay UI 820 will be described. As shown in FIG. 28, the match replay UI 820 displays, for example, an exit icon 821, a notification icon 822, an angle changeable label 823, a camera position switching icon 824, an avatar mode switching icon 825, a highlight label 826, a match progress information display 827, and a display switching icon 828. Note that the arrangement and shape of each display are not limited to the example shown in FIG. 28.

The exit icon 821 is an operation button for ending the match replay mode in which display control of player objects is performed based on the tracking data (specifically, bone data) of each player according to this embodiment. The match replay mode according to this embodiment may be a highlight replay in which a replay is performed using tracking data of one or more specific scenes from the tracking data of a single match. In this specification, this type of match replay mode using highlight replay, which is performed in the stadium 430 in the virtual space, is also referred to as a "highlight stadium."

The notification icon 822 indicates whether or not a notification has been sent to the user. For example, the notification icon 822 displays a display showing the number of notifications, and when the notification icon 822 is tapped, a list of notifications is displayed in a pop-up.

The angle changeable label 823 is an indication of whether the user can arbitrarily manipulate the camera angle. For example, in the case of camera position 810: Basic and camera position 810: Basic shown in FIG. 27, the camera angle (i.e., the line of sight direction) is set to be user-operable. On the other hand, in the case of camera position 812: GK and camera position 813: Shooter shown in FIG. 27, the camera angle is set to be user-inoperable. If the camera angle is user-operable, the user can change the camera angle in the up, down, left, or right direction by dragging or flicking any location on the screen with one finger or the like in any direction.

The camera position switching icon 824 is an operation button for switching between the multiple camera positions described with reference to FIG. 27. For example, the user can tap any icon among the icons included in the camera position switching icon 824, Basic, Bird's eye, GK, and Shooter, to switch the image in the match reproduction UI 820 to an image from the camera position corresponding to the icon tapped.

The avatar mode switching icon 825 is an operation button for changing the viewing mode of the match replay to avatar mode. The viewing modes in this embodiment include, for example, View mode and avatar mode. If the avatar mode switching icon 825 is not selected, the default is displayed in View mode. In View mode, as shown in Figures 28 and 29, the user avatar is not displayed. In avatar mode, the user avatar is displayed within the field, and the user can operate the user avatar to view the match from a free viewpoint. An example of the display in avatar mode will be described later with reference to Figure 32.

The highlight label 826 is a display indicating that the function currently being used by the user in the virtual stadium 430 (see FIG. 10) is highlight stadium (match replay mode for highlight replay). Note that in the virtual stadium 430 of this embodiment, the function of the "live stadium" presentation event, which is carried out in response to the progress of the match in real time in the real space, as described with reference to FIGS. 20 to 26, can also be used. In this case, a label indicating, for example, "LIVE VIEW" is displayed on the screen.

The match progress information display 827 shows the score, opponent, and elapsed time of the match being reproduced.

The display switching icon 828 is an operation button for switching the playback indicator display. Each time the display switching icon 828 is tapped, the playback indicator display is switched between ON and OFF. An example of the playback indicator display will be described with reference to FIG. 30.

Figure 30 is a diagram showing an example of the display of a playback indicator according to this embodiment. In the match replay UI 830 shown in Figure 30, when the display switch icon 828 is tapped, a playback indicator 840 is displayed at the bottom of the screen. Note that the display position of the playback indicator 840 is not particularly limited. The playback indicator 840 includes a seek bar 841, a 5-second rewind button 842, a previous chapter jump button 843, a play and stop button 844, a slow play button 845, a next chapter jump button 846, a 5-second forward button 847, and a chapter list display button 848.

The seek bar 841 indicates the playback position of the reenacted match. Furthermore, the match reenacted in this embodiment is made up of a collection of one or more specific scenes (hereinafter also referred to as chapters). For this reason, the seek bar 841 may indicate breaks for each chapter, as shown in FIG. 30.

When the chapter list display button 848 is tapped, a chapter list is displayed. FIG. 31 is a diagram showing an example of a chapter list display according to this embodiment. A chapter list 852 is displayed on the match replay UI 850 shown in FIG. 31. The chapter list 852 includes a list of one or more specific scenes (chapters) in a match replayed in this embodiment. By selecting any one of the chapters, the user can jump the playback time of the match video displayed on the match replay UI 850 to the playback start time of the selected chapter.

(Avatar mode)
Next, the avatar mode executed when the avatar mode switching icon 825 is tapped will be described with reference to Fig. 32. Fig. 32 is a diagram showing an example of the display of a match reproduction UI in the avatar mode according to this embodiment.

As shown in FIG. 32, the match reproduction UI 860 in avatar mode displays a user avatar U. The user can operate the controller 521 or the jump icon 522 to move the user avatar U around the field. At this time, the camera position (i.e., the user's viewpoint) is a position that includes the user avatar U in its angle of view, and follows the movement of the user avatar U. In addition, in avatar mode, the user can arbitrarily operate the camera angle, so the display manner of the angle changeable label 823 changes, as shown in FIG. 32.

Note that no collision detection is performed between the user avatar U and virtual objects in the field, such as player objects and the ball. The client terminal 10 may be configured not to allow scene control, such as pausing or fast-forwarding, in the avatar mode.

(Highlight playback)
Next, highlight playback for playing back one or more specific scenes (chapters) of a match by the client terminal 10 will be described with reference to Fig. 33. Fig. 33 is a diagram for explaining highlight playback of a match according to this embodiment.

The client terminal 10 first acquires In and Out point information for each chapter in a match. The In and Out point information is included in the match content data distributed from the content distribution server 306, for example. The In and Out point information for each scene may be automatically set in advance in the match data server 50 based on the match situation.

Next, the client terminal 10 searches for the In point of scene 1 in the tracking data of the players and the ball included in the match content data, and starts playing (recreating the match by moving the player objects), and after playing up to the Out point of scene 1, searches for the In point of the next scene (scene 2) and starts playing the next scene. The client terminal 10 then repeats the above steps until it has played all the scenes set in one match (scenes 1 to 4 in the example shown in FIG. 33).

＜＜6. Supplementary Information＞＞
Although the preferred embodiment of the present disclosure has been described in detail above with reference to the accompanying drawings, the present technology is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modified or amended examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure.

A computer program can also be created to cause the hardware, such as the CPU, ROM, and RAM, built into the client terminal 10 or the virtual space management server 200 to perform the functions of the client terminal 10 or the virtual space management server 200. A computer-readable storage medium storing the computer program is also provided.

Furthermore, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology disclosed herein may achieve other effects that are apparent to a person skilled in the art from the description in this specification, in addition to or in place of the above effects.

The present technology can also be configured as follows.
(1)
Control of arranging, within the virtual space, virtual objects including a display object that displays information related to an event taking place in the real space or the virtual space that is the subject of a conversation within the communication room;
a control for transmitting information regarding a selection to enter the communication room to a terminal used by the user when a position of an avatar operated by a user in the virtual space is included in a first area that is a communication point associated with the communication room where communication between users takes place;
a control for transmitting information related to the communication to the terminal when a position of the avatar in the virtual space is included in a second area located outside the first area;
An information processing system comprising a control unit that performs the above.
(2)
The information processing system according to (1), wherein the display object is a display object that displays a specific scene of an event taking place in a real space.
(3)
The information processing system according to (2), wherein the display object is a display object that displays highlight footage of a match taking place in real space.
(4)
The information processing system according to (1), wherein the display object is a display object that displays information about a team that a specific user group supports in the event.
(5)
the display object is a display object related to a team supported by a specific user group in the event;
The information processing system according to (4), further comprising transmitting information regarding the selection of entry to a terminal used by a user belonging to the specific user group.
(6)
The information processing system described in any one of (1) to (5), wherein when the control unit receives from the terminal a user's request to enter the communication room based on user input regarding information regarding the entry selection, the control unit sends connection information to the terminal used by the user to obtain information via one or more different communication means used in the communication room.
(7)
The information processing device according to (6), wherein the information by the communication means is voice, text, or image information transmitted from the terminals of the users present in the communication room.
(8)
The information processing system according to any one of (1) to (7), wherein the information regarding the selection of entry into the communication room includes information for connecting to a server that generates the communication room.
(9)
The information processing system according to (8), wherein the server controls the transmission of information about users who have entered the communication room to the terminals of each user.
(10)
The information processing system according to any one of (1) to (9), wherein the control unit controls the transmission of information regarding the distribution content being viewed in the communication room to the user's terminal.
(11)
The information processing system according to (10), wherein the information relating to the distribution content includes information for specifying the distribution content and information for connecting to a content distribution server to which the distribution content is distributed.
(12)
The information processing system described in any one of (1) to (11), wherein the control unit receives information regarding the display of a user avatar displayed on a communication room screen displayed on the user's terminal from the user's terminal and transmits it in real time to the terminals of other users who have entered the communication room.
(13)
The information processing system according to (12), wherein the information relating to the display of the user avatar includes emote information for moving the user avatar, or stamp information which is an image displayed near the user avatar and indicates an emotional expression.
(14)
The information processing system described in any one of (1) to (13), wherein the control unit recognizes the intention to enter the communication room when a user avatar corresponding to the user moves within a predetermined distance from the communication point, and controls the transmission of connection information for entering the communication room to the user's terminal.
(15)
The information processing system described in any one of (1) to (14), wherein the control unit controls the user's terminal to instruct the user to transition to a communication room screen when the user enters the communication room.
(16)
The information processing system according to (15), wherein the communication room screen displays user avatars of users who have entered the communication room.
(17)
The information processing system according to (16), wherein characters or a stamp image input by a user corresponding to the user avatar are displayed near the user avatar.
(18)
The information processing system according to any one of (15) to (17), wherein content is displayed on the communication room screen.
(19)
The processor:
Placing virtual objects in the virtual space, the virtual objects including display objects that display information related to an event taking place in the real space or the virtual space that is the subject of a conversation in the communication room;
When a position of an avatar operated by a user in the virtual space is included in a first area which is a communication point associated with the communication room where communication between users takes place, transmitting information regarding a selection to enter the communication room to a terminal used by the user;
When a position of the avatar in the virtual space is included in a second area located outside the first area, transmitting information related to the communication to the terminal;
An information processing method comprising:
(20)
Computer,
Control of arranging, within the virtual space, virtual objects including a display object that displays information related to an event taking place in the real space or the virtual space that is the subject of a conversation within the communication room;
a control for transmitting information regarding a selection to enter the communication room to a terminal used by the user when a position of an avatar operated by a user in the virtual space is included in a first area which is a communication point associated with the communication room where communication between users takes place;
a control for transmitting information related to the communication to the terminal when a position of the avatar in the virtual space is included in a second area located outside the first area;
A program that functions as a control unit to perform the above.
(21)
A streaming video of an event taking place in a real space is placed in a virtual space where a user can view the streaming video of the event;
An information processing system comprising: a control unit that determines an event-linked presentation to be experienced by the multiple users in the virtual space in accordance with the development of an event taking place in the real space and the reactions of the multiple users.
(22)
The information processing system described in (21), wherein the performance is determined according to the amount of reaction from the multiple users.
(23)
The information processing system according to any one of (21) to (22), wherein the performance is determined based on the reaction having the largest amount of reaction among the reactions of the multiple users.
(24)
The information processing system according to (21), wherein the effect is an emote of an avatar, and the emote is reflected in one or more avatars existing in the virtual space.
(25)
The information processing system according to (24), wherein the emote of the avatar is reflected in an avatar in a non-operated state existing in the virtual space.
(26)
The information processing system described in (24), wherein the avatar's emote is reflected in an NPC present in the virtual space.
(27)
The information processing system described in any one of (21) to (26), wherein the reactions of the multiple users include at least any of the following reactions: reaction information of a user operating an avatar existing in one virtual space, reaction information of users operating avatars existing in multiple virtual spaces, and reaction information of a user experiencing an event in real space.

20 Virtual space system 200 (200a, 200b, 200c, ...) Virtual space management server 210 Communication unit 220 Control unit 221 Virtual space management unit 222 Content management unit 223 User management unit 230 Memory unit 10 Client terminal 110 Communication unit 120 Control unit 121 Virtual space processing unit 122 Display control unit 130 Operation input unit 140 Sensor 150 Display unit 160 Audio output unit 170 Memory unit 30 Back-end system 301 User information management server 302 Hosting server 303 Matching server 304 Text chat server 305 Voice chat server 306 Content distribution server 307 Talking table server 308 Event linkage server 309 Trading management server 310 Data analysis server

Claims (20)

Control of arranging, within the virtual space, virtual objects including a display object that displays information related to an event taking place in the real space or the virtual space that is the subject of a conversation within the communication room;
a control for transmitting information regarding a selection to enter the communication room to a terminal used by the user when a position of an avatar operated by a user in the virtual space is included in a first area that is a communication point associated with the communication room where communication between users takes place;
a control for transmitting information related to the communication to the terminal when a position of the avatar in the virtual space is included in a second area located outside the first area;
An information processing system comprising a control unit that performs the above. The information processing system according to claim 1, wherein the display object is a display object that displays a specific scene of an event taking place in real space. The information processing system according to claim 2, wherein the display object is a display object that displays highlight footage of a match taking place in real space. The information processing system according to claim 1, wherein the display object is a display object that displays information about a team that a specific user group supports in the event. the display object is a display object related to a team supported by a specific user group in the event;
The information processing system according to claim 4 , further comprising: transmitting information regarding said selection of room entry to a terminal used by a user belonging to said specific user group. The information processing system according to claim 1, wherein when the control unit receives from the terminal a user's request to enter the communication room based on the user's input regarding the entry selection information, the control unit transmits connection information for acquiring information by one or more different communication means used in the communication room to the terminal used by the user. The information processing system according to claim 6, wherein the information from the communication means is voice, text, or image information transmitted from the terminals of each user who is in the communication room. The information processing system according to claim 1, wherein the information regarding the selection of entry into the communication room includes information for connecting to a server that generates the communication room. The information processing system according to claim 8, wherein the server controls the transmission of information about users who have entered the communication room to the terminals of each user. The information processing system according to claim 1, wherein the control unit controls the transmission of information about the distributed content being viewed in the communication room to the user's terminal. The information processing system according to claim 10, wherein the information relating to the distribution content includes information specifying the distribution content and information for connecting to a content distribution server to which the distribution content is distributed. The information processing system according to claim 1, wherein the control unit receives information about the display of a user avatar displayed on a communication room screen displayed on the user's terminal from the user's terminal and transmits the information in real time to the terminals of other users who are in the communication room. The information processing system according to claim 12, wherein the information relating to the display of the user avatar includes emote information for moving the user avatar, or stamp information that is an image displayed near the user avatar and indicates an emotional expression. The information processing system according to claim 1, wherein the control unit recognizes an intention to enter the communication room when a user avatar corresponding to the user moves within a predetermined distance from the communication point, and controls the transmission of connection information for entering the communication room to the user's terminal. The information processing system according to claim 1, wherein the control unit controls the user's terminal to transition to a communication room screen when the user enters the communication room. The information processing system of claim 15, wherein the communication room screen displays user avatars of users who have entered the communication room. The information processing system according to claim 16, wherein text or a stamp image input by a user corresponding to the user avatar is displayed near the user avatar. The information processing system of claim 15, wherein the communication room screen displays content. The processor:
Placing virtual objects in the virtual space, the virtual objects including display objects that display information related to an event taking place in the real space or the virtual space that is the subject of a conversation in the communication room;
When a position of an avatar operated by a user in the virtual space is included in a first area which is a communication point associated with the communication room where communication between users takes place, transmitting information regarding a selection to enter the communication room to a terminal used by the user;
When a position of the avatar in the virtual space is included in a second area located outside the first area, transmitting information related to the communication to the terminal;
An information processing method comprising: Computer,
Control of arranging, within the virtual space, virtual objects including a display object that displays information related to an event taking place in the real space or the virtual space that is the subject of a conversation within the communication room;
a control for transmitting information regarding a selection to enter the communication room to a terminal used by the user when a position of an avatar operated by a user in the virtual space is included in a first area that is a communication point associated with the communication room where communication between users takes place;
a control for transmitting information related to the communication to the terminal when a position of the avatar in the virtual space is included in a second area located outside the first area;
A program that functions as a control unit to perform the above.

Images