WO2023152976A1

WO2023152976A1 - Communication system

Info

Publication number: WO2023152976A1
Application number: PCT/JP2022/005691
Authority: WO
Inventors: 千春梶塚; 美美梶塚; 琴未梶塚; 時央梶塚
Original assignee: 千春梶塚; 美美梶塚; 琴未梶塚; 時央梶塚
Priority date: 2022-02-14
Filing date: 2022-02-14
Publication date: 2023-08-17

Abstract

Provided is a communication system that enables a vicarious experience of a past in a metaverse as if riding in a time machine in a real world to participate in a scene of an interesting event in the past.　A communication system (100) includes: a multiple virtual spaces construction/distribution unit (200) as a first section; a virtual space construction information storing/updating unit (231) as a second section; a virtual space construction information reproduction unit (232) as a third section; and a virtual space construction information history managing/selecting unit (233) as a fourth section. The multiple virtual spaces construction/distribution unit (200) simultaneously constructs and controls a plurality of virtual spaces on the basis of multimodal information. The virtual space construction information storing/updating unit (231) stores or updates virtual space construction information. The virtual space construction information reproduction unit (232) reproduces virtual space construction information. The virtual space construction information history managing/selecting unit (233) stores and manages an update history of the virtual space construction information.

Description

communication system

The present disclosure relates to a communication system in which a user's communication method is executed in an experience-based service using a virtual space such as the Metaverse, in which at least one or more users participate, and a community service using a virtual space.

A three-dimensional virtual space built on computers and networks that mimics the real world or is different from the real world is called the Metaverse. In the Metaverse, users communicate with each other using avatars that represent users (see

Patent Documents

1 and 2, for example).

　The Metaverse is highly synchronous, just like online games. Here, "high synchronism" means that when users interact with each other, they need to be online at the same time and in close proximity to each other in the same virtual space.

Users participate in the Metaverse by transmitting multimodal information (various information transmission means) from various user terminals. The system constructs and controls a virtual space based on multimodal information from all users, and distributes information about the constructed virtual space to users in the same virtual space. Based on the information, the user terminal displays the virtual space mainly for audiovisual purposes using computer graphics or the like.

JP-A-2000-50226 JP-A-2002-123841

On the other hand, in social network services such as "Twitter" and "Facebook" where synchronism is not important, popular topics written by many users are accumulated, and users who join later can view and comment. However, in the Metaverse, even if a popular event attracts a large number of users, when you visit the space later, it will be empty and lonely. The visualization of the decrease in users after events in the Metaverse is one of the reasons why various Metaverse services have declined.

Furthermore, on SNS, you can reread what you like as many times as you like, but on the Metaverse, you can only relive past events by watching the recorded videos. Despite the fact that the Metaverse is characterized by the removal of various real-world constraints, you cannot go back to the past in the Metaverse any more than you can go to the past by a time machine in the real world.

The invention described in Patent Document 1 (Japanese Patent Application Laid-Open No. 2000-50226) makes it possible to record, play back, and edit past meetings in online meetings. The events that occur in continue without ending. It is desirable to make the means for storing/updating events and the means for replaying events independent processes so that replaying can be started in parallel while storing continues.

In addition, the invention described in Patent Document 2 (Japanese Patent Application Laid-Open No. 2002-123841) assumes a vast virtual space in which many users participate simultaneously rather than a virtual space of limited size such as a conference room. However, Patent Document 1 does not assume such a wide space.

In recent years, the falsification of past events, as typified by fake news, has become a problem. For this reason, it is necessary to manage the history of events that occurred first and updated information so that the user can easily trace the update process.

Therefore, the present disclosure saves multimodal (various information transmission means) inputs regarding the actions and conversations of users and avatars in past events in the Metaverse, and enables them to be played back later, thereby enabling real-world time It allows you to relive the past of the Metaverse as if you were riding in a machine and participating in an interesting event in the past. In addition, users who join later can modify past events.
Systematize a parallel world-like "what-if world" such as "what if the user was there" and "what if the user was acting differently" for events that occurred in the virtual space from the original world. It is possible to realize it as a world that spreads like a tree.

To achieve the above object, the communication system of the present disclosure is a communication system comprising one or more computers, comprising a first part, a second part, a third part, and a fourth and a portion of In a virtual space environment or an information space environment corresponding to a real space in which at least one or more users participate, the first part includes text, voice, still images, moving images, and avatar operation commands input from user terminals. In addition to multimodal information including any of the above, virtual space information, user information, and avatar information acquired based on the input information, and information from the third part, multiple virtual spaces are simultaneously constructed and controlled. and distributes the control result to the user terminal, and the second part saves or updates the information used when the first part constructs and controls the virtual space in a memory in association with the passage of time, The third part reproduces and transmits to the first part information stored or updated by the second part, and the fourth part reproduces information stored or updated by the second part. phylogenetically manages the history of the user's visits, and indicates to the third portion which information has been saved or updated to be reproduced according to the user's selection.

According to the present disclosure, the problem caused by highly synchronous services in experience-based services using virtual space, such as Metaverse, and community services using virtual space, is solved. By adding asynchronous and storage type advantages such as network services, users can efficiently experience the Metaverse according to their limited time.

1 is a conceptual diagram showing a configuration example of a communication system according to a first embodiment; FIG. FIG. 11 is a conceptual diagram showing a configuration example of a communication system according to a second embodiment; FIG. FIG. 10 is an explanatory diagram for explaining an example of a multimodal input procedure for the Metaverse using a smartphone; FIG. 10 is an explanatory diagram for explaining an example of a multimodal input procedure for the Metaverse using a smartphone; FIG. 10 is an explanatory diagram for explaining an example of a procedure for storing virtual space construction information on the basis of space; FIG. 10 is an explanatory diagram for explaining an example of a procedure for storing virtual space construction information based on users and avatars; FIG. 4 is an explanatory diagram for explaining a phylogenetic tree of virtual space construction information;

An embodiment of a communication system in the Metaverse according to the present disclosure will be described below with reference to the drawings.

(First embodiment)
FIG. 1 is a conceptual diagram showing a configuration example of a communication system 100 according to the first embodiment of the present disclosure. As shown in FIG. 1, a communication system 100 according to the first embodiment includes a plurality of user terminals 101, a first portion of a plurality of virtual space construction/distribution units 200, and a second portion of A virtual space construction information holding/updating unit 231, a virtual space construction information reproduction unit 232 as the third part, a virtual space construction information history management/selection unit 233 as the fourth part, and data stored in memory. It comprises a multimodal document 234 which is a structure, a spatial information acquisition section 401 and a user information/avatar information acquisition section 501 . Reference numerals "1100", "1101", "1102", and "1301" in FIG. 1 indicate the flow of information transmitted and received between units.

In the first embodiment, a multiple virtual space construction/distribution unit 200, a virtual space construction information holding/update unit 231, a virtual space construction information reproduction unit 232, a virtual space construction information history management/selection unit 233, The multimodal document 234, the spatial information acquisition unit 401, and the user information/avatar information acquisition unit 501 are installed on the server 110, which is an information processing device, but the configuration is not limited to this. For example, these may be installed in separate servers, may be installed in the cloud or in the user terminal 101, or may be installed in a distributed manner using blockchain technology. By executing the communication program stored in the memory, the server 110 performs multiple virtual space construction/distribution unit 200, virtual space construction information holding/updating unit 231, virtual space construction information reproduction unit 232, virtual space construction information history. It functions as a management/selection unit 233 , a space information acquisition unit 401 and a user information/avatar information acquisition unit 501 .

The user terminal 101 is a device held by a user who participates in the Metaverse provided by the communication system 100 according to the first embodiment. Users participate in the Metaverse using various user terminals 101 such as personal computers, smartphones, and head-mounted displays. There are various types of Metaverses, but all of them use the three-dimensional positional information of the avatar in the virtual space or the positional information of the user in the real space. It determines the range of physical perception.

From the user terminal 101, together with the ID of the user and the ID of the avatar used by the user, text information, audio information, and video information of the conversation, avatar operation information specifying the movement, facial expression, and clothing of the avatar, avatar or user's Multimodal information (via various information transmission means) such as three-dimensional coordinate positions in space and spatial movement information representing changes in movement is transmitted to multiple virtual space construction/distribution unit 200 .

From the user terminal 101, the user may send stamps (stickers) and photos during conversation, send attached documents, or use a slide show. Data that accompanies these conversations is also transmitted as multimodal information.

The multimodal information of the user terminal 101 is transmitted to the multiple virtual space construction/distribution unit 200, which is the first part. This multiple virtual space construction/distribution unit 200 is installed on the server 110 in the first embodiment, but is not limited to this configuration, and may be installed on the cloud or installed on the user terminal 101. However, each user terminal 101 may be connected by P2P and communicate with each other.

The multiple virtual space construction/distribution unit 200 puts together multimodal information transmitted by a user in a certain virtual space and multimodal information of other users in the virtual space, and the user terminal 101 Information necessary to construct a virtual space is processed in real time, and the processing result is delivered to the user terminal 101. - 特許庁At the same time, the multiple virtual space construction/distribution unit 200 performs the same processing for users in another virtual space, and distributes the processing result to the user terminal 101 . The multiple virtual space construction/distribution unit 200 simultaneously performs parallel processing of the virtual spaces in which each user exists for all users, and sequentially distributes the processing results.

The multiple virtual space construction/distribution unit 200 also processes events occurring in each virtual space and distributes them to the user terminal 101 . When users compete in a game, judgments necessary for the game, such as judgment of contact between users and judgment of victory or defeat, are performed and distributed to each user terminal 101 . As described above, the multiple virtual space construction/distribution unit 200 processes information about what happens in the virtual space and distributes it to each user terminal 101 .

The user terminal 101 renders a virtual space image based on the distributed information, and reproduces audio and video.
The space, objects in the space, and graphic data and sound data of the avatar may be distributed by the multiple virtual space construction/distribution unit 200, or may be stored on another system, service, cloud, or blockchain and provided to the user. The information may be acquired by the terminal 101 as necessary, or may be included in advance in the program of the user terminal 101, or the above methods may be combined.

At the same time that the multiple virtual space construction/distribution unit 200 manages and distributes events that occur in the virtual space, the virtual space construction information holding/updating unit 231, which is the second means, distributes to the user terminal 101 and the virtual space. Information similar to the multimodal information that constitutes what is happening in the multimodal document 234 is stored along with time course information. A "multimodal document 234" is a data structure stored in memory.
These "information representing what happened in the virtual space at a specific time" to be saved or updated are called "virtual space construction information". The storage of the "virtual space construction information" by the virtual space construction information holding/updating unit 231 may be instructed by the user from the user terminal 101, or may be performed by the virtual space administrator or system administrator from the administrator terminal. You can instruct me to do it.

It may be possible to store "virtual space construction information" for a specific virtual space. Alternatively, "virtual space construction information" spanning multiple virtual spaces may be collectively saved in accordance with a user who moves between multiple virtual spaces.
The former is called "storing the virtual space construction information based on the space", and the latter is called "storing the virtual space construction information based on the user or avatar".

Various methods are conceivable for the timing of starting and ending saving of the “virtual space construction information”. For example, a method in which the user explicitly specifies the start and end of an online conference, a method in which the virtual space administrator or system administrator sets the start and end times at regular intervals, a method in which no one A method of automatically starting when an avatar enters the space and ending when there are no more avatars can be considered.

The “virtual space building information” is written in the same format as the multimodal information from the user terminal 101 . A virtual space construction information reproduction unit 232, which is a third means, reproduces specific "virtual space construction information" stored in the multimodal document 234 based on time information like video recording and reproduction, and reproduces a plurality of virtual spaces. It sequentially transmits to the construction/distribution unit 200 (see reference numeral 1102 in FIG. 1). The multiple virtual space constructing/distributing unit 200 distributes the information to the user terminal 101, so that the user can re-experience what happened in a specific virtual space at a certain time in the past as if time-tripping into the past.
In addition, when reproducing the "virtual space construction information", the multiple virtual space construction/distribution unit 200 or the virtual space construction information reproduction unit 232 changes the reproduction speed or skips the reproduction information so as to meet the user's request. You can fast-forward, rewind, pause, and skip the event occurrence time in the virtual space according to the time.
Also, the processing of the virtual space construction information holding/updating unit 231 and the processing of the virtual space construction information reproducing unit 232 may be performed independently. As a result, the virtual space construction information reproducing unit 232 can start reproducing information at the same time as the processing by the virtual space construction information holding/updating unit 231 or after the completion of this processing.

The communication system 100 according to the first embodiment not only re-experiences what happened in the past virtual space as a single user, but also allows a plurality of users to form a group, register the group in the system, and participate as group members. It may be configured so that all members can play back the "virtual space construction information" at the same time and act together in the past virtual space.

Furthermore, the user terminal 101 transmits multimodal information of the user's behavior in the reproduced past virtual space to the multiple virtual space construction/distribution unit 200, and the new information from the user terminal 101 and the past " "Virtual space construction information" can be processed and distributed together. As a result, the user not only sees and hears the virtual space in the past, but also adds new actions, adds new remarks, replaces past actions and remarks, and updates what happened in the past virtual space. can.

The virtual space construction information storage/update unit 231 may store in the multimodal document 234 what happened in the updated virtual space. At that time, the past "virtual space construction information" may be reproduced and updated from the start to the end, or only a part of the time may be reproduced and updated.

The first saved virtual space construction information is called "original virtual space construction information", and the updated virtual space construction information is called "updated virtual space construction information". The "updated virtual space construction information" may be further updated and saved. Updated "Original virtual space construction information" to "Updated virtual space construction information" Updated "Virtual space construction information" which further updated "Updated virtual space construction information" of the 1st generation and updated 1st generation 2 The "virtual space construction information" obtained by updating the "virtual space construction information" of the update (n)th generation is referred to as the update (n+1)th generation.

The virtual space construction information saving/updating unit 231 saves only difference information from the "original virtual space construction information" when saving the "updated virtual space construction information". may be saved to avoid tying up storage.

The multiple virtual space construction/distribution unit 200 treats "original virtual space construction information" and "updated virtual space construction information" as belonging to different independent worlds such as parallel worlds. Users in the original virtual space cannot see users or avatars who join later in the updated virtual space. On the other hand, users who join the updated virtual space later can see the users and avatars in the original virtual space.

The virtual space construction information reproducing unit 232 starts reproducing the original virtual space construction with a slight delay before the saving of the original virtual space construction is completed, and saves the original virtual space construction and updates the virtual space. Storage of spatial information may be performed in parallel at the same time. As a result, a state is created in which a parallel world with a slight time lag exists. The multiple virtual space construction/distribution unit 200 processes virtual spaces belonging to different time axes and world lines as different virtual spaces in the same way as parallel processing of multiple virtual spaces at the same time.

When the space, objects in the space, avatar graphic data and sound data are held on another system, service, site, cloud, or blockchain, the multiple virtual space construction/distribution unit 200 acquires space information. Information necessary to reproduce the space and avatar at that time is acquired via the unit 401 and the user information/avatar information acquisition unit 501 (see reference numeral 1301 in FIG. 1) and stored in the multimodal document 234 .

Since graphic data and sound data may be rewritten or deleted, the multiple virtual space construction/distribution unit 200 archives them. As for graphic data and sound data, the author or distributor of each data may be allowed to set user authority as to whether or not to archive the data.

The user sends a request for playback time control from the user terminal 101 to the multiple virtual space building/distribution unit 200 or the virtual space building information playback unit 232, and time-controls what happens in the virtual space like fast-forwarding or rewinding playback of a moving image. You may allow
For example, a user can walk around at normal speed in a virtual space where other avatars are acting at double speed, or participate in a long conversation efficiently by shortening the time. You can also experience a world where time is different.

At the same time when the virtual space construction information storage/update unit 231 saves or updates the "virtual space construction information" in the multimodal document 234, the virtual space construction information history management/selection unit 233, which is the fourth part, ID pointing to "virtual space building information", ID pointing to the original "virtual space building information" to be updated if the "virtual space building information" has been updated, information such as the time when it was saved or updated is saved in the multimodal document 234 as an update history and managed.

The history information managed by the virtual space construction information history management/selection unit 233 is made accessible 1100 from the user terminal 101 so that the user can select past "virtual space construction information". According to the user's selection, the virtual space construction information history management/selection unit 233 instructs the virtual space construction information reproduction unit 232 to reproduce the "virtual space construction information" (see reference numeral 1101 in FIG. 1). By doing so, the user can designate reproduction of the "virtual space construction information".
The user or administrator who designates the storage of the “virtual space construction information” may be allowed to designate the user access authority for browsing and reproduction in the virtual space construction information history management/selection section 233 .

The virtual space construction information history management/selection unit 233 treats the "updated virtual space construction information" derived from the "original virtual space construction information" as a phylogenetic tree starting from the "original virtual space construction information". structure, and displayed to the user via the user terminal 101 .
By displaying like a phylogenetic tree in this way, the user can trace how what happened in the virtual space has been altered. The phylogenetic tree and the difference information of each update may be simply displayed in text or the like so that the change process can be viewed easily.

In addition, by enabling the user to trace the alteration history of the virtual space (metaverse) in this way, the user can know how past alterations were made in the virtual space. Furthermore, by combining the virtual space construction information history management/selection unit 233 with an AI equipped with appropriate judgment means, it is possible to detect tampering with malicious events.

When reproducing the virtual space, the virtual space construction information reproduction unit 232 adds different audiovisual effects to the users and avatars who were in the original virtual space and the users and avatars added in the updated virtual space. can be identified. The updated utterance may be identified by changing the text display color of the utterance or by adding a different sound effect to the voice. It is also possible to display a numerical value indicating how many generations the update corresponds to next to the balloon that displays the avatar or remarks.

A dedicated editor may be used to edit the "virtual space construction information" to create "updated virtual space information" without taking any action in the virtual space. The "updated virtual space information" may be automatically processed by some algorithm, for example, updated such as removing a specific avatar from past events.
As described above, in the communication system 100 of the first embodiment, it is possible to freely modify what happened in the past in the Metaverse, so it is necessary to prevent the dissemination of history-altering fakes.
Therefore, it is desirable that the virtual space construction information history management/selection unit 233 strictly manages the history. Furthermore, history data may be distributed and managed by blockchain so that even system administrators cannot easily modify it.

(Second embodiment)
FIG. 2 is a conceptual diagram showing a configuration example of a communication system 100A according to the second embodiment of the present disclosure. As shown in FIG. 2, the communication system 100A according to the second embodiment includes a plurality of user terminals 101, a plurality of data format conversion units 301 provided corresponding to each user terminal 101, and a first part. Multiple virtual space construction/distribution unit 200, second part virtual space construction information holding/updating unit 231, third part virtual space construction information reproducing unit 232, and fourth part virtual space A construction information history management/selection unit 233, a multimodal document 234 which is a data structure stored in memory, a space information acquisition unit 401, a user information/avatar information acquisition unit 501, and a virtual space construction information learning unit 261. , and an AI avatar/AI event generator 271 . The multiple virtual space constructing/distributing unit 200 includes an input interface 201, an output interface 202, a space sorting unit 203, and multiple virtual space constructing units 221-223. Reference numerals "1100", "1101", "1102", "1200", "1201", "1301", etc. in FIG. 1 indicate the flow of information transmitted and received between the units.

In the second embodiment, a multiple virtual space construction/distribution unit 200, a virtual space construction information storage/update unit 231, a virtual space construction information reproduction unit 232, a virtual space construction information history management/selection unit 233, and a multimodal The document 234, the spatial information acquisition unit 401, the user information/avatar information acquisition unit 501, the virtual space construction information learning unit 261, and the AI avatar/AI event generation unit 271 are installed on the server 110. , but not limited to this configuration. For example, these may be installed in separate servers, may be installed in the cloud or in the user terminal 101, or may be installed in a distributed manner using blockchain technology.

In the communication system 100A of the second embodiment, a user operates an avatar in a virtual space with the user terminal 101 to have a conversation, or moves in real space holding the user terminal 101, tilts the user terminal 101, and specifies in the direction of A user terminal 101 includes a camera, a microphone, a GPS, and the like. User terminal 101 transmits multimodal information to input interface 201 of multiple virtual space construction/distribution unit 200 .
Also, the processing result of the multiple virtual space information constructing/distributing unit 200 is distributed to the user terminal 101 through the output interface 202 .

In the process of transmitting multimodal information from the user terminal 101 to the input interface 201, the data format conversion unit 301 converts data in a format that can be input by the user or a format that is highly convenient for the user into a format that is easy for the system to handle. Convert. For example, the data format conversion unit 301 converts multimodal information into a format that facilitates temporal manipulation by the virtual space construction information reproduction unit 232, a format that facilitates updating later, and a format that reduces the amount of data and does not impose a load on the system. do.

The information delivered from the output interface 202 is also converted by the data format conversion unit 301 into a format that is easy for the user to utilize.

For example, if the user is in a quiet place and cannot speak, the data format conversion unit 301 converts the text so that it can be conveyed to other users by voice, and the other user's voice is converted to text. Convert.
In a situation where the user cannot distribute the self-portrait video, the data format conversion unit 301 or the virtual space construction information reproduction unit 232 generates a video of the conversation according to the conversation based on the face photo prepared in advance by AI or the like. and may be transmitted to the input interface 201 .
When users converse in different languages, the data format conversion unit 301 may translate.

In addition, in the Metaverse, visually and hearing impaired people and able-bodied people will be able to interact seamlessly without being aware of each other. For example, in the case of a hearing-impaired person, the data format conversion unit 301 converts the voice conversation of another user into text and displays it, so that the text-inputted conversation can be conveyed to the other user by synthesized speech. For the visually impaired, the data format conversion unit 301 converts text chat of other users into voice.

Such a data format conversion unit 301 may be provided in each user terminal 101 and data conversion processing may be performed on the client side, or may be provided on the server 110 or cloud side and data conversion processing may be performed on these sides. If the user terminal 101 has sufficient processing power, placing the data format conversion unit 301 on the client side will balance the load of the entire system. If an excellent system such as translation conversion already exists, the data format conversion unit 301 may use the function of such other system or service to perform the conversion. Details of the data format conversion will be described later with reference to FIGS.

Although the input interface 201 and the output interface 202 are installed on the server 110, they may be installed on the cloud, or may be installed on each user terminal 101 so that the user terminals 101 communicate with each other. Also good.

The input interface 201 handles the multimodal information from the user terminal 101, the multimodal information from the virtual space construction information reproduction unit 232, and the multimodal information from the AI avatar/AI event generation unit 271, which will be described later.

Multimodal information received by the input interface 201 is distributed to a plurality of virtual space construction units 221 to 223 by a space distribution unit 203 . The spatial distribution unit 203 is a mechanism for distributing loads by parallel processing similar to the load balancer of the server 110, but the distribution logic is different from that of the load balancer.
In the case of the original virtual space construction, the space distribution unit 203 distributes users in the same virtual space to the same virtual space construction units 221-223. When the loads on the virtual space construction units 221 to 223 increase and the processing cannot keep up, the space distribution unit 203 distributes users in the same virtual space to different virtual space construction units 221 to 223. If the user has designated to act in a group, the space distribution unit 203 preferentially distributes to the same virtual space construction units 221 to 223 .

In the case of the updated virtual space construction, the space sorting section 203 sorts the users acting in groups to the same virtual space building sections 221-223. Alternatively, when distributing the updated virtual space construction information to an unspecified number of users, the space distribution unit 203 distributes the received users to the same virtual space construction units 221 to 223 .
Even if they are the same virtual space, it is not necessary to distribute the original virtual space, the updated virtual space, and the virtual spaces of different update generations to the same virtual space constructing units 221 to 223 .

In the Metaverse, the system needs to handle many virtual spaces at the same time, and the load is too high for a single server or process to handle. Therefore, the virtual space constructing units 221 to 223 process a plurality of virtual spaces in parallel. The virtual space constructing units 221 to 223 can be increased or decreased according to the load of the entire system and the number of active users in the virtual space. A thread for executing the virtual space construction units 221 to 223 is automatically created when a user or avatar enters a specific empty virtual space, and when there is no user or avatar left in the virtual space. Allow threads to terminate automatically. In FIG. 2, three virtual

space construction units

221, 222, and 223 are displayed, and two rectangles marked with "..." are displayed between them. represents what is possible.

The virtual space construction units 221 to 223 are installed on a specific server 110, but they may be installed virtually on the cloud or distributed on each user terminal 101 connected by P2P connection. The space sorting unit 203 is designed to be able to grasp when the virtual space constructing units 221 to 223 are newly opened or closed.

What happens in the space containing the information of other users in the same space where the user or his/her avatar is is processed by the virtual space constructing units 221 to 223 and delivered to the user terminal 101 via the output interface 202. be. At the same time, the virtual space construction information storage/update unit 231 stores or updates what happened in the virtual space in the multimodal document 234 .

The virtual space construction information storage/update unit 231 may perform parallel processing in the same way as the virtual space construction units 221 to 223 to distribute the load. Since "virtual space information" that spans multiple virtual spaces may be saved, it is installed independently of the virtual space constructing units 221-223.

Also, the virtual space construction information storage/update unit 231 acquires graphic data required for the virtual space through the space information acquisition unit 401 and the user information/avatar information acquisition unit 501, and stores them in the multimodal document 234 (1301). .
The virtual space construction information storage/update unit 231 acquires graphic data via the space information acquisition unit 401 and the user information/avatar information acquisition unit 501 so that the user terminal 101 can reproduce the virtual space at that time. Good (see reference numeral 1303 shown in FIG. 2).

The "virtual space construction information" is stored in the multimodal document 234, and this information is also information about what actions the user or avatar took in a specific situation. The multimodal document 234 stores a large amount of "virtual space construction information", and the virtual space construction information learning unit 261 uses machine learning to learn data representing necessary and sufficient behavior of the user or avatar under a specific situation. user or avatar behavior can be predicted and behavior can be generated.
Similarly, the virtual space construction information learning unit 261 can also predict and generate what kind of event will occur in the virtual space if users or avatars with what attributes gather.
As a result, the communication system 100A can construct a Metaverse that is crowded with many avatars and various events occur even though no user actually participates. In addition, the communication system 100A can construct a village-like village that is filled with NPC (non-player character) villagers in an adventure game and that conveys information to the user.

The learning result of the virtual space construction information learning unit 261 is transmitted to the AI avatar/AI event generation unit 271, and the AI avatar/AI event generation unit 271 generates an avatar acting by AI and an event controlled by AI. These avatars and events are described in the same format as multimodal information and "virtual space construction information" from the user terminal 101, and are transmitted from the AI avatar/AI event generation unit 271 to the input interface 201 (Fig. 2 1201) can be handled by the multiple virtual space construction/distribution unit 200 in the same way as the user's avatar.
The AI avatar/AI event generation unit 271 changes the response according to the user's multimodal information, thereby generating an avatar and a situation that appropriately responds to the user's behavior. For example, generated avatars and situations can be used to build dynamic learning environments in conversational style.

In the Metaverse, a user can normally operate only one avatar at a time, but if an AI avatar with a personality that comprehends the user's behavior becomes possible, it will be possible for the user to act in multiple virtual spaces at the same time. Possessing an avatar, you can experience what happened in each virtual space later in fast-forward, and you will be able to use your limited time effectively.
Also, even if the user is offline, the avatar of the user's alter ego will perform the task or mission of the user.
Such possibilities also provide a solution to the problem of the Metaverse, in which the time in the real world and the time in the Metaverse progress in a one-to-one relationship, and the problem of the user's time being consumed more and more.

If the user can operate the setting of the virtual space construction information learning unit 261, it becomes possible to customize AI avatars and AI events. Users can buy and sell customized AI avatars and AI events.
For example, users can use NFT ("Non-fungible token", non-fungible token) to secure the uniqueness of the virtual space where the dialogue event is held and the AI avatar that can explain their specialized knowledge. It will be possible to treat it as such and sell it.

　Figures 3 and 4 are explanatory diagrams for explaining one example and another example of a multimodal input procedure for the Metaverse using a smartphone. User operations at the user terminal 101 and data conversion at the data format conversion unit 301 will be described below with reference to FIGS. 3 and 4. FIG.

Although a smart phone is used as the user terminal 101 in this example, the user terminal 101 is not limited to a smart phone. The user terminal 101 is provided with a camera 102 and a microphone 103, and incorporates a GPS function for acquiring the user position in real space and various sensors (not shown) for detecting the tilt and movement of the smartphone. For the sake of explanation, in FIGS. 3 and 4, the smartphone screen is divided into upper and lower parts, the virtual space 1001 by computer graphics is displayed in the upper part, and the screen corresponding to each input is displayed in the lower part, but this is just an example. It does not limit interface design.
The user terminal 101 may express the position of the user, the surrounding environment, the positions of other users, and the like with stereoscopically localized sounds without visually displaying the virtual space.

In the virtual space 1001, an avatar 1002 of the user and avatars 1003 of other users in the same space are displayed.

FIG. 3(a) is an example of conversation input by text input. A text input screen 1010 is displayed at the bottom of the screen of the user terminal 101 , and the user inputs characters using a keyboard (touch panel) and transmits a series of characters input using a send button 1011 .
In this example, the text "Hello" is sent, followed by a sticker with an illustration image. The user makes a series of 'tap inputs' 3012 on the screen along the 'time lapse' 3000 . In the example, "H", "e", "l", "l", "o", the send button, and the stamp are tapped in order. In the explanation, the operation of the stamp is simplified. The data to be transmitted is processed as two data groups of “text “Hello”” 3013 and “stamp ID” 3014 .

"Common text data" 3001 is encrypted and transmitted together with "text "Hello"" 3013 and "stamp ID" 3014. As "common text data" 3001, a user ID, an avatar ID, the local time of the user terminal 101, the user's position in real space, and the like are transmitted.

(b) of FIG. 3 is an example of voice conversation input. In this example, a “send voice button” 1020 is displayed below the “voice chat screen” 1021 . The user speaks while pressing the “send voice button” 1020 for a long time. "Voice" 3023 of "from start to end of long press" 3022 of "send voice button" 1020 is processed as a series of speech data. This “speech” is subjected to “speech text conversion processing” 3024 by the data format conversion unit 301 . A series of “conversation voice data” 3025 is composed of text converted into voice-to-text, ID indicating voice data, original “voice data” 3026 , and “common text data” 3001 .

By transmitting the conversation voice as text, the virtual space construction information storage/update unit 231 can text-edit the past conversation in the "updated virtual space construction information". The data format conversion unit 301 can also convert the edited text into text-to-speech and replace the past conversation speech. If AI-synthesized voice mimics the user's voice, the data format conversion unit 301 can replace the past conversation in the virtual space as if the user had spoken at that time.

(c) of FIG. 3 is an example of a conversation input using a moving image. This example is similar to the voice chat example. The user converses while pressing and holding the 'video transmission button' 1030 on the 'video chat screen' 1031 displayed on the screen of the user terminal 101 . In the case of video chat, since hands-free operation is often desired, the user terminal 101 may identify the start and end of speech by volume.
"Audio" 3033 is subjected to "speech text conversion" 3024 by the data format conversion unit 301, "moving image" 3034 is subjected to "moving image compression" 3035, and text converted as a series of data 3036, "sound data" 3023, "moving image data 3037 and “common text data” 3001 are transmitted from the user terminal 101 to the server 110 .
In the "moving image compression" 3035, the data format conversion unit 301 uses a general moving image compression algorithm that reduces the data transfer load, and also converts moving images into frame images of faces corresponding to vowels of speech and characteristic frame images with movement. and send only those keyframe images. The user terminal 101 of another user interpolates the keyframe image into a moving image to generate a moving image of the user speaking.
When part of the utterance is edited in the “updated virtual space construction information”, the data format conversion unit 301 replaces and interpolates the key frame images to create a video, thereby converting the speech in the past virtual space at that point in time. It is replaced as if the user had spoken.
There is a technique for generating an animation of an animated face from a photograph of a face. A moving image during normal conversation may be generated from a still image of the user.

FIG. 4(a) relates to movement of the avatar in space. At the bottom of the screen of the user terminal 101, an "avatar movement operation screen" 1040 is displayed obliquely from above (quarter view) to clearly display the position of the avatar in the virtual space. A display viewed from directly above (bird's eye view) like a map application may be used, or an avatar movement operation may be performed using a dedicated input terminal for a head-mounted display.
On the "avatar movement operation screen" 1040, "own avatar" 1043 and "another user's avatar" 1042 are drawn. Also, a “virtual space where the user is present” 1041 and an “adjacent virtual space” 1042 are displayed. A technique described in Patent Document 2 (Japanese Unexamined Patent Application Publication No. 2002-123841), for example, can be used as a method of dividing a virtual space into blocks like a website page and connecting those blocks to represent a large virtual space. .

The user taps the "avatar movement operation screen" 1040 to move the avatar. When you tap a specific location on the screen, an animation of your avatar walking to that location is played. In accordance with the "passage of time" 3000, the user continuously taps a plurality of points a, b, c, and d on the "avatar movement operation screen" 1040 (3044) to move the avatar. In b, tap twice to stop. The user terminal 101 reduces the communication load and the processing load of the server 110 by transmitting only the movement position to the server 110, but may transmit detailed movement data like a game.

The avatar's position information follows the general method of indicating the position of an object in computer graphics. The angle (x, y, z) and the scaling factor (x, y, z) may be represented by a set of numeric text data. In addition, it sends how to interpolate the movement animation between each position and the next position. Common specification methods for keyframe animation include "linear", which connects specified positions in a straight line, and "spline", which connects specified positions with a smooth curve.
The location information is transmitted together with the “common text data” 3001 . "P1" 3045 of the transmission data represents movement within the same virtual space, and "P2" 3046 represents movement accompanied by movement to a different virtual space. In the case of “P2” 3046 , information indicating the destination virtual space, such as an ID uniquely held by the Metaverse system and the URL of the virtual space data, is sent from the user terminal 101 to the server 110 .

(b) of FIG. 4 relates to avatar operations other than movement. An avatar waving or greeting (motion), a handshake that cooperates with other avatars (interaction), facial expressions such as emotions, avatars appearing in a virtual space, or specific actions It includes information such as visual effects (effects) associated with the avatar, clothes, hats, shoes, and belongings (items) worn by the avatar. Here, motion and facial expressions are explained as an example.

An avatar “motion designation screen” 1050 is displayed at the bottom of the screen of the user terminal 101 . Several "thumbnails of poses" 1051 are displayed at the top of the screen, and "selected poses" 1050 are displayed at the bottom of the screen. The user taps the "thumbnail of pose" 1051 according to the "passage of time" 3000, connects poses, and designates a motion (3052). Similarly, the user terminal 101 displays options for facial expressions on the screen, and the user taps them to designate facial expressions (3053).
This is a simple method of specifying motion and facial expressions, but there is also a method in which the user attaches a sensor device to the body and records the movement (motion capture) and transmits it in real time. The user terminal 101 transmits "common text data" 3001 together with motion and facial expression data designated by various methods. A motion or facial expression may be linked to the conversation stamp.

(c) of FIG. 4 is a case of transmitting continuous sound and moving images. In such a case, the user continues talking into the camera 102 and the microphone 103 of the user terminal 101 without operating the screen, and confirms his own "camera image" 1060 .
The user terminal 101 divides the continuous “speech” 3061 and “moving image” 3062 accompanying the “passage of time” 3000 at regular time intervals so that the server 110 can easily handle them, and divides them into “speech-to-text conversion” 3024 and “moving image compression” 3024 . After performing data format conversion such as 3035, it is transmitted to the server 110 as a continuous data group together with the converted text and "common text data" 3001 (3061).

Next, referring to FIGS. 5, 6, and 7, how the virtual space construction information storage/update unit 231 stores the multimodal information from the “user terminal 101” as a series of “virtual space construction information”. and how the virtual space construction control information history management/selection unit 233 manages them.

FIG. 5 is an explanatory diagram for explaining an example of a procedure for "saving virtual space construction information based on space". FIG. 6 is an explanatory diagram for explaining an example of a procedure for "saving virtual space construction information based on a user or avatar".
In either case, "Start Saving" 2001 is the point at which the virtual space construction information storage/update unit 231 starts saving the "virtual space construction information", and "End Saving" 2004 is the point at which the saving ends. treated as a set of "virtual space construction information". 5 to 7, the symbols "2000", "2010" to "2019" shown on the left side of the page indicate a series of data groups sent to the virtual space construction information storage/update unit 231 over time. For example, “recorded in space A” 2000 is a data group recorded in space A and transmitted to the virtual space construction information storage/update unit 231 . Codes “2100” to “2103” indicate “updated virtual space construction information” which is the result of putting these data groups together by the virtual space construction information storage/update unit 231 . In FIGS. 5 to 7, the symbols shown on the right side of the paper indicate the time (point of time) at which an event occurs or the process to be executed.

The “original virtual space construction information” shown in FIG. 5 will be described. In the example of “recording in space A” 2000 , some event starts at “Start Event” 2002 and ends at “End Event” 2003 . For example, a guitarist's avatar appears in the virtual space at the "Start Event" and performs, and then exits the space at the "End Event".
Before the start of the virtual event, users B and C converse in the space and wait for the event, and user A appears in the space just before the start of the event. User A leaves the virtual space shortly after the event ends.

In ``virtual space construction information to be saved'' 2100, ``recording in space A'', a series of multimodal information described in ``user A's input (behavior)'' 2010, and ``user B's input (behavior) 2011, and a series of multimodal information described in 'User C's input (behavior)' 2012 are saved. In 2010, 2011, and 2012, the conversation text, conversation voice, avatar action, etc. shown in FIGS. 3 and 4 are displayed. The "input (behavior) of user A" indicates "Enter the Space" 2005 when user A enters space A and "Exit the Space" 2006 when user A exits space A. FIG.
This series of "virtual space construction information" is "original virtual space construction information".

The “updated first generation+user D” shown in FIG. 5 will be described. User D updates the “original virtual space construction information” and saves the update result as “updated virtual space construction information” 2101 . User D starts the reproduction from the beginning of the "original virtual space construction information" 2007, conducts several voice conversations, selects an avatar action such as clapping for the guitarist, and then ends the reproduction 2008.
The organizer of the event has specified to automatically update the 'updated virtual space construction information', and the 'updated virtual space construction information' 2101 has been updated. This is the first updated generation.

The “update first generation + user E, user B (revisit)” shown in FIG. 5 will be described. User E forms a group with user B, reproduces the "virtual space construction information", and acts in it. User B's avatar also participates in the "original virtual space construction information", but participates in the event again with user E. User B joins late, talks with user E, and leaves early.
Due to the update by user B, part of the "input of user B" in the "original virtual space construction information" is rewritten. The updated utterance overwrites the previously spoken utterance, or is inserted as a new utterance.
The standing position and clothes of user B's avatar may be different from those of the "original virtual space construction information". In such a case, the original user B's avatar may be replaced only while user B is participating.
Alternatively, like the movie "Back to the Future", the original avatar of user B and the updated avatar of user B1 may be displayed simultaneously.

"Updated second generation + user D + user F" shown in Fig. 5 will be explained. When user F reproduces the "updated virtual space construction information" by user D, user F's behavior at the time of this reproduction is further saved as the updated second generation "updated virtual space construction information" 2103 .

FIG. 6 is an explanatory diagram for explaining an example of a procedure for "saving virtual space construction information based on users and avatars".
The “original virtual space construction information” shown in FIG. 6 will be described. User A first visits space B, then moves to space A to participate in an event, and moves to space C. Different "virtual space construction information" are summarized as "stored virtual space construction information" 2100 as a result of user A's actions processed in space A, space B, and space C. FIG.
User B and user C are in space A, and user A can talk with their avatars while staying in space A.
User G visits space A halfway through, meets user A and has a conversation with him, then moves to space C with user A, and continues the conversation with user A.

The “updated first generation+user H, user I” shown in FIG. 6 will be described. In the first update generation, user H and user I reproduce the "original virtual space building information" as a group, and move in space B, space A, and space C as if accompanied by user A.
User H moves in the space without leaving user A, but user I moves in the space with a slight delay. Therefore, in updating the virtual space information stored with user A as a reference, user H is temporarily out of the space where user A is, and disappears from the update information.
When the participating users are temporarily divided into a plurality of spaces in this way, the information about the plurality of spaces may be included in the "updated virtual space construction information" at the same time.
However, if the avatars of a large number of users are scattered in different virtual spaces, or if update generations are superimposed and new spaces are added each time an update is performed, a large amount of information about the virtual space is included in the virtual space construction information. Since it will be necessary, it is appropriate to exclude users who have moved to a different space from user A from saving.

FIG. 7 is an explanatory diagram for explaining the phylogenetic tree of the virtual space construction information. Based on the example of FIG. 5, this FIG. 3103 is updated, and a phylogenetic branch is formed.

Depending on the intended use of the user, there is a possibility that the way in which the "virtual space construction information" branches in a phylogenetic tree in this way is not necessarily easy to use. However, when various participants deepen the content of the dialogue and archive it, such a method of preserving it in a phylogenetic tree is effective.
In the case of light contents, it is possible to display only the virtual space construction information viewed by many users instead of displaying the phylogenetic tree, or to introduce only the popular virtual space construction information by introducing an evaluation system by users. may be displayed to the user.

As described above, the embodiments of the present disclosure have been described in detail with reference to the drawings, but the above embodiments are merely examples of the present disclosure, and the present disclosure is not limited only to the configurations of the above embodiments. . Even if there is a change in design within the scope of the present disclosure, it is of course included in the present disclosure.

This disclosure can be used for experience-based services and community services using virtual spaces such as the Metaverse.

In the Metaverse, users' online statuses are visualized, so spaces without other avatars appear to be depopulated spaces. In addition, when many users have not been acquired at the start of the service, users rarely meet other users, and users who do not meet other users lose their motivation to continue the service.
Even in a space without such a user, by accumulating the time when the space was busy and allowing the user to return to that time, they can experience the "bustle" of other users accumulated in the past, and further By being able to participate in the "bustle", it is possible to avoid the impression that the service as a whole has depopulated. In this way, we can solve one of the fundamental problems of the Metaverse, the problem of "depopulated spaces," and contribute to the profitability of the service.

This disclosure can also be used to create interactive educational content that makes use of conversations within the Metaverse. The content provided by the communication system of the present disclosure is different from mere interactive educational content. Active participation as if you traveled back in time to a past event, such as walking around in a group and listening to an explanation of how it works. becomes possible. As shown in the example of FIG. 6, by storing the space construction information from the viewpoint of the user who moves across multiple virtual spaces, for example, it is possible not only to explain in a nuclear reactor in a single space, but also to You can experience explanations while moving around the entire space of the power plant, or while moving from the power plant space to the virtual space of a wide area such as the urban space where electricity is used.

By using AR ("Augmented Reality" mixed reality) technology that projects an avatar in a real place using dedicated devices such as smartphone screens and smart glasses, users can see historical sites where historical battles were fought, for example. While actually visiting various places in the historic site, you will be able to actively participate by listening to explanations from avatars and asking questions. At a later date, the facilitator checks the added avatar questions, further updates the virtual space, and adds answers to the questions.
In this case as well, as in the previous example, it is possible to carry out experiential learning in which the presenter's avatar and the participant's avatar walk together around historical sites spanning a plurality of large areas. Participants can also participate only at points of interest to them.

The communication system of the present disclosure accumulates many dialogues, cuts and compiles multiple pieces of "virtual space construction information", eliminates redundant parts, and creates new content such as a collection of dialogues to experience. can.

The communication system of the present disclosure can also be used to make a monologue in the virtual space and update it later to add a comment. You can also use the Metaverse as an idea processor to deepen your thoughts. You can use two different avatars to ask yourself about conflicting opinions within yourself, or you can make a monologue linked to a real place. For example, you can organize your thoughts about cultural heritage while walking around it.
Collected "virtual space construction information" can be monetized as content unique to the Metaverse.

The communication system of the present disclosure may be equipped with artificial intelligence, and with the development of artificial intelligence technology, it becomes possible for the avatar to behave more highly human-like, allowing the user to accurately respond to requests, If a part of the user's knowledge and way of thinking is inherited and it becomes an existence that acts as an alter ego that cuts out a certain characteristic of the user, it will be possible to realize a new service for sharing knowledge. Since such an AI avatar resides in the reproduced virtual space, it is possible to respond to new participants as if the original user were there. Furthermore, if users can nurture avatars with their own specialized knowledge and ways of thinking, the usage rights can be attached to virtual space dialogue events and sold or lent. The “knowledge market” that buys and sells such expertise creates value.

The communication system of this disclosure does not keep avatars in the virtual world, but when combined with robots and self-driving cars, it will be possible to participate in dialogues in each place while acting in the real world.

In this way, the present disclosure can be used as a mechanism for creating new uses of the Metaverse in industrial fields such as education and entertainment.

Claims

A communication system comprising one or more computers, comprising a first portion, a second portion, a third portion, and a fourth portion,
In a virtual space environment or an information space environment corresponding to a real space in which at least one or more users participate,
The first part includes multimodal information including any one of text, voice, still image, video, and avatar operation command input from the user terminal, and virtual space information obtained based on the input information; Simultaneously constructing and controlling a plurality of virtual spaces based on the information from the third part in addition to the user information and the avatar information, and delivering the control results to the user terminal;
The second part saves or updates the information used when the first part constructs and controls the virtual space in a memory in association with the passage of time,
said third portion reproduces and transmits to said first portion information stored or updated by said second portion;
The fourth part systematically manages the history of information saved or updated by the second part, and stores the saved or updated information to be reproduced according to the user's selection in the third part. A communication system characterized by instructing.
The third part changes the playback speed or skips the playback information when playing back the information to fast-forward, rewind, pause, or pause the event occurrence time in the virtual space according to the user's request. 2. The communication system of claim 1, configured to skip.
The storage or update of information by the second part and the reproduction of information by the third part are configured to be performed independently, and the third part stores the information by the second part. 3. The communication system according to claim 1, wherein reproduction of information can be started simultaneously with updating or after completion of saving or updating of information by said second part.