JP2023033844A - Experience sharing system, and experience sharing method - Google Patents

Abstract

To provide an experience sharing system and an experience sharing method which can be more flexibly utilized according to a user's various convenience.SOLUTION: An experience sharing system comprises: one or more user terminals in which an experience sharing application PN is installed; and an experience sharing server 1 connected with the user terminals through a network as a communication network. Within a session in which one or more users UN constituting a party are included, an experience can be shared on a site through the experience sharing application PN in each user terminal. Session information to manage the session is stored in the experience sharing server 1, and the experience sharing application PN in each user terminal, respectively. A change made in an experience sharing state within the session when the experience is supplied is reflected in each session information.SELECTED DRAWING: Figure 15

Description

The present invention relates to an experience sharing system and an experience sharing method.

2. Description of the Related Art Conventionally, an experience sharing system has been proposed and put into practice, which enables a remote user to share various experiences using a user terminal and via a communication network such as the Internet.

For example, in Patent Document 1, when visiting a grave from a remote location, the user does not need to actually go to the grave at all. Disclosed is a remote grave-visiting system that enables confirmation of what has been done.

Furthermore, partly due to the impact of the current global pandemic, while maintaining social distancing, people who are far apart can connect in all aspects of life, such as education, tourism, medical care, and sports, so that no one is left behind. , a new lifestyle that allows people to experience things together safely and securely.

For that reason, there are new and various new technologies that utilize a highly integrated space (hereafter referred to as CP (Cyber Physical) space) that incorporates real (physical) information into computer virtual space (Cyber). Many experience sharing systems have also been proposed.

Although many different experience-sharing systems have been proposed in this way, under the circumstances described above, in the future such experience-sharing systems will be used more flexibly according to the various conveniences of users. There is a need to be able to However, no such experience-sharing system has been seen so far.

JP 2005-284676 A

Accordingly, the present invention has been devised in view of the above-described problems, and aims to provide an experience sharing system and an experience sharing system that can be used more flexibly according to the various conveniences of users. It is to provide a method.

In order to solve the above-described problems, the inventor of the present invention provides an experience sharing system comprising one or more user terminals in which an experience sharing application is installed, and a server connected to the user terminals via a communication network. Experience sharing is possible on the site via an experience sharing application on the user terminal within a session in which one or more users constituting a party are included, and the server and the experience sharing application on the user terminal share the experience through the session. session information for managing experience sharing is stored, and when the session information stored in the server or any one of the experience sharing applications is changed in sharing the experience, the other server and the change is reflected in the session information stored in the experience sharing application.

The experience sharing system according to claim 1 is an experience sharing system comprising one or more user terminals in which an experience sharing application is installed, and a server connected to the user terminals via a communication network. experience sharing on the site via an experience sharing application on the user terminal within a session in which one or more users are included, and the server and the experience sharing application on the user terminal share experiences through the session are stored, and when the session information stored in the server or any one of the experience sharing applications is changed in sharing the experience, the other servers and the experience The change is reflected in the session information stored in the shared application.

The experience sharing system according to claim 2 is characterized in that in the invention according to claim 1, the server stores at least party information for authenticating the user's participation in the party.

The experience sharing system according to claim 3 is characterized in that, in the invention according to claim 1 or 2, the server manages the site for experience sharing within the session via a URI.

The experience sharing system according to claim 4 is the invention according to any one of claims 1 to 3, wherein the one experience sharing application includes difference data of the session information according to a change in the experience sharing state. is transmitted to the server, and the server reflects the received difference data in the session information stored therein, and transmits the difference data to other experience sharing applications.

In the experience sharing system according to claim 5, in the invention according to claim 4, the server accumulates the difference data for a predetermined time and transmits the difference data to another experience sharing application. When the same data as newly received difference data exists in the difference data, the same data is deleted from the accumulated difference data.

The experience sharing method according to claim 6 is an experience sharing method performed by one or more user terminals in which an experience sharing application is installed and a server connected to the user terminals via a communication network. Experience sharing is enabled on the site via an experience sharing application on the user terminal within a session that includes one or more users, and the server and the experience sharing application on the user terminal manage the session. session information is stored for each session, and when a change occurs in the experience sharing state in the session during the supply of the experience, it is reflected in each of the session information.

According to the present invention configured as described above, when the session information stored in the experience sharing server or one of the experience sharing applications is changed, the session information stored in the other experience sharing server and the experience sharing application is changed. The change is reflected in As a result, various changes that occur when experiences are shared in one experience sharing application can be reflected in other experience sharing applications, and experience sharing between users can be smoothly advanced.

FIG. 1 is a schematic diagram showing a schematic configuration of the overall experience sharing system according to the embodiment of the present invention. FIG. 2 is a block diagram showing a schematic configuration of the experience sharing system according to the embodiment of the invention. FIG. 3(a) is a schematic diagram for explaining a site, FIG. 3(b) is a schematic diagram for explaining a party, and FIG. 3(c) is a schematic diagram for explaining a session. It is a diagram. FIG. 4 is an explanatory diagram showing a specific example 1 of experience sharing in the experience sharing system according to the embodiment of the present invention. FIG. 5 is an explanatory diagram showing a specific example 2 of experience sharing in the experience sharing system according to the embodiment of the present invention. FIG. 6 is an explanatory diagram showing a specific example 3 of experience sharing in the experience sharing system according to the embodiment of the present invention. FIG. 7 is an explanatory diagram showing a specific example 4 of experience sharing in the experience sharing system according to the embodiment of the present invention. FIG. 8 is an explanatory diagram showing a specific example 5 of experience sharing in the experience sharing system according to the embodiment of the present invention. FIG. 9 is a plan view showing a state in which a panorama video camera placed at a certain position in the physical space is shooting images in all directions. FIG. 10 is a plan view showing an example in which avatars are placed at any of the positions in the CP space reproduced based on the past video. FIG. 11 is a diagram showing an example in which an experience sharing system to which the present invention is applied is configured by an experience sharing application installed in a user terminal used by a user and an experience sharing server. FIG. 12 is a diagram showing an example in which one party configures a plurality of sessions. FIG. 13 is a diagram showing an example of specifying a site by URI. FIG. 14 is a diagram showing the main components of the experience sharing system. FIG. 15 is a diagram for explaining the synchronization method of the experience sharing system according to this embodiment. FIG. 16 is a diagram illustrating an example of distributing difference data based on changes made to the session information on the experience sharing application side. FIG. 17 is a diagram for explaining details of session information for managing sessions. FIG. 18 is a diagram showing an example of exchanging immediate messages for immediately notifying changes in session information in the experience sharing system to which the present invention is applied. FIG. 19 is a diagram showing an example in which each user designates a recipient and transmits/receives an immediate message to/from the designated recipient without changing the contents of the session information. FIG. 20 is a diagram showing an example of having a session clock for keeping session time between the experience sharing server and the experience sharing application. FIG. 21 is a diagram showing an example of synchronizing event occurrence times in the experience sharing system to which the present invention is applied. FIG. 22 is an explanatory diagram for explaining details of account information and party information recorded in the experience sharing server. FIG. 23 is a diagram for explaining a management method at the start of a party in session administration management. FIG. 24 is a diagram for explaining methods of joining and leaving a party in session management. FIG. 25 is a diagram for explaining a method of moving a session and changing a site in session administration management. FIG. 26 is a diagram for explaining an example of terminating a session and a party in session administration management. FIG. 27 is a diagram showing an example of providing an extended function serving as a window for exchanging contents etc. in cooperation with an external service or the like. FIG. 28 is a diagram for explaining a specific example of an extended function in the experience sharing system according to this embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments exemplified by applying the present invention will be described below with reference to the drawings.

[Embodiment]
First, an experience sharing system 100 according to an embodiment of the present invention will be described using FIGS. 1 and 2. FIG. FIG. 1 is a schematic diagram showing a schematic configuration of the overall experience sharing system 100 according to the embodiment of the present invention, and FIG. 2 is a block diagram showing a schematic configuration of the experience sharing system 100 according to the embodiment of the present invention. .

An experience sharing system 100 according to an embodiment of the present invention includes an experience sharing server 1 as a server, as shown in FIG. The experience sharing server 1 is connected to a plurality of user terminals 31 to 3N in which an experience sharing application is installed and a management terminal 5 via a network N as a communication network.

The experience sharing server 1 enables experience sharing on the sites 41 to 4N via an experience sharing application on the user terminals 31 to 3N within a session including one or more users forming a party, details of which will be described later. It is a host computer that serves as a platform for The experience sharing server 1 also serves as a central control unit that controls the entire experience sharing system 100 .

The user terminals 31 to 3N are terminal devices for users themselves to share their experiences through an experience sharing application, and are composed of, for example, personal computers (PCs), smart phones, tablet terminals, wearable terminals, and the like. Further, the user terminals 31 to 3N may be composed of head-mounted displays (HMDs) worn on the heads of users or spectacle-type terminals. The user terminals 31 to 3N are provided with at least displays for displaying various contents, images taken by cameras, and the like. By visually recognizing the contents and images displayed on the user terminals 31 to 3N, the user can feel as if he himself is actually experiencing the space. These user terminals 31 to 3N can be wired or wirelessly connected to the network N. FIG.

The management terminal 5 is a terminal device for the administrator to operate the experience sharing system 100 to which the present invention is actually applied and for the user to share the experience through the experience sharing application. , wearable terminals, etc. The management terminal 5 can be connected to the network N by wire or wirelessly. The management terminal 5 can become the management terminal 5 by setting one or more of the other user terminals 31 to 3N to be used for management.

As shown in FIG. 2, the experience sharing server 1 includes an integrated management unit 10 including a CPU (Central Processing Unit).

The overall management section 10 is connected to an interface 11 for transmitting and receiving information to and from a plurality of user terminals 31 to 3N via a network N, and a storage section 2 such as a memory and a database. The general management section 10 manages various data stored in the storage section 2 . In addition, the general management unit 10 records various data in the storage unit 2, and reads out various data stored in the storage unit 2 when necessary for operating the experience sharing system 100. , and control the distribution to the outside.

Next, the operation of the experience sharing system 100 to which the present invention having the configuration described above is applied will be described.

In the experience sharing system 100 to which the present invention is applied, information is shared on the CP space with users who want to have a joint experience or users who want to have a joint experience in various scenes such as life, work, and entertainment. The CP space here refers to a more realistic cyber space that fuses the physical (real world) and cyber (virtual world). In addition, the CP space is not limited to a rectangular parallelepiped, but may be a sphere in the case of a 360° image, or any other shape. good. In addition, the joint experience here means sharing the same characters and voices with each other on the user terminals 31 to 3N owned by each user, and interacting as avatars in the same video space. , it means that each user enjoys the same experience with the same sense of realism. In other words, by sharing the content displayed on one user terminal 31 and the video captured by the camera with the other user terminals 32 to 3N, the displayed content and the video captured by the camera can be shared among the users. You can share what you are experiencing in a space where images and the like are projected. When the content displayed on one user terminal 31 changes, the other user terminals 32 to 3N also change the content in the same way, so that the users can always enjoy the feeling of sharing the experience. can be done.

In the present invention, the following attributes are defined for realizing experience sharing between users.

First, as shown in FIG. 3(a), a site means "something/place to be shared", and is composed of characters, voices, real images, 3D data, etc., and spaces in which these are combined. The site is a space that the user experiences, such as a classroom, a conference room, a model room, a sightseeing spot, a construction site, or the like.

A party is a group of one or more users U1 to UN, as shown in FIG. 3(b). Users U1-UN who wish to share experiences through the site must belong to at least this party, and users who do not belong to the party cannot share experiences through the site. This party is a group formed by one or more of, for example, a school class, a contest participant, a group travel group, etc., and a group that can only participate in those who are given specific authority and a group that does not have any authority. can form a group that can freely come and go.

A session, as shown in FIG. 3(c), is a state in which one or more users U1 to UN share information at a certain site. In other words, users enter a site and experience content for the purpose of sharing experiences. The session here also includes a state in which one user participates in the site. When a plurality of users U1-UN participate in a site and form a session, the users U1-UN participating in the site can experience the same content, realizing so-called experience sharing. The Rukoto.

Assuming that the above-mentioned party is a class at school, the term "session" here means sharing the experience in some groups of the class. However, the session is not limited to this, and since it is possible for all the users who make up the party to participate, assuming that the above party is a school class, all the classes Experience sharing through the site may be used.

Next, specific examples 1 to 5 in which users share experiences through sessions in the experience sharing system 100 according to the embodiment of the present invention will be described with reference to FIGS. 4 to 8. FIG. FIG. 4 is an explanatory diagram showing a specific example 1 of experience sharing in the experience sharing system 100 according to the embodiment of the present invention, and FIG. 5 is a specific example of experience sharing in the experience sharing system 100 according to the embodiment of the present invention. 2, FIG. 6 is an explanatory diagram showing a specific example 3 of experience sharing in the experience sharing system 100 according to the embodiment of the present invention, and FIG. 7 is an explanatory diagram showing experience sharing according to the embodiment of the present invention. FIG. 8 is an explanatory diagram showing a specific example 4 of experience sharing in the system 100, and FIG. 8 is an explanatory diagram showing a specific example 5 of experience sharing in the experience sharing system 100 according to the embodiment of the present invention.

First, in the specific example 1 of the experience sharing system 100 shown in FIG. 4, the display image C1, which is the CP space, is a real image of various still images or moving images, and is assigned to each of the users U1 to UN who form a session with each other. The avatars A1 to AN walk around in the display image C1, which is the CP space, and display and visually recognize the images of the angle of view in any direction in the CP space consisting of 360° on each of the user terminals 31 to 3N. By exchanging letters, voice, etc. with other avatars, it is possible to have a joint experience.

In addition to each user independently strolling in the CP space and communicating with each other, among the users U1 to UN, the authorized user U1 operates his/her own user terminal 31, and from 360° When an image with an angle of view in an arbitrary direction in the CP space is viewed, or when an operation to zoom up or zoom back is performed, similar images are displayed on the other user terminals 32 to 3N in conjunction with this. or point a point in the video to other user terminals 32-3N. Therefore, each of the users U1-UN can share an experience by visually recognizing the common video displayed on the user terminals 31-3N and listening to the common sound. Such an experience sharing method can be applied, for example, to compensate for joint experience activities such as field trips and outdoor learning, and to be used for remote on-site inspections, room previews, and the like.

A specific example 2 of the experience sharing system 100 shown in FIG. 5 is an example of telemedicine. Patients as users U2 to UN are made to wear user terminals 32 to 3N consisting of HMDs in the CP space, and images viewed by users U2 to UN through the user terminals are viewed by a doctor as user U1 who forms the same session. is displayed on the user terminal 31. As a result, the doctor as the user U1 can share the situation visually recognized by the users U2 to UN as patients, and the efficiency of diagnosis can be improved. Experience sharing via user terminals 32 to 3N composed of such HMDs can be applied to telemedicine through remote medical measurement of a plurality of subjects, mindfulness in the relaxation industry, marriage hunting, etc. It can be applied to diagnosis of compatibility in matching, etc., empathy for experiences with care recipients in nursing homes and nursing care facilities, communication tools, and the like.

Next, a specific example 3 of the experience sharing system 100 shown in FIG. 6 is an example in which a CP space reproduced by a still image or a moving image is projected in a planetarium dome. A plurality of avatars A1 to AN wearing space suits assigned to the respective users U1 to UN who form sessions with each other are displayed through the user terminals 31 to 3N within the display image C1 in the CP space consisting of the dome of the planetarium. By flying around freely and visually recognizing an arbitrary direction in the CP space consisting of 360°, it is possible to display the angle of view of the image in that viewing direction on the user terminals 31 to 3N. In such a case, similarly, among the users U1 to UN, the user U1 who has the authority operates the avatar A1 via his/her own user terminal 31, and also visually recognizes an arbitrary direction in the CP space consisting of 360°, or When a zoom-up or zoom-back operation is performed, similar images can be displayed on the other user terminals 32 to 3N in conjunction with this operation. Therefore, each of the users U1-UN can remotely share the planetarium experience with each other. In the dome of the planetarium, in addition to being able to view any direction in the CP space with the user terminals 31 to 3N, it is also possible to view the images projected in the dome and the movements of the avatars A1 to AN from a bird's-eye view. .

Next, in a specific example 4 of the experience sharing system 100 shown in FIG. 7, the display image C1, which is the CP space, is composed of the actual image captured by the camera mounted on the robot R. As shown in FIG. The movement of the head of the user U1 wearing the user terminal 31 consisting of an HMD corresponds to the movement of the robot R, etc., and the control device 7 held in both hands corresponds to the operation of the movement of the robot R with both hands. , the robot R can be remotely operated while viewing the display image C1. In such a case, the user U1 can exchange the operation right with another user U2 who composes the same session. By sharing the operation of C1 and the robot R, the user U2 can remotely operate the robot R in the same way as the user U1. As a result, the robot R can be moved by the movement of the head of a user wearing a user terminal consisting of an HMD. Collaborative work of high remote robots can be performed.

Next, a specific example 5 of the experience sharing system 100 shown in FIG. 8 implements a module that takes in 3D data from outside in the CP space. The 3D data from the outside may be static objects such as works of art or products for exhibition, or dynamic performances such as plays by incorporating actors as 3D data. good too. The example shown in FIG. 8 is an example in which a practical haircutting training course is conducted online, and the user uses a user terminal such as an HMD or a tablet terminal to view the 3D data of the mannequin head displayed in the CP space. 32 can be operated to observe the practical skill.

In order to realize such experience sharing, images captured in real time or past images captured in the past may be applied. These images are captured using a capturing module. Specifically, an image is captured by a video camera, a digital camera, a 3D scanner, or the like in this image capturing module. The plan view of FIG. 9 shows a state in which a panorama video camera 37 arranged at a position P in the physical space is shooting images in all directions. The imaging device 321 in the panorama video camera 37 performs 360-degree imaging in the horizontal direction without omission. This imaging is also performed in the vertical direction at the same time, but in the following example, the CP space reproduced by recording the imaging in the horizontal direction and using that image will be described as an example.

When the images are captured in this way, the images are sequentially captured in time series in each direction by the plurality of imaging devices 321 . The captured images are accumulated sequentially. At the stage when the imaging of the past video is completed, still images or moving images captured in all directions in the horizontal direction centering on the position P are accumulated. At this time, voice may be recorded simultaneously by a microphone (not shown). Such omnidirectional images are similarly captured for each of the other positions Q to T in the physical space. As a result, 360° omnidirectional images are accumulated at each position in part or all of the physical space.

When the experience sharing described above is performed using such past images, as shown in FIG. avatars exist. For example, when the avatar AN is positioned at the position P, when displaying the past video on the user terminals 31 to 3N, the user terminals 31 to 3N read the past video stored in advance for the position P and display it for display. can be displayed above.

At this time, if the line-of-sight direction, position, state, etc. of the avatar AN in the CP space are changed via the user terminals 31 to 3N, the change may be reflected in the display on the display. In such a case, an image corresponding to the line-of-sight direction of the avatar is cut out from the previously accumulated images and displayed on the display for display of the user terminals 31 to 3N, or the image is enlarged or reduced according to the position of the avatar. Let When this avatar AN moves from the position P to another position Q to T, the pre-stored image of the moved position Q to T is clipped or enlarged or reduced according to the operation of the avatar AN. indicate. In this way, by displaying an image cut out or enlarged or reduced according to the operation of the avatar AN, the user can visually recognize the image as if it were in accordance with the direction of the line of sight of the avatar AN. It is possible to give a sense of reality that is being done. Moreover, since pre-stored images can be similarly displayed for each of the positions P to T to which the avatar AN is moved, each of the users U1 to UN can enjoy the same realism as the avatar AN walking freely in the CP space. becomes possible.

Incidentally, in the case of an application in which users U1 to UN wearing user terminals 31 to 3N such as HMDs, instead of avatars AN, can freely move around in this CP space, each user in the CP space in which the past video was captured It is assumed that users U1-UN exist at any one of positions P-T. For example, when the user is positioned at position P, when past images are displayed on the user terminals 31 to 3N, the user terminals 31 to 3N read the past images stored in advance for the position P and display them on the display for display. to display. When the users U1-UN move from the position P to another position Q-T, the pre-stored images of the moved positions Q-T are clipped or enlarged according to the operation of the users U1-UN, or Display a reduced image. In such a case, similarly, the user terminals 31 to 3N composed of HMDs can be operated, or the line-of-sight direction and the position of the user wearing the user terminals 31 to 3N can be reflected in the display on the display. In the same way, it is possible to give a sense of reality to visually recognizing an image according to the direction of the line of sight or the like. When the user terminals 31 to 3N comprising HMDs are used, it is possible to reflect the display on the display based on the operation controller provided in the HMD, the hand tracking function, the positional information of the HMD itself, and the like. good.

When sharing experiences in such a CP space, for example, user U1 himself or his avatar A1 is positioned at position P, and user U2 himself or his avatar A2 is positioned at position Q. At this time, if the person having the authority is the user U1, the image displayed on the display of the user terminal 31 by the user U1 can be displayed on the display of the user terminal 32 operated by the user U2. can. The image displayed on the display of the user terminal 31 is the image viewed at the position P, and can be viewed not only by the user U1 but also by the user U2 at another position Q. Therefore, the user U1 and the user U2 can simultaneously view the image viewed by the user U1 or his/her avatar at the position P, and can share the experience with each other. At this time, it is also possible to share the experience including the voice by playing the voice recorded in the past at each of the positions P to T.

Next, details such as control management of the experience sharing system 100 according to the present embodiment will be described with reference to FIGS. 11 to 28. FIG. 11 to 28 are explanatory diagrams for explaining details such as control management of the experience sharing system 100 according to the present embodiment.

First, a schematic configuration of an experience sharing system 100 according to this embodiment will be described.

As shown in FIG. 11, the experience sharing system 100 includes experience sharing applications P1 to PN installed in user terminals 31 to 3N used by users U1 to UN, and an experience sharing server 1 connected to a network N. be. The experience sharing applications P1-PN provide the same session to the users U1-UN. The video and audio captured at each of the above-described positions P to T are transferred to the experience sharing applications P1 to PN via the experience sharing server 1, and are transferred in response to the operations of the experience sharing applications by the users U1 to UN. are read out and displayed on the user terminals 31 to 3N.

Note that in this experience sharing system 100, an image captured by one user U1 operating the experience sharing application P1 is not stored in the experience sharing server 1, and the same image is transferred to the other experience providing applications P2 to PN. You may make it distribute. Alternatively, the video once stored in the experience sharing server 1 may be streamed to the other experience providing applications P2 to PN by the operation of the experience sharing application P1 by the one viewing user U1. Alternatively, the video once stored in the experience sharing server 1 may be downloaded to the other experience providing applications P2 to PN by the operation of the experience sharing application P1 by the one viewing user U1.

As a result, a large number of experience sharing applications P1 to PN communicate with the experience sharing server 1 via the experience sharing protocol, thereby sharing information for the audiovisual experience and interaction described above.

Note that if the semantics of using the experience sharing protocol (contents and handling of data transferred through the experience sharing protocol) are the same, different experience sharing applications P1 to PN can be mixed and implemented.

The party described above is composed of users U1 to UN and U1' to UN' who can share information through the experience sharing system 100, as shown in FIG. A session for sharing information with one or more users U1-UN at a certain site among the parties is formed via the experience sharing applications P1-PN. Similarly, other sessions for sharing information with one or more users U1'-UN' at other sites are formed via experience sharing apps P1'-PN'. The experience sharing applications P1 to PN and the experience sharing applications P1' to PN' form different sessions.

In this experience sharing system 100, users U1 to UN and U1' to UN' of the experience sharing application specify a party, connect to the experience sharing server 1, and optionally specify a site to participate in a session within the party. can participate. In addition, in this experience sharing system 100, it is also possible to create a new session within the party or move to a different session on the way.

As shown in FIG. 13, in this experience sharing system 100, each site such as content or CP space shared by users U1 to UN or users U1' to UN' participating in the same session is unique to the session. tied to. Therefore, changing the site will switch the session. A site consists of all kinds of resources that can be specified by a URI (Uniform Resource Identifier). For this reason, this site also includes contents such as online still images and moving images, live streaming images, PDF data, 3D data, CP spatial data in unique formats, etc., which can be specified by URL.

The main components of the experience sharing system 100 are, as shown in FIG. It comprises a server program 53 , a storage unit 2 , an experience sharing protocol 55 that defines communication between the client library 52 and the server program 53 . In the experience sharing system 100, the developer of the experience sharing application PN can use the client library 52 to create the original program 51 to develop the experience sharing application PN such as the above application examples. can be done. Various joint experiences can be provided to the user UN by this experience sharing application PN. In this experience sharing system 100, the experience sharing server 1 may be provided with an extension function 54 serving as a window for extension functions for cooperating with an external server or the like.

Next, a method for synchronizing the states of the experience sharing system 100 according to the present embodiment will be described.

In the experience sharing system 100, as shown in FIG. 15, information about the state of a session, such as audiovisual information provided by the experience sharing applications P1 to PN to the users U1 to UN, is described and stored as session information. Also, the experience sharing server 1 side stores session information serving as a master. The experience sharing server 1 provides the latest session information to the experience sharing applications P1 to PN that have newly joined the session and handle the same semantics. Furthermore, in the experience sharing system 100, changes made to the session information by the experience sharing applications P1 to PN are propagated to the other experience sharing applications P1 to PN and the experience sharing server 1 that handle the same semantics via the experience sharing protocol 55. and keep their session information synchronized.

More specifically, in the experience sharing system 100, the entire latest session information is sent to each of the experience sharing applications P1 to PN with the same semantics when participating in a session (step S11). If, for example, one of the experience sharing applications P1 to PN is changed (step S12), the master session information in the experience sharing server 1 is updated based on the change. (step S13). As a result, on the experience sharing server 1 side, the difference between the session information before updating and the session information after updating is distributed to the other experience sharing applications P2 to PN as stream data (step S14). As a result, changes made in one experience-sharing application P1 can be reflected in the other experience-sharing applications P2 to P2N.

Further, in the experience sharing system 100, as shown in FIG. 16, the experience sharing applications P1 to PN experience, at regular intervals, difference data based on changes to the session information added to the experience sharing applications P1 to PN. You may make it transmit to the shared server 1 (step S15). When difference data based on changes in session information added in each of the experience sharing applications P1 to PN is sent to the experience sharing server 1, the experience sharing server 1 aggregates the sent changes and are reflected at regular intervals (for example, 0.2 sec) in the session information to be the master within (step S16). At the same time, the aggregated changes, that is, the difference data are transmitted to the experience sharing applications P1 to PN (step S17). Thereby, the difference data based on the changes made in each of the experience sharing applications P1 to PN are collectively accumulated on the experience sharing server 1 side, and the accumulated difference data are distributed to all of the experience sharing applications P1. ~ PN side can also be reflected. Data of session information sent at regular intervals is called a session stream. Session streams include differential streams that transmit only differential data and full streams that collectively transmit all data.

FIG. 17 shows the configuration of session information for managing each session. The session information includes a session management section 61, a party common section 62, a user list section 63, a session specific section 64, a site specific section 65, and the like. Note that the experience sharing applications P1 to PN participating in the session are any or all of the session management section 61, the party common section 62, the user list section 63, the session specific section 64, and the site specific section 65 in the session information. may be shared. Also, the party common section 62 is shared (synchronized) by all sessions within the party.

Here, the session management unit 61 includes information such as a session identifier, a site (URI) corresponding to the session, and time within the session.

The party common portion 62 includes information shared by all sessions within the party, and some of it may be permanently recorded in the experience sharing server 1 even after the session ends. In other words, the party common section 62 records information that is updated each time a session is started to its end, and that is deleted at the end of the session. may be

The user list portion 63 includes information on users U1 to UN participating in the session. The information of each user U1-UN recorded in the user list section 63 is added to the experience sharing applications P1-PN when the user U1-UN participates in the session, and is deleted when the user leaves the session. is.

The session specific part 64 records the temporary state of the experience sharing applications P1-PN shared within the session. The information of the experience sharing applications P1 to PN recorded in the session unique section 64 is discarded when the session ends.

The site-specific portion 65 records setting information for each site and site-specific information generated by the experience sharing applications P1 to PN. The site-specific information recorded in the site-specific portion 65 may be permanently recorded in the experience sharing server 1 even after the session ends. The site-specific part 65 may record, as site information, information that cannot be described only by the URI depending on the session. Also, the site-specific part 65 may record information, knowledge, etc. newly generated in the space as a memo, and is utilized when necessary knowledge is left after the end of the session.

The semantics of the information included in the four sections other than the session management section 61 (the party common section 62, the user list section 63, the session specific section 64, and the site specific section 65) among the session information are as follows. Developers of the experience sharing applications P1 to PN can also arbitrarily determine.

An example of information actually described as session information in certain semantics will be described below. The semantics exemplified here are that users U1 to UN appear as avatars in a CP space of 360° video, move freely in the space, freely change their viewpoint, and point to a location in the video for enlarged display. Alternatively, it is an experience sharing application PN that can be displayed in reduced size.

In the session management unit 61, the URL of the online 360° video file is set as the URL of the site.

In the party common part 62, as permanent information, a list of contents that can be displayed by the experience sharing application PN, and information such as operation authority for each of the users U1 to UN who have participated in the party or have participated in the party in the past are described. . The operation authority here includes information such as the playback state of the 360° video, operating the viewpoint of the 360° video, pointing to a certain part of the 360° video, and displaying an avatar in the 360° video.

In the user list section 63, information such as the display name of each user U1 to UN, the position and orientation of the avatar, and the position of the image pointed by the user U1 to UN is described in each user information. The information of each of the users U1 to UN recorded in the user list section 63 may be updated each time, for example, when the position or orientation of the avatar operated by the user changes from moment to moment.

Whether the session-specific part 64 has the operation authority to operate the playback state of the 360° video, to operate the viewpoint of the 360° video, to point to a certain part of the 360° video, to display the avatar in the 360° video, etc. etc. is described.

The site-specific portion 65 describes additional content information such as texts and reference photographs placed in the 360° video.

That is, according to the experience sharing system 100 to which the present invention is applied, session information for managing sessions is stored in the experience sharing server 1 and the experience sharing applications P1 to PN, respectively. Under this premise, for example, when the experience sharing application P1 is operated to change the position or orientation of the avatar, session information stored in the experience sharing application P1 is changed. be. In addition, this session information change may also occur on the experience sharing server 1 side. When the session information stored in the experience sharing server 1 or any one of the experience sharing applications P1 to PN is changed, the sessions stored in the experience sharing server 1 or the other experience sharing applications P1 to PN The change is reflected in the information. As a result, various changes that occur during experience sharing in one experience sharing application P1 to PN can be reflected in the other experience sharing applications P1 to PN, thereby smoothly promoting experience sharing among the users U1 to UN. be able to.

Also, in the experience sharing system 100, instant messages can be exchanged as shown in FIG.

Specifically, the experience sharing applications P1 to PN can transmit instant messages to the experience sharing server 1 (step S18). This immediate message uses differential data to be synchronized as a parameter in order to immediately synchronize the session information without waiting for delivery from the experience sharing server 1 at regular intervals. Upon receiving the instant message from one experience sharing application P1, the experience sharing server 1 immediately distributes the same instant message to the other experience sharing applications P2 to PN (step S19). At the same time, the contents of this instant message are overwritten and reflected in the session information to be mastered in the experience sharing server 1 (step S20). At this time, when the experience sharing server 1 accumulates difference data for a predetermined period of time and then transmits the difference data to another experience sharing application, the same data as the newly received difference data exists in the already accumulated difference data. In this case, the same data may be deleted from the accumulated differential data (step S21). This makes it possible to avoid accumulation of duplicate difference data.

Further, in the experience sharing system 100, as shown in FIG. 19, each of the users U1 to UN may specify a receiving target and immediately send and receive messages to and from the specified target.

When specifying other users as recipients, specifically, one user U1 specifies other users U2 to UN as recipients and transmits an immediate message (step S22). Next, the instant message is delivered to the experience sharing applications P2-PN of the designated recipients (other users U2-UN) (step S23). Note that the instant message may use the recipient itself as a parameter, or arbitrary data as a parameter. Also, the user as recipient can specify either the whole party, the whole session, a specific user UN, or a specific data object. Furthermore, the method (semantics) of handling the received parameters (data) can be arbitrarily determined by the developers of the experience sharing applications P1 to PN independently of the session information.

Further, in the experience sharing system 100, as shown in FIG. 20, the experience sharing server 1 and the experience sharing application PN may have a session clock that is commonly used in a session to keep session time.

Specifically, when the experience sharing application PN participates in the session (step S24), the current session time is notified from the experience sharing server 1 (step S25), and the session on the experience sharing application PN side is started at the notified time. A clock starts (step S26).

In the experience sharing system 100, the experience sharing application PN continuously observes the session time of the experience sharing server 1, and synchronizes the session time in the experience sharing application PN with the session time of the experience sharing server 1. Let

Specifically, the session time t0 on the experience sharing application PN side is transmitted to the experience sharing server 1 (step S27). The experience sharing server 1 returns the received session time t0 and the session time t1 on the experience sharing server 1 side to the experience sharing application PN (step S28). Based on the received session time t2 on the experience sharing application PN side and the received t0 and t1, the communication delay tL and the time difference tD are calculated by the formula (tL = (t2-t0)/2 and tD = (t0 + t2) /2-t1) to adjust the deviation of the session clock (step S29). It should be noted that the method of adjusting the deviation of the session clock described above is merely an example, and it goes without saying that the adjustment may be performed based on any other method.

Moreover, in the experience sharing system 100, as shown in FIG. 21, event occurrence times can be synchronized.

Specifically, the session time is added to the message for synchronizing the state of each session from the experience sharing server 1 . This message for synchronizing the state of the session is a session stream, an immediate message, or the like. The experience sharing applications P1 to PN update the expressions of states and events in consideration of the difference between the session time given to the message and the current session time.

As a method of synchronizing the state, for example, when synchronizing video playback, the session specific unit 64 described above stores in advance the session time when the video operation was completed, the new playback state (play/pause/reverse), and the video playback at that time. Describe the position (frame number and time code), etc. Then, when the experience sharing applications P1 to PN perform various control operations for moving images such as playback stop and seek, the experience sharing applications P1 to PN transmit such information as parameters to the experience sharing server 1 as an immediate message. The experience-sharing server 1 distributes the reproduction state of the moving images to be synchronized to the experience-sharing applications P1 to PN via a session stream, an instant message, or the like. As a result, the video playback states of the experience sharing applications P1 to PN are synchronized. The experience providing application PN that newly participates in the session can use the information included in the session stream to reproduce the playback state of the moving image.

Next, a session management method of the experience sharing system 100 according to the present embodiment will be described.

In the experience sharing system 100, as shown in FIG. 22, the storage unit 2 of the experience sharing server 1 records account information about a plurality of accounts, party information about parties, and the like.

The account information includes an account identifier, authentication information, type, authority, and the like.

The party information includes party attribute information, party common part persistent information, party participant information, site-specific information for each site, and the like. The party attribute information describes information such as the identifier and name of the party, the creator of the party, various authority information, and the initial site. Permanent information in the common part of the party describes information and knowledge that will be used permanently instead of being erased after the session ends. The party participant information describes accounts of users who can participate in the party, identifiers of each participant valid only within the party, authority of each participant, and the like. The site-specific information describes detailed information for each site. The site-specific information may describe the URI of the site.

The experience sharing system 100 uses such account information and party information to manage session operations. First, the management method at the start (activation) of a party will be described. As shown in FIG. 23, when the experience sharing application PN connects to the experience sharing server 1 (step S30), the connection is first authenticated based on the account information in the storage unit 2 (step S31). If the connection is authenticated based on this account information, the process proceeds to step S32, and if the connection is not authenticated, the activation process ends.

When the process proceeds to step S32, the account information and the party participant information in the party information can be collated and restricted. Since the party participant information describes the accounts of the users U1 to UN who can participate in the party, the users who have accessed via the experience sharing applications P1 to PN are authenticated for participation by collating them. be able to. If the participation is authenticated, the process proceeds to step S33 to proceed to party activation processing, and if the participation is not authenticated, the activation processing ends immediately.

If the party is activated in step S33, a session corresponding to the site defined by the party is provided and session information is created. When creating the session information, first, a party-specified site is provided based on the party attribute information in the party information. If there is permanent information of the party common part in the party information, it is shared in the permanent information of the party common part 62 in the session information. Also, the user who participates in the session is registered in the user list section 63 in the session information from the party participant information in the party information. The site-specific information in the party information is written in the site-specific part 65 in the session information. In this way, it is possible to generate session information corresponding to the site defined by the party.

As a result, the user UN can participate in the session corresponding to this session information via the connected experience sharing application PN (step S34).

Next, I will explain how to join and leave a party. As shown in FIG. 24, when a new experience sharing application PN connects to an already active party (step S35), the connection is authenticated based on the account information in the storage unit 2 (step S36). If the connection is authenticated based on this account information, the process proceeds to step S37, and if the connection is not authenticated, the connection is disconnected.

When the process proceeds to step S37, the party participant information in the party information is referenced as necessary, and the participation right of the user UN who owns the newly connected experience sharing application PN is checked. As a result, if the participation right is not required, the user UN is directly added to the session corresponding to the site specified by the party. If participation rights are required and valid, add user UN to the session corresponding to the party-defined site. Specifically, the newly added user UN is added to the user list section 63 in the session information. As a result, the user UN can participate in the session corresponding to this session information via the connected experience sharing application PN (step S38). If there is no session corresponding to the prescribed site, a new session corresponding to the prescribed site may be created and made to participate.

Also, when the user UN who has participated in the session so far ends the joint experience by the experience sharing application PN, the experience sharing application PN is disconnected from the experience sharing server 1 (step S39). As a result, the experience sharing application PN leaves the session. At that time, the user UN may leave the party (step S40). Also, when the user UN leaves the party, the user UN who has left may be deleted from the user list section 63 in the session information.

Next, a method of moving a session, that is, changing a site will be explained. As shown in FIG. 25, the experience sharing application PN connects to the experience sharing server 1 and requests a site change (step S41). Next, the connection is moved to the session corresponding to the destination site (step S42). In addition, a process of removing the corresponding user UN from the user list portion 63 of the session information before movement is also performed (step S43). Also, the party participant information in the party information is referenced, and the corresponding user UN is added to the user list section 63 of the destination session information (step S44). Here, there are three types of target users UN to be moved, depending on the site change mode: only the requesting user UN, all participants in the same session, and all party participants. If there is no session corresponding to the destination site, a new session may be created based on the party information to participate. As a result, the user UN can participate in the session at the destination from the session before the movement.

Next, the case of ending the session and party will be explained. In the experience sharing system 100, as shown in FIG. 26, the experience sharing application PN disconnects from the experience sharing server 1 (step S45). Then, the experience sharing application PN itself withdraws from the session (step S46). The user UN who operates the experience sharing application PN is deleted from the user list section 63 in the session information. Next, in step S47, if there is site-specific information to be permanently managed in the site-specific portion 65 in the session information, it is reflected in the party information. Also, the session is terminated and the session information is deleted (step S48). When all the sessions are completed, if there is permanent information in the party common section 62, it is saved in the permanent information of the party common section in the party information (step S49), and the party is terminated and deactivated (step S50). .

In this way, in the experience sharing system 100 to which the present invention is applied, when the user UN starts a party, participates in a session, and shares an experience, participation authentication can be performed via party information. In addition, when the user UN changes the site where the experience is shared, or when the session or party ends, by reflecting the situation in the party information, participation or withdrawal of the entire experience sharing system 100, site change, etc. can be collectively managed through party information. Therefore, it is possible to prevent a suspicious person who does not belong to a party from freely participating in a session and freely moving between sessions.

As shown in FIG. 27, the experience sharing system 100 may be provided with an extension function 54 that functions as a window for exchanging contents in cooperation with an external service (server) or the like. Finally, a method for extending the function of the experience sharing system 100 by linking the extension function 54 with another external server 70 will be described.

FIG. 28 shows an example in which the experience sharing system 100 cooperates with a voice call server. The experience sharing server 1 creates a voice channel corresponding to the session in the voice call server 1' (step S51). When the experience sharing application PN connects to the experience sharing server 1 and participates in the session (step S52), it issues a connection token to the voice call server 1' (step S53) and notifies the experience sharing application PN (step S54). ). When the experience sharing application PN connects to the voice call server 1' with a token (step S55), it is made to participate in the channel for that session (step S56), and a voice call is started (step S57). Note that when the experience sharing application PN moves between sessions, the participation channel of the voice call server 1' is also changed accordingly.

Although the embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

1 Experience sharing server 1' Voice call server 2 Storage unit 31-3N User terminals 41-4N Site 5 Management terminal 7 Control device U1-UN User U1'-UN' User U User P1-PN Experience sharing application P1'-PN 'Experience sharing application R Robot A1-AN Avatar C1 Display image (CP space)
N network (communication network)
10 General management unit 11 Interface 37 Panorama video camera 41 Site 51 Original program 52 Library 53 Server program 54 Extended function 55 Experience sharing protocol 61 Session management unit 62 Common party unit 63 User list unit 64 Session specific unit 65 Site specific unit 100 Experience sharing system

Claims (6)

An experience sharing system comprising one or more user terminals in which an experience sharing application is installed, and a server connected to the user terminals via a communication network,
Experience sharing is possible on the site via an experience sharing application on the user terminal in a session including one or more users constituting a party,
session information for managing experience sharing through the session is stored in the server and the experience sharing application in the user terminal, respectively;
When the session information stored in the server or any one of the experience sharing applications is changed in the sharing of the experience, the session information stored in the other server and the experience sharing application is updated with the changed amount. An experience sharing system characterized by reflecting 2. The experience sharing system according to claim 1, wherein said server stores at least party information for authenticating a user to participate in said party. 3. The experience sharing system according to claim 1, wherein the server manages the site for experience sharing within the session via a URI. The one experience sharing application transmits difference data of the session information corresponding to the change in the experience sharing state to the server,
4. The server reflects the received differential data in session information stored therein, and transmits the differential data to another experience sharing application. Experience sharing system described. The server accumulates the difference data for a predetermined period of time and then transmits the difference data to another experience sharing application. 5. The experience sharing system according to claim 4, wherein the same data is deleted from the stored difference data. In an experience sharing method performed by one or more user terminals in which an experience sharing application is installed and a server connected to the user terminals via a communication network,
Experience sharing is possible on the site via an experience sharing application on the user terminal in a session including one or more users constituting a party,
Session information for managing the session is stored in the server and the experience sharing application in the user terminal, respectively. An experience sharing method characterized by reflecting session information.

Images