JP7312337B1 - CONFERENCE SYSTEM, CONTROL DEVICE, CONTROL METHOD, PROGRAM, AND RECORDING MEDIUM - Google Patents

Abstract

Kind Code: A1 Seamlessly switching between devices used in a conference in a virtual space. A conference system includes a server 10 and a plurality of terminals 30 and is capable of holding a remote conference in a virtual space. Terminals 30A and 30B are used by participant A, and transmit data for controlling participant A's avatar in the virtual space. The server 10 has a provision unit 11 that provides a remote conference within the virtual space, and a control unit 12 that controls the avatar within the virtual space with data received from the terminal 30 . When the control unit 12 receives data from the terminal 30B logged in with the same account of the participant A as the terminal 30A, the control unit 12 transfers the participant A to the terminal 30B while keeping the avatar of the participant A existing in the virtual space. Toggles control of the avatar. [Selection drawing] Fig. 2

Description

The present disclosure relates to a conference system, control device, control method, program, and recording medium.

Remote meetings have become commonplace, and there are also signs that meetings held in virtual spaces will become more common. Some meetings in the virtual space can be attended not only by using a VR device equipped with an HMD, but also by using a smartphone. VR equipment has the advantage of being able to move avatars in the virtual space more freely, and smartphones have the advantage of being able to participate in meetings regardless of location.

Patent No. 7140890

"Transfer FaceTime calls to other devices", [online], Apple Inc., Internet <URL: https://support.apple.com/ja-jp/guide/iphone/iph74e013a95/ios> "Use Handoff to continue working on other devices", [online], Apple Inc., Internet <URL: https://support.apple.com/ja-jp/HT209455>

It would be more convenient if the device could be switched seamlessly when the user's situation in the real world changed. For example, when a user operates an avatar with an HMD and needs to go out, if the user can switch to operation with a smartphone, there is no need to interrupt the conference.

However, if the avatar disappears from the virtual conference room in the virtual space when switching devices, it is assumed that the communication status of the participant of the disappeared avatar has deteriorated, or that the participant has left the conference, and the conference progresses. There is a problem that it interferes with

Japanese Patent Laid-Open No. 2003-200000 discloses a technique for continuing to view content selected or viewed on a personal computer, for example, on a head mounted display (HMD). If there is content selected in advance or in the middle of viewing, the content is reproduced by wearing the HMD. However, it does not seamlessly switch the devices participating in the conference in the virtual space.

Non-Patent Document 1 is a technique for handing over a video call to another device. Non-Patent Document 2 is a technique for handing over the work of an application to another device. However, none of them hand over the conference in the virtual space to another device.

The present disclosure has been made in view of the above, and aims to seamlessly switch devices used in a conference in virtual space.

A disclosed system according to one aspect of the present disclosure is a conference system that includes a control device and a plurality of terminals and is capable of holding a remote conference in a virtual space, wherein a first terminal and a second terminal among the plurality of terminals is used by the same user, and transmits data for controlling the user's avatar in the virtual space, the control device includes a providing unit for providing a remote conference in the virtual space; a control unit that controls the avatar in the virtual space with data received from one terminal, and the control unit receives data from a second terminal that is logged in with the same account as the first terminal; Control of the avatar is switched to the second terminal while the avatar exists in the virtual space.

According to the present disclosure, it is possible to seamlessly switch devices used in a conference in virtual space.

FIG. 1 is a diagram showing an example of the configuration of a conference system according to this embodiment. FIG. 2 is a diagram illustrating an example of a configuration of a server; FIG. 3 is a diagram illustrating an example of the configuration of a terminal; FIG. 4 is a diagram illustrating an example of the processing flow of the conference system. FIG. 5 is a diagram illustrating an example of the processing flow of the conference system. FIG. 6 is a diagram illustrating an example of the processing flow of the conference system. FIG. 7 is a diagram illustrating an example of the processing flow of the conference system.

[Conference system]
The conference system shown in FIG. 1 is a system in which users can participate in a remote conference held in a virtual space using terminals 30A, 30B, and 30C. The conference system comprises a plurality of terminals 30A, 30B, 30C and a server 10 which are communicably connected via a network. Hereinafter, the terminals will be referred to as terminals 30 when not distinguished from each other.

Accounts are provided to registered users of the conference system. A user can use the conference system by using the terminal 30 to log in to the conference system with an account given to the user. It may be possible to log in with the same account from a plurality of terminals 30 . The login state is managed for each terminal 30 . Unless the user explicitly logs out, the logged-in state of the terminal 30 may be maintained.

An avatar corresponding to each user (participant) is placed in the virtual conference room in the virtual space. An avatar is a computer graphic character representing a participant in a remote conference. Participants use terminals 30 to participate in the conference in the virtual space as avatars. It should be noted that the term "conference" includes chats such as well-end conferences.

In the example of FIG. 1, participant A switches between two terminals 30A and 30B depending on the situation to participate in the remote conference. For example, the terminal 30A is a smart phone, and the terminal 30B is a VR device with an HMD (hereinafter referred to as HMD). Participant C uses terminal 30C to participate in the remote conference. Participant C may also switch between a plurality of terminals to participate in the remote conference. A remote meeting may have three or more participants.

The server 10 provides a virtual conference room in the virtual space, receives voice data and control data from each terminal 30, and distributes them to each terminal 30. FIG.

The terminal 30 collects the voice of the participant with a microphone and transmits voice data and control data for controlling the movement and posture of the avatar. The terminal 30 receives the voice data and control data of other participants, outputs the voice data, controls the avatar according to the control data, and displays the rendered video of the virtual space.

As the terminal 30, software can be installed and used in various types of portable terminals and devices such as VR devices with HMDs, smart phones, tablet terminals, personal computers, television receivers, and game machines. Software distributed from the server 10 may be installed.

[Server configuration]
An example of the configuration of the server 10 will be described with reference to FIG. A server 10 shown in FIG. 2 includes a providing unit 11 and a control unit 12 . Each unit included in the server 10 may be configured by a computer including an arithmetic processing unit, a storage device, and the like, and the processing of each unit may be executed by a program. This program is stored in a storage device included in the server 10, and can be recorded in a non-temporary computer-readable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or can be provided through a network. .

The providing unit 11 provides a remote conference within the virtual space. For example, the providing unit 11 prepares a virtual conference room with an identifier such as a name or number. Upon receiving a request to use the virtual conference room from the terminal 30, the providing unit 11 allocates a virtual conference room to the user (participant) using the terminal 30, and transmits the identifier of the allocated virtual conference room. When receiving the room entry request from the terminal 30, the provision unit 11 registers the participant using the terminal 30 in the virtual conference room indicated by the room entry request, arranges the avatar of the registered participant in the virtual conference room, and remotely Information on the participants of the conference and information necessary to display the virtual conference room are transmitted to the terminal 30 . The entry request may include the virtual meeting room identifier and the user's account.

During the remote conference, the providing unit 11 distributes remote conference data to the terminal 30 . The data of the remote conference is, for example, voice data of the participants received from each terminal 30 of the participants and control data for controlling the avatars of the participants.

The control unit 12 receives voice data of the participants and control data for controlling the avatars of the participants from each terminal 30, and distributes the received control data to each terminal 30 of the participants via the providing unit 11. to control the avatar in the virtual conference room.

When participant A participates in the remote conference using both terminal 30A and terminal 30B, control unit 12 controls the avatar using control data received from either terminal 30A or terminal 30B.

For example, when the terminal 30A logged in with the account of the participant A is participating in the remote conference, if the terminal 30B logs in with the account of the same participant A, the control unit 12 receives control data instead of the control data received from the terminal 30A. , the avatar is controlled by the control data received from the terminal 30B that joined later. That is, the control data received from the terminal 30A is discarded, and the control data received from the terminal 30B is distributed via the providing unit 11. FIG.

Control unit 12 may switch terminals 30A and 30B to be controlled based on switching signals received from terminals 30A and 30B.

When the control unit 12 switches between the terminals 30A and 30B, the control source of the avatar is switched while the avatar exists in the virtual space.

[Device configuration]
An example of the configuration of the terminal 30 will be described with reference to FIG. A terminal 30 shown in FIG. Each unit included in the terminal 30 may be configured by a computer including an arithmetic processing unit, a storage device, and the like, and the processing of each unit may be executed by a program. This program is stored in the storage device of the terminal 30, and can be recorded on a non-temporary computer-readable recording medium such as a magnetic disk, optical disk, or semiconductor memory, or can be provided through a network. .

The input unit 31 inputs voice data of participants collected using a microphone provided in the terminal 30 or a microphone connected to the terminal 30 . The input unit 31 may input a participant's operation or movement from the terminal 30 . For example, if the terminal 30 is an HMD, the HMD is used to input the movement of the participant's head, and the controller is used to input the operation and movement of the participant. If the terminal 30 is a personal computer or smart phone, it inputs an operation to the personal computer or smart phone. The input unit 31 may input an image of the participant captured by a camera.

The control unit 32 generates control data for controlling the avatars of the participants. For example, the control unit 32 may generate control data based on the voice of the participant. Specifically, the control unit 32 generates control data for moving the mouth of the avatar and moving the hands of the avatar while the participant is speaking, and the avatar nods when the participant is not speaking. Generate control data. When the input unit 31 inputs operations and movements of the participants, the control unit 32 may generate control data based on the operations and movements of the participants.

The transmission unit 33 transmits audio data and control data. The transmitter 33 may transmit a signal for controlling the remote conference. Signals for controlling a remote conference include, for example, a request to use a virtual conference room, an entry request to participate in a conference, a switching request to switch terminals, and an exit request to leave the conference. Other signals may be transmitted.

The receiving unit 34 receives voice data and control data from other terminals 30 via the server 10 . The receiver 34 may receive a signal regarding the remote conference. Signals related to the remote conference include, for example, responses to signals for controlling the remote conference, signals notifying entry and exit of participants, and signals related to display of virtual space. Other signals may be received.

The display unit 35 reproduces the received audio data and displays the virtual conference room. When the terminal 30 is a device having a screen such as a smartphone or a personal computer, it displays a virtual conference room captured by a virtual camera in the virtual space. This virtual camera may be operated by the participants, or may be operated by the server according to the state of the conference. The display unit 35 may combine and display virtual conference rooms photographed by a plurality of virtual cameras based on the frame division specified by the server. When the terminal 30 is equipped with an HMD, the display unit 35 displays the virtual conference room from the viewpoint of the avatars of the participants in the virtual space.

[Conference system operation]
Next, an example of the processing flow of the conference system will be described with reference to the sequence diagrams of FIGS. 4 to 7. FIG. FIGS. 4 to 7 show an example of a conference between participants A and C. FIG. Participant A switches between a smart phone and an HMD, and participant C uses a terminal. The server in the figure is the server 10 of this embodiment, and the smartphone, HMD, and terminal are the terminals 30 of this embodiment.

(Operation example 1)
An example in which the terminal used by participant A for the remote conference is switched from the HMD to the smartphone will be described with reference to FIG. 4 .

At step S11, participant A uses a smartphone to participate in the remote conference. For example, Participant A uses a smartphone to log in to the conference system with Participant A's account, designates the number of the virtual conference room, and participates in the remote conference. The virtual meeting room number may be assigned in advance or may be newly assigned in step S11. When participant A enters the virtual conference room, participant A's avatar is placed in the virtual conference room.

At step S12, participant C uses a terminal to participate in the remote conference. For example, participant C uses a terminal to log into the conference system with participant C's account, designates the same virtual conference room number as participant A, and participates in the remote conference. When the participant C enters the virtual conference room, the avatar of the participant C is arranged in the virtual conference room.

When the participants A and C enter the virtual conference room, the avatars of the participants A and C are arranged and the remote conference starts.

During the conference in step S13, voice data of participants A and C and control data of avatars are transmitted and received between the smart phone and the terminal of participant C, and a remote conference is held. Specifically, participant A's smartphone transmits participant A's voice data and participant A's avatar control data, and participant C's terminal transmits participant C's voice data and participant C's avatar. control data. Participant A's smart phone receives participant C's voice data and control data, outputs voice, and controls avatars of participants A and C with the control data. The terminal of participant C receives the voice data and control data of participant A, outputs voice, and controls the avatars of participants A and C with the control data. The voice data and control data of participants A and C are distributed via the server.

The server may deliver remote conference data in addition to avatar control data. For example, the server includes, as remote conference data, information about participants, information about the virtual camera that captures the virtual conference room, information about how frames are divided when displayed on the screen of a smartphone, and information about objects that exist in the virtual conference room. may be delivered.

During the remote meeting, other participants may join in the middle, and participants may leave the virtual meeting room in the middle of the meeting.

In step S14, participant A switches the terminal used for the remote conference from the smartphone to the HMD. For example, participant A switches from the smartphone to the HMD when he wants to give an explanation using gestures or when he wants to participate in the conference in a more immersive manner.

When switching terminals, participant A uses the HMD to participate in the remote conference without leaving the virtual conference room. For example, while participating in the remote conference, participant A puts the smart phone in use on the table and prepares the HMD. At this time, the HMD may be prepared while talking on the smartphone in the hands-free mode. When switching terminals, that is, while preparing the HMD, Participant A's avatar remains in the virtual conference room, and Participant A can listen to the conference audio and speak via the smartphone. .

In step S15, when the preparation of the HMD is completed, participant A uses the HMD to participate in the remote conference. For example, participant A uses the HMD to log in to the conference system with the same account of participant A as the smartphone. Since Participant A's account has already joined the remote conference in the conferencing system, when Participant A logs in with the HMD, the server connects the HMD to the remote conference. Participant A can now participate in the remote conference with the HMD. When Participant A logs in with the HMD, the server may ask Participant A whether to join the already connected remote conference.

In step S16, the server switches the control source of participant A's avatar to the HMD. Specifically, the server distributes the control data received from the HMD as the control data for the participant A's avatar, not the control data received from the smartphone. Similarly, the voice data received from the HMD is distributed as the voice data of the participant A instead of the voice data received from the smart phone.

After switching the control source, during the conference in step S17, the voice data of the participants A and C and the control data of the avatar are transmitted and received between the HMD and the terminal of the participant C, and a remote conference is held.

Even after switching the control source to the HMD, the voice data of the remote conference and the control data of the avatar may be transmitted to the smartphone as well. The server may stop distributing the audio data and the control data to the smartphone after a predetermined period of time (for example, several seconds to several minutes) has elapsed after switching the control source. In other words, the server may distribute voice data and control data of the remote conference to both the previous control source (smartphone) and the current control source (HMD) for a predetermined time after switching the control source.

The smartphone may stop transmitting the audio data and the control data to the server when the distribution of the audio data and the control data from the server is stopped. That is, the transmission of the voice data and the control data from the smart phone is stopped after a predetermined time has elapsed after the participant A switches the terminal.

While the process of FIG. 4 is being performed, the participant A's avatar remains in the virtual conference room, and the participant C is unlikely to notice that the participant A has switched terminals. Since the avatar of the participant A does not disappear from the virtual conference room even when the terminals are switched, the progress of the conference is not hindered. When the terminal is switched, the server notifies the terminal of participant C that the terminal of participant A has been switched, and the terminal of participant C is notified of the sound quality and volume of the voice from the terminal after switching. Interpolate the pre-switching sound and the post-switching sound so that the sound quality and volume of the sound from the previous terminal do not differ greatly, or make the sound quality and volume of the sound after switching closer to the sound quality and volume of the sound before switching. It may be corrected as follows. The terminal may weaken the sound correction step by step, and cancel the sound correction after a predetermined period of time has elapsed. The terminal may interpolate the control of the avatar so that switching terminals is not conspicuous. For example, when the terminal is switched from the smartphone to the HMD, the control data from the HMD is gradually reflected on the avatar without suddenly changing the posture of the avatar. For example, the terminal increases the parts to which the control data is reflected in the order of the avatar's neck, arms, upper body, and lower body. The server may process the voice data and control data of participant A who switched terminals before distributing them.

Participant C may be notified that participant A has switched terminals. For example, when the terminal of participant C receives a notification from the server that participant A has switched terminals, the terminal of participant C displays an icon indicating the type of terminal used where the name of participant A is displayed in the remote conference. It may be displayed or a message may be sent to notify the participant that the terminal has been switched. The terminal may add an effect to the avatar of participant A to notify that the terminal has been switched. The server may add an effect in the virtual space when the participant switches terminals.

An example in which participant A switches the terminal used in the remote conference from the smartphone to the HMD will be described with reference to FIG. 5 .

In step S21, participant A uses an HMD and participant C uses a terminal, and a remote conference is in progress.

In steps S22 and S23, participant A removes the HMD, switches to a smartphone, and uses the smartphone to participate in the remote conference. For example, in the continuation of the processing in FIG. 4, when the smartphone continues to participate in the remote conference, when the smartphone detects the use of participant A, the smartphone starts transmitting voice data and control data. The smart phone may send a request to join the remote conference. Participant A's use of the smartphone can be detected by the sensor of the smartphone when the smartphone is picked up, when voice is input by the microphone of the smartphone, when the sleep state is canceled, and so on.

In step S24, when the server receives the voice data and the control data from the smart phone, it switches the control source of the participant A's avatar to the smart phone. The server may switch the control source to the smartphone after it stops receiving audio data and control data from the HMD.

After switching the control source, voice data of participants A and C and control data of avatars are transmitted and received between the smart phone and the terminal of participant C during the conference in step S25, and a remote conference is held.

While the process of FIG. 5 is being performed, the participant A's avatar is in the virtual conference room.

(Operation example 2)
In Operation Example 1, the server switches control of the avatar to another terminal at the timing of login or data reception from a terminal other than the terminal used in the remote conference. In operation example 2, the transfer of control is notified to another terminal, and the participant responds to the notification with another terminal, thereby switching the control of the avatar to another terminal.

An example in which the terminal used by participant A for the remote conference is switched from the HMD to the smartphone will be described with reference to FIG. 6 .

In step S31, participant A uses an HMD and participant C uses a terminal, and a remote conference is in progress.

At step S32, participant A removes the HMD. Participant A may perform a terminal switching operation on the HMD. For example, participant A operates the controller of the HMD and selects the terminal switching item from the menu.

In step S<b>33 , when the HMD detects that it has been removed or that a switching operation has been performed, it notifies the smartphone of handover of control.

In steps S34 and S35, participant A operates the smartphone to respond to the notification, and uses the smartphone to log in to the conference system with participant A's account. In response to Participant A's response, the smart phone may send a switching signal to the server.

In step S35, the server switches the control source of participant A's avatar to the smart phone.

After switching the control source, voice data of participants A and C and control data of avatars are transmitted and received between the smart phone and the terminal of participant C during the conference in step S36, and a remote conference is held.

While the process of FIG. 6 is being performed, the participant A's avatar is in the virtual conference room.

FIG. 7 is an example of transmitting a handover notification to a smartphone via a server.

In step S41, participant A uses an HMD and participant C uses a terminal, and a remote conference is in progress.

At step S42, participant A removes the HMD. Participant A may perform a terminal switching operation on the HMD.

In step S43, when the HMD detects that it has been removed or that a switching operation has been performed, it transmits a notification of takeover to the server.

In step S44, the server transmits a handover notification to the smartphone. For example, the server stores a terminal used by participant A to log in to the conference system, and transmits a handover notification to that terminal.

In steps S45 and S46, participant A operates the smartphone to respond to the notification, and uses the smartphone to log in to the conference system with participant A's account. In response to Participant A's response, the smart phone may send a switching signal to the server.

In step S47, the server switches the control source of participant A's avatar to the smart phone.

After switching the control source, voice data of participants A and C and control data of avatars are transmitted and received between the smart phone and the terminal of participant C during the conference in step S48, and a remote conference is held.

While the process of FIG. 7 is being performed, the participant A's avatar is in the virtual conference room.

In the above description, the terminal 30 transmits and receives voice data and control data via the server 10, but both or one of the voice data and control data may be directly transmitted and received peer-to-peer (P2P). When the server 10 is not interposed, at least one of the terminals 30 has the function of each part of the server 10 .

As described above, the conference system of the present embodiment is a conference system that includes the server 10 and the plurality of terminals 30 and is capable of holding remote conferences in virtual space. Terminals 30A and 30B are used by participant A, and transmit data for controlling participant A's avatar in the virtual space. The server 10 has a provision unit 11 that provides a remote conference within the virtual space, and a control unit 12 that controls the avatar within the virtual space with data received from the terminal 30 . When the control unit 12 receives data from the terminal 30B logged in with the same account of the participant A as the terminal 30A, the control unit 12 transfers the participant A to the terminal 30B while keeping the avatar of the participant A existing in the virtual space. Toggles control of the avatar. As a result, it is possible to seamlessly switch the devices used in the virtual space meeting without disturbing the meeting.

The conference system of the present embodiment provides a remote conference to both the terminal 30A and the terminal 30B for a predetermined period of time after the control source terminals 30A and 30B are switched. As a result, participant A who switches between terminals 30A and 30B can more reliably switch between terminals.

DESCRIPTION OF SYMBOLS 10... Server 11... Provision part 12... Control part 13... Switching part 30, 30A, 30B, 30C... Terminal 31... Input part 32... Control part 33... Transmission part 34... Reception part 35... Display part

Claims (8)

A conference system comprising a control device and a plurality of terminals and capable of holding a remote conference in a virtual space,
A first terminal and a second terminal among the plurality of terminals are used by the same user and transmit data for controlling the user's avatar in the virtual space;
The control device is
a provider that provides a remote conference in a virtual space;
a control unit that controls the avatar in the virtual space with the data received from the first terminal;
When data is received from a second terminal logged in with the same account as the first terminal, the control unit causes the second terminal to control the avatar while allowing the avatar to exist in the virtual space. Conference system to switch. A control device for controlling an avatar in a virtual space,
a provider that provides a remote conference in a virtual space;
a control unit that controls the avatar in the virtual space with data received from the first terminal;
When data is received from a second terminal logged in with the same account as the first terminal, the control unit causes the second terminal to control the avatar while allowing the avatar to exist in the virtual space. switching controller. The control device according to claim 2,
A control device that provides a remote conference to both the first terminal and the second terminal for a predetermined period of time after switching control of the avatar to the second terminal. The control device according to claim 2,
The control unit switches control of the avatar to the second terminal after receiving a switching signal. The control device according to claim 2,
A control device that, when switching control of the avatar to the second terminal, notifies other terminals participating in the remote conference that the control of the avatar has been switched from the first terminal to the second terminal. A control method for controlling an avatar in a virtual space, comprising:
the computer
Providing remote meetings in virtual space,
controlling the avatar in the virtual space with the data received from the first terminal;
A control method of switching control of the avatar to the second terminal while the avatar exists in the virtual space when data is received from the second terminal logged in with the same account as the first terminal. A program for operating a computer as a control device for controlling an avatar in a virtual space,
to the computer,
a process for providing a remote conference within a virtual space;
a process of controlling the avatar in the virtual space with the data received from the first terminal;
When data is received from a second terminal logged in with the same account as the first terminal, causing the second terminal to execute a process of switching control of the avatar while the avatar remains in the virtual space. program. A recording medium recording a program for operating a computer as a control device for controlling an avatar in a virtual space,
to the computer,
a process for providing a remote conference within a virtual space;
a process of controlling the avatar in the virtual space with the data received from the first terminal;
When data is received from a second terminal logged in with the same account as the first terminal, causing the second terminal to execute a process of switching control of the avatar while the avatar remains in the virtual space. A recording medium on which a program is recorded.

Images