WO2024116763A1

WO2024116763A1 - Management device

Info

Publication number: WO2024116763A1
Application number: PCT/JP2023/040286
Authority: WO
Inventors: 祐貴田中; 圭一村上
Original assignee: 株式会社Ｎｔｔドコモ
Priority date: 2022-12-01
Filing date: 2023-11-08
Publication date: 2024-06-06

Abstract

This management device comprises: a first generation unit that generates first image information indicating a two-dimensional image that is obtained by imaging a conference room in a virtual space by means of a first virtual camera; a second generation unit that generates second image information indicating a three-dimensional image that is obtained by imaging the conference room in the virtual space by means of a second virtual camera; and a communication unit that transmits the first image information to an external device, and transmits the second image information to one or a plurality of terminal devices participating in a second webconference. In the conference room in the virtual space, a virtual object is disposed indicating a two-dimensional image that is output from the external device and that relates to a first webconference.

Description

Management Device

The present invention relates to a management device.

In recent years, systems that realize virtual web conferences in virtual spaces such as the metaverse have come into use, similar to web conferences in the real world.

For example, Patent Document 1 discloses a system in which multiple avatars, which correspond one-to-one with multiple users, can participate in a web conference in a virtual space. In this system, the state of the web conference is distributed to the user terminals used by each of the multiple users.

JP 2020-149400 A

However, in a web conference realized using the technology disclosed in Patent Document 1, all users had to participate in the web conference as avatars in a virtual space. Users who could participate in this web conference were limited to those who subscribed to the virtual space service. Therefore, users who were not subscribed to the virtual space service could not participate in a web conference held in a virtual space. In other words, users of a web conference in real space could not participate in a web conference held in a virtual space.

The present disclosure therefore aims to provide a management device that allows users of a web conference in real space and users of a web conference in virtual space to both participate in the web conference.

A management device according to a preferred embodiment of the present disclosure is a management device that communicates with an external device that manages a first web conference held in a real space and manages a second web conference held in a virtual space, and includes a first generation unit that generates first image information showing a two-dimensional image obtained by capturing an image of a conference room in the virtual space with a first virtual camera, a second generation unit that generates second image information showing a three-dimensional image obtained by capturing an image of a conference room in the virtual space with a second virtual camera, and a communication unit that transmits the first image information to the external device and transmits the second image information to one or more terminal devices participating in the second web conference, and a virtual object showing a two-dimensional image related to the first web conference output from the external device is placed in the conference room in the virtual space.

According to the present disclosure, users of a web conference in real space and users of a web conference in virtual space can both participate in the web conference.

FIG. 1 is a block diagram showing the overall configuration of a conference system 1. FIG. 3 is a block diagram showing a configuration example of a terminal device 30-3. FIG. 3 is a block diagram showing a configuration example of a terminal device 30-1. FIG. 4 is a diagram showing an example of a virtual space VS displayed on a display 38. FIG. 4 is a diagram showing an example of a display image DP. 13A and 13B are diagrams showing a method of generating an image showing a virtual space VS displayed on a terminal device 30-1 and an image showing a virtual space VS displayed on a terminal device 30-3. FIG. 2 is a block diagram showing an example of the configuration of a second conference management device 20. FIG. 2 is a functional block diagram of an avatar control unit 212. FIG. 2 is a block diagram showing an example of the configuration of a first conference management device 10. FIG. 4 is a sequence diagram showing the operation of the conference system 1.

The following describes the conference system 1 according to the embodiment, with reference to Figures 1 to 9.

1: Configuration of the embodiment 1-1: Overall configuration FIG. 1 is a block diagram showing the overall configuration of a conference system 1. As shown in FIG. 1, the conference system 1 includes a first conference management device 10, a second conference management device 20, and terminal devices 30-1 to 30-4. The first conference management device 10 manages a first web conference held in real space. The first conference management device 10 provides a service related to the first web conference to a user who subscribes to the service. In the example shown in FIG. 1, a user U1 and a user U2 subscribe to a service related to the first web conference. More specifically, the first conference management device 10 is a server that provides the first web conference to the terminal device 30-1 and the terminal device 30-2. The first web conference is a conference in which users in remote locations in real space can participate on the web.
The second conference management device 20 manages the second web conference held in a virtual space. The second web conference is held in a virtual space such as a metaverse space. The conference room used in the second web conference is a virtual conference room in the virtual space. The second conference management device 20 provides a service related to the second web conference to a user who subscribes to the service. In the example shown in FIG. 1, user U3 and user U4 subscribe to the service related to the second web conference. More specifically, the second conference management device 20 is a server that provides a web conference in a virtual space to terminal device 30-3 and terminal device 30-4. The second conference management device 20 is an example of a "management device".

The first conference management device 10 and the second conference management device 20 are communicatively connected to each other via the communication network NET. The first conference management device 10 and the terminal device 30-1 and the terminal device 30-2 are communicatively connected to each other. The second conference management device 20 and the terminal device 30-3 and the terminal device 30-4 are communicatively connected to each other.

The terminal device 30-1 is a device through which the user U1 participates in the first web conference in the real space provided by the first conference management device 10. Similarly, the terminal device 30-2 is a device through which the user U2 participates in the first web conference. The users U1 and U2 participate in the first web conference using, for example, an application such as ZOOM or Teams. Note that ZOOM and Teams are registered trademarks. The terminal devices 30-1 and 30-2 may be a PC (Personal Computer), a smartphone, or a tablet. The terminal devices 30-1 and 30-2 display two-dimensional images.

The terminal device 30-3 is a device for user U3 to participate in the second web conference held in the virtual space. Similarly, the terminal device 30-4 is a device for user U4 to participate in the second web conference. The terminal devices 30-3 and 30-4 are, for example, goggle-type head-mounted displays (HMDs). The terminal devices 30-3 and 30-4 display three-dimensional images.

In the example shown in FIG. 1, two terminal devices 30-1 and 30-2 are connected to the first conference management device 10. However, the number of terminal devices connected to the first conference management device 10 is not limited to two and can be any number. Similarly, two terminal devices 30-3 and 30-4 are connected to the second conference management device 20. However, the number of terminal devices connected to the second conference management device 20 is not limited to two and can be any number.

1-2: Configuration of the terminal device Fig. 2A is a block diagram showing a configuration example of the terminal device 30-3. The terminal device 30-3 includes a processing device 31A, a storage device 32A, an input device 33A, a communication device 34, a sound pickup device 35, a speaker 36, a display 38A, and a display 38B. Each element of the terminal device 30 is connected to each other by a single or multiple buses for communicating information.

The processing device 31A is a processor that controls the entire terminal device 30-3. The processing device 31A includes, for example, one or more chips. The processing device 31A also includes, for example, a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, and registers. Some or all of the functions of the processing device 31A may be realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The processing device 31A executes various processes in parallel or sequentially.

The storage device 32A is a recording medium that can be read and written by the processing device 31A. The storage device 32A also stores a number of programs including the control program PR3A executed by the processing device 31A. The storage device 32A functions as a work area for the processing device 31A.

The input device 33A accepts operations from the user U3. For example, the input device 33A includes a touch panel. The input device 33A may also include an imaging device. The imaging device is, for example, a camera. When the input device 33A includes an imaging device, the input device 33A detects a gesture of the user U3 based on an image captured by the imaging device. The input device 33A outputs an operation signal indicating the detected gesture as input information to the processing device 31A.

The communication device 34 is hardware that functions as a transmitting/receiving device for communicating with other devices. The communication device 34 is also called, for example, a network device, a network controller, a network card, or a communication module. The communication device 34 may further include a wireless communication interface. Examples of connectors and interface circuits for wired connections include products that comply with wired LAN, IEEE 1394, and USB. Examples of wireless communication interfaces include products that comply with wireless LAN and Bluetooth (registered trademark), etc.

The sound pickup device 35 includes, for example, a microphone. The sound pickup device 35 picks up the voice of the user U3. The sound pickup device 35 supplies audio data indicating the picked-up voice to the processing device 31A.

The speaker 36 is a device that emits audio. The speaker 36 emits various types of audio under the control of the processing device 31A. For example, audio data, which is a digital signal, is converted into an audio signal, which is an analog signal, by a DA converter (not shown). The amplitude of the audio signal is amplified by an amplifier (not shown). The speaker 36 emits audio represented by the audio signal having the amplified amplitude. In this embodiment, the speaker 36 emits the audio of the web conference.

Displays

38A and 38B are devices that display images.

Displays

38A and 38B display various images under the control of processing device 31A. Display 38A displays an image for the left eye. Display 38B displays an image for the right eye. The image for the left eye and the image for the right eye are different images that take into account the parallax between the left eye and the right eye. These images are generated by processing device 31A based on a three-dimensional image received from second conference management device 20. In this embodiment, displays 38A and 38B display an image of a virtual conference room where the second web conference is held, as described below.

In the above configuration, the processing device 31A reads out the control program PR3A from the storage device 32A. The processing device 31A executes the read out control program PR3A to function as a communication control device 311A and a display control device 312.

The communication control unit 311A causes the second conference management device 20 to transmit to the communication device 34 the input information input by user U3 operating the input device 33A and the audio information acquired from the audio pickup device 35. The communication control unit 311A also causes the communication device 34 to receive conference information for holding the second web conference from the second conference management device 20. The conference information includes image information showing an image of a virtual conference room, image information showing images of the avatars of users U3 and U4, video information showing videos obtained by taking actual photographs of the bodies of users U1 and U2, and audio information of users U1 to U4.

The display control unit 312 uses the conference information received by the communication device 34 by the communication control unit 311A to display a three-dimensional image of a virtual space showing a virtual conference room on the

displays

38A and 38B.

FIG. 3 is a diagram showing an example of a virtual space VS displayed on

displays

38A and 38B. A long table T is installed in the virtual space VS, which serves as a virtual conference room. Two chairs C1 and C2 are arranged on either side of the long table T. Avatar A3 corresponding to user U3 is displayed sitting in chair C1. Similarly, avatar A4 corresponding to user U4 is displayed sitting in chair C2. A panel P is installed on the wall W in front of avatars A3 and A4, on which a live-action image R1 of user U1 and a live-action image R2 of user U2 are displayed. Live-action images R1 and R2 are two-dimensional images serving as virtual objects.

In the virtual conference room shown in FIG. 3, user U3 can participate in the second web conference using avatar A3 corresponding to the user. User U4 can participate in the second web conference using avatar A4 corresponding to the user. Furthermore, each of users U1 and U2 can participate in the second web conference even if they are not subscribed to the service related to the second web conference.

The configuration of terminal device 30-3 has been described with reference to Figures 2A and 3, but terminal device 30-4 has a similar configuration to terminal device 30-3.

FIG. 2B is a block diagram showing an example configuration of terminal device 30-1. In the following, to simplify the explanation, the same reference numerals are used for components common to terminal device 30-1 and terminal device 30-3, and their explanation is omitted. In addition, the following mainly describes the differences between terminal device 30-1 and terminal device 30-3.

The terminal device 30-1 has an input device 33B instead of the input device 33A of the terminal device 30-3. The terminal device 30-1 also has an imaging device 37, unlike the terminal device 30-3. The imaging device 37 is, for example, a camera. The terminal device 30-1 also has a single display 38 instead of the

displays

38A and 38B of the terminal device 30-3. The terminal device 30-1 also has a storage device 32B instead of the storage device 32A of the terminal device 30-3. The storage device 32B stores the control program PR3B instead of the control program PR3A stored in the storage device 32A. The terminal device 30-1 also has a processing device 31B instead of the processing device 31A. The processing device 31B has a communication control device 311B instead of the communication control device 311A of the processing device 31A.

The input device 33B accepts operations from the user U1. For example, the input device 33B includes a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse. Here, if the input device 33B includes a touch panel, it may also serve as the display 38.

The imaging device 37 outputs imaging information indicating an image obtained by imaging the outside world. The imaging device 37 includes, for example, a lens, an imaging element, an amplifier, and an AD converter. Light collected through the lens is converted by the imaging element into an imaging signal, which is an analog signal. The amplifier generates an amplified imaging signal by amplifying the imaging signal. The AD converter converts the amplified imaging signal, which is an analog signal, into imaging data, which is a digital signal. The imaging information including the imaging data is supplied to the processing device 31B.

The display 38 is a device that displays images. The display 38 displays various images under the control of the processing device 31B. The display image DP, which will be described later, is displayed on the display 38. The display image DP is a two-dimensional image.

The communication control unit 311B causes the communication device 34 to receive participation data for participating in the second web conference held in the virtual space from the first conference management device 10. The participation data is originally data received by the first conference management device 10 from the second conference management device 20. The participation data is, for example, text data with a link for participating in the second web conference. User U1 participates in the second web conference by operating the input device 33B and clicking on the link. The terminal device 30-1 can participate in the first web conference under the control of the first conference management device 10, and can also participate in the second web conference under the control of the second conference management device 20.

FIG. 4 is a diagram showing an example of a display image DP that the display control unit 312 of the terminal device 30-1 displays on the display 38. The display image DP includes an image showing the virtual space VS. The image showing the virtual space VS included in the display image DP is an image of the virtual space VS viewed from the opposite direction to the direction in which the virtual space VS is viewed so as to obtain the image shown in FIG. 3. Specifically, in the virtual space VS shown in FIG. 4, the front of the avatar A3 and the avatar A4 are displayed. As a result, the user U1 of the terminal device 30-1 can visually recognize the facial expressions and movements of the avatar A3 and the avatar A4. In addition, a live-action image R1 of the user U1 and a live-action image R2 of the user U2 are displayed in the lower right corner of the display image DP. The display image DP is a two-dimensional image.

FIG. 5 is a diagram showing a method of generating an image showing the virtual space VS displayed on terminal device 30-1, and an image showing the virtual space VS displayed on terminal device 30-3. As shown in FIG. 5, in the virtual space VS, a virtual first camera M1 is placed in front of avatar A3 and avatar A4. Meanwhile, in the virtual space VS, a virtual second camera M2 is placed behind avatar A3 and avatar A4. The first camera M1 is an example of a "first virtual camera." The second camera M2 is an example of a "second virtual camera."

In the virtual space VS, an image generated by the first camera M1 is displayed on the terminal device 30-1 as an image showing the virtual space VS. As shown in FIG. 4, the image generated by the first camera M1 is a two-dimensional image obtained by the first camera M1 capturing an image of the conference room in the virtual space VS.

In addition, in the virtual space VS, an image generated by the second camera M2 is displayed on the terminal device 30-3 as an image showing the virtual space VS. As shown in FIG. 3, the image generated by the second camera M2 is a three-dimensional image obtained by the second camera M2 capturing an image of the conference room in the virtual space VS. Note that the real-life image R1 of user U1 and the real-life image R2 of user U2 included in the three-dimensional image shown in FIG. 3 are two-dimensional images.

The configuration of terminal device 30-1 has been described with reference to Figures 2B, 4, and 5, but terminal device 30-2 has the same configuration as terminal device 30-1.

6 is a block diagram showing a configuration example of the second conference management device 20. The second conference management device 20 includes a processing device 21, a storage device 22, an input device 23, a communication device 24, and a display 25. The elements of the second conference management device 20 are connected to each other by one or more buses for communicating information.

The processing device 21 is a processor that controls the entire second conference management device 20. The processing device 21 includes, for example, one or more chips. The processing device 21 also includes, for example, a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, and registers. Some or all of the functions of the processing device 21 may be realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The processing device 21 executes various processes in parallel or sequentially.

The storage device 22 is a recording medium that can be read from and written to by the processing device 21. The storage device 22 also stores a plurality of programs including a control program PR2 that the processing device 21 executes.
The storage device 22 also stores an image information database PD. The image information database PD stores image information for generating an image showing the virtual space VS. The image information includes image information showing an image of a virtual conference room in the virtual space VS, image information showing an image of the avatar A3, and image information showing an image of the avatar A4. The image of the virtual conference room includes a three-dimensional image of the long table T, a three-dimensional image of the chairs C1 and C2, a three-dimensional image of the wall W, and a three-dimensional image of the panel P. The image information database PD also stores placement location information showing the placement locations of the three-dimensional image of the long table T, the three-dimensional image of the chairs C1 and C2, the three-dimensional image of the wall W, and the three-dimensional image of the panel P in the virtual conference room in the virtual space VS shown in FIG. 3. The storage device 22 functions as a work area for the processing device 21.

The input device 23 accepts operations from the administrator of the second conference management device 20. For example, the input device 23 includes a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse.

The communication device 24 is hardware that functions as a transmitting/receiving device for communicating with other devices. The communication device 24 is also called, for example, a network device, a network controller, a network card, or a communication module. The communication device 24 may also be equipped with a wireless communication interface. Examples of connectors and interface circuits for wired connections include products that comply with wired LAN, IEEE 1394, and USB. Examples of wireless communication interfaces include products that comply with wireless LAN and Bluetooth (registered trademark), etc.

The display 25 is a device that displays images. The display 25 displays various images under the control of the processing device 21.

In the above configuration, the processing device 21 reads out the control program PR2 from the storage device 22. By executing the read out control program PR2, the processing device 21 functions as a space generation unit 211, an avatar control unit 212, a movement control unit 213, a first image generation unit 214, a first audio acquisition unit 215, a second image generation unit 216, a second audio acquisition unit 217, and a communication control unit 218.

The space generation unit 211 generates a three-dimensional image showing a virtual conference room in the virtual space VS. The three-dimensional image includes a three-dimensional image of the long table T, a three-dimensional image of chairs C1 and C2, a three-dimensional image of the wall W, and a three-dimensional image of the panel P. The space generation unit 211 generates a three-dimensional image showing the virtual conference room using the image information and placement information of these three-dimensional images stored in the image information database PD.

The avatar control unit 212 displays avatar A3 and avatar A4 in a virtual conference room in the virtual space VS. FIG. 7 is a functional block diagram of the avatar control unit 212. The avatar control unit 212 includes an avatar generation unit 212-1 and an action control unit 212-2.

Avatar generation unit 212-1 generates a three-dimensional image of avatar A3 using image information corresponding to avatar A3 stored in image information database PD. Similarly, avatar generation unit 212-1 generates a three-dimensional image of avatar A4 using image information corresponding to avatar A4 stored in image information database PD. Furthermore, avatar generation unit 212-1 displays the three-dimensional images of avatar A3 and avatar A4 in virtual space VS.

The motion control unit 212-2 controls the motion of avatar A3 and avatar A4. As an example, the motion control unit 212-2 controls the motion of avatar A3 based on the operation of user U3 on terminal device 30-3. Similarly, the motion control unit 212-2 controls the motion of avatar A4 based on the operation of user U4 on terminal device 30-4.

The movement control unit 213 moves at least one of the first camera M1 and the second camera M2 in the virtual space VS. More specifically, the movement control unit 213 controls the placement locations of the first camera M1 and the second camera M2 in a virtual conference room in the virtual space VS, and the movement direction and movement speed of each of the first camera M1 and the second camera M2. The movement control unit 213 also controls the imaging direction of the first camera M1 and the second camera M2. Furthermore, the movement control unit 213 controls the zoom function of the first camera M1 and the second camera M2, i.e., the magnification ratio of the captured image.

The first image generating unit 214 generates image information showing a two-dimensional image obtained by the first camera M1 capturing an image of the conference room in the virtual space VS. The first image generating unit 214 is an example of a "first generating unit." The image information generated by the first image generating unit 214 is an example of "first image information." As described above, the display image DP is displayed on the display 38 provided on the terminal device 30-1 and the terminal device 30-2. In other words, the first image generating unit 214 generates image information showing a two-dimensional image of the virtual space VS included in the display image DP. As described above, the image information shows a two-dimensional image obtained by the first camera M1 capturing an image of the virtual space VS. For this reason, the image information as the "first image information" does not include the live-action image R1 of the user U1 and the live-action image R2 of the user U2 shown in FIG. 3.

The first audio acquisition unit 215 acquires audio in the first web conference held in real space. Furthermore, the "audio in the first web conference held in real space" is an example of the "first audio." Furthermore, the first audio acquisition unit 215 is an example of the "first acquisition unit."

When user U1 participates in the first web conference using terminal device 30-1, the sound pickup device 35 of terminal device 30-1 picks up the voice of user U1. The sound pickup device 35 also supplies the picked up voice of user U1 to the processing device 31B. The processing device 31B transmits voice information indicating the voice of user U1 to the first conference management device 10 via the communication device 34. Similarly, the processing device 31B of terminal device 30-2 transmits voice information indicating the voice of user U2 to the first conference management device 10 via the communication device 34. The first conference management device 10 transmits voice information indicating the voice of user U1 and voice information indicating the voice of user U2 to the second conference management device 20 via the communication network NET, as described below. The first voice acquisition unit 215 acquires voice information indicating the voice of user U1 and voice information indicating the voice of user U2 via the communication device 24.

The second image generating unit 216 generates image information showing a three-dimensional image obtained by capturing an image of the conference room in the virtual space VS with the second camera M2. The second image generating unit 216 is an example of a "second generating unit." The image information generated by the second image generating unit 216 is an example of "second image information."

As shown in FIG. 3, the above three-dimensional image includes a live-action image R1 of user U1 and a live-action image R2 of user U2. That is, the image information as the "second image information" includes the live-action image R1 of user U1 and the live-action image R2 of user U2 shown in FIG. 3. These live-action images R1 and R2 are examples of "virtual objects showing two-dimensional images related to the first web conference."

As described above, a three-dimensional image showing the virtual space VS illustrated in FIG. 3 is displayed on the

displays

38A and 38B of the terminal devices 30-3 and 30-4. The second image generator 216 generates image information showing the three-dimensional image.

The second audio acquisition unit 217 acquires audio in the second web conference held in the virtual space VS. Furthermore, the "audio in the second web conference held in the virtual space VS" is an example of the "second audio." Furthermore, the second audio acquisition unit 217 is an example of the "second acquisition unit."

When user U3 participates in the second web conference using terminal device 30-3, the sound pickup device 35 of terminal device 30-3 picks up the voice of user U3. The sound pickup device 35 also supplies voice information indicating the picked-up voice of user U3 to the processing device 31A. The processing device 31A transmits the voice information indicating the voice of user U3 to the second conference management device 20 via the communication device 34. Similarly, the processing device 31A of terminal device 30-4 transmits voice information indicating the voice of user U4 to the second conference management device 20 via the communication device 34. The second voice acquisition unit 217 acquires the voice information indicating the voice of user U3 and the voice information indicating the voice of user U4 via the communication device 24.

The communication control unit 218 causes the communication device 24 to transmit the first image information generated by the first image generation unit 214 to the first conference management device 10 as an external device. The communication control unit 218 also causes the communication device 24 to receive first imaging information indicating the live-action image R1 from the terminal device 30-1. The communication control unit 218 also causes the communication device 24 to receive second imaging information indicating the live-action image R2 from the terminal device 30-2. The communication control unit 218 also causes the communication device 24 to transmit the second image information generated by the second image generation unit 216 to the terminal devices 30-3 and 30-4. The communication control unit 218 also causes the communication device 24 to transmit first audio information indicating the first audio to the terminal devices 30-3 and 30-4. The communication control unit 218 also causes the communication device 24 to transmit second audio information indicating the second audio to the first conference management device 10 as an external device. The communication control unit 218 is an example of a "communication unit."

8 is a block diagram showing a configuration example of the first conference management device 10. The first conference management device 10 includes a processing device 11, a storage device 12, an input device 13, a communication device 14, and a display 15. The elements of the first conference management device 10 are connected to each other by one or more buses for communicating information.

The processing device 11 is a processor that controls the entire first conference management device 10. The processing device 11 includes, for example, one or more chips. The processing device 11 also includes, for example, a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, and registers. Some or all of the functions of the processing device 11 may be realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The processing device 11 executes various processes in parallel or sequentially.

The storage device 12 is a recording medium that can be read from and written to by the processing device 11. The storage device 12 also stores a plurality of programs including a control program PR1 that the processing device 11 executes.
The storage device 12 also stores application data AD. The application data AD is data necessary for the terminal device 30-1 and the terminal device 30-2 to use an application for conducting a web conference, such as ZOOM or Teams.
The storage device 12 functions as a work area for the processing device 11 .

The input device 13 accepts operations from the administrator of the first conference management device 10. For example, the input device 13 includes a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse. Here, if the input device 13 includes a touch panel, it may also serve as the display 15.

The communication device 14 is hardware that functions as a transmitting/receiving device for communicating with other devices. The communication device 14 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 14 may also be equipped with a wireless communication interface. Examples of connectors and interface circuits for wired connections include products that comply with wired LAN, IEEE 1394, and USB. Examples of wireless communication interfaces include products that comply with wireless LAN and Bluetooth (registered trademark), etc.

The display 15 is a device that displays images. The display 15 displays various images under the control of the processing device 11.

In the above configuration, the processing device 11 reads the control program PR1 from the storage device 12. The processing device 11 functions as a communication control unit 111 by executing the read control program PR1.

The communication control unit 111 causes the communication device 14 to receive participation data for participating in the second web conference held in the virtual space from the second conference management device 20. The communication control unit 111 also causes the communication device 14 to transmit the participation data received from the second conference management device 20 to the terminal device 30-1 and terminal device 30-2.

The communication control unit 111 also causes the communication device 14 to receive the first image information from the second conference management device 20. The communication control unit 111 also causes the terminal devices 30-1 and 30-2 to transmit the first image information received from the second conference management device 20 to the communication device 14. The communication control unit 111 also causes the communication device 14 to receive the first audio information from the terminal devices 30-1 and 30-2. The communication control unit 111 also causes the second conference management device 20 to transmit the first audio information received from the terminal devices 30-3 and 30-4 to the communication device 14. The communication control unit 111 also causes the communication device 14 to receive the second audio information from the second conference management device 20. The communication control unit 111 also causes the communication device 14 to transmit the second audio information received from the second conference management device 20 to the communication device 14.

2: Operation of the embodiment FIG. 9 is a sequence diagram showing the operation of the conference system 1 according to the embodiment.

In step S1, the processing device 21 provided in the second conference management device 20 functions as a space generation unit 211. The processing device 21 generates a three-dimensional image showing a virtual space VS. The virtual space VS includes a virtual conference room. The three-dimensional image showing the virtual conference room also includes a three-dimensional image showing a long table T, a three-dimensional image showing chairs C1 and C2, a three-dimensional image showing a wall W, and a three-dimensional image showing a panel P.

In step S2, the processing device 21 included in the second conference management device 20 functions as the avatar control unit 212. The processing device 21 generates a three-dimensional image showing avatar A3 and avatar A4.

In step S3, the processing device 21 included in the second conference management device 20 functions as the first image generation unit 214. The processing device 21 generates the first image information PI1.

In step S4, the processing device 21 provided in the second conference management device 20 functions as a communication control unit 218. The processing device 21 causes the first conference management device 10 to transmit the first image information PI1 to the communication device 24. The processing device 11 provided in the first conference management device 10 functions as a communication control unit 111. The processing device 11 causes the communication device 14 to receive the first image information PI1 from the second conference management device 20.

In step S5, the processing device 11 provided in the first conference management device 10 functions as a communication control unit 111. The processing device 11 causes the terminal device 30-1 to transmit the first image information PI1 to the communication device 14. The processing device 31B provided in the terminal device 30-1 functions as a communication control unit 311B. The processing device 31B causes the communication device 34 to receive the first image information PI1 from the first conference management device 10.

In step S6, the processing device 31B provided in the terminal device 30-1 functions as a communication control unit 311B. The processing device 31B causes the first conference management device 10 to transmit the first imaging information RI1 to the communication device 34. The processing device 11 provided in the first conference management device 10 functions as a communication control unit 111. The processing device 11 causes the communication device 14 to receive the first imaging information RI1 from the terminal device 30-1.

In step S7, the processing device 11 provided in the first conference management device 10 functions as a communication control unit 111. The processing device 11 causes the second conference management device 20 to transmit the first imaging information RI1 to the communication device 14. The processing device 21 provided in the second conference management device 20 functions as a communication control unit 218. The processing device 21 causes the communication device 24 to receive the first imaging information RI1 from the first conference management device 10.

In step S8, the processing device 21 included in the second conference management device 20 functions as the second image generation unit 216. The processing device 21 generates the second image information PI2.

In step S9, the processing device 21 provided in the second conference management device 20 functions as the communication control unit 218. The processing device 21 causes the terminal device 30-3 to transmit the second image information PI2 to the communication device 24. The processing device 31A provided in the terminal device 30-3 functions as the communication control unit 311A. The processing device 31A causes the communication device 34 to receive the second image information PI2 from the second conference management device 20.

In step S10, the processing device 31B provided in the terminal device 30-1 functions as a communication control unit 311B. The processing device 31B causes the first conference management device 10 to transmit first voice information SI1 indicating the first voice to the communication device 34. The processing device 11 provided in the first conference management device 10 functions as a communication control unit 111. The processing device 11 causes the communication device 14 to receive the first voice information SI1 from the terminal device 30-1.

In step S11, the processing device 11 provided in the first conference management device 10 functions as a communication control unit 111. The processing device 11 causes the second conference management device 20 to transmit the first audio information SI1 to the communication device 14. The processing device 21 provided in the second conference management device 20 functions as a communication control unit 218. The processing device 21 causes the communication device 24 to receive the first audio information SI1 from the first conference management device 10.

In step S12, the processing device 31A provided in the terminal device 30-3 functions as a communication control unit 311A. The processing device 31A causes the second conference management device 20 to transmit second voice information SI2 indicating the second voice to the communication device 34. The processing device 21 provided in the second conference management device 20 functions as a communication control unit 218. The processing device 21 causes the communication device 24 to receive the second voice information SI2 from the terminal device 30-3.

In step S13, the processing device 21 provided in the second conference management device 20 functions as the communication control unit 218. The processing device 21 causes the terminal device 30-3 to transmit the first audio information SI1 to the communication device 24. The processing device 31A provided in the terminal device 30-3 functions as the communication control unit 311A. The processing device 31A causes the communication device 34 to receive the first audio information SI1 from the second conference management device 20.

In step S14, the processing device 21 provided in the second conference management device 20 functions as a communication control unit 218. The processing device 21 causes the first conference management device 10 to transmit the second audio information SI2 to the communication device 24. The processing device 11 provided in the first conference management device 10 functions as a communication control unit 111. The processing device 11 causes the communication device 14 to receive the second audio information SI2 from the second conference management device 20.

In step S15, the processing device 11 provided in the first conference management device 10 functions as a communication control unit 111. The processing device 11 causes the terminal device 30-1 to transmit the second audio information SI2 to the communication device 14. The processing device 31B provided in the terminal device 30-1 functions as a communication control unit 311B. The processing device 31B causes the communication device 34 to receive the second audio information SI2 from the first conference management device 10.

Note that terminal device 30-2 performs the same operations as terminal device 30-1. Terminal device 30-4 performs the same operations as terminal device 30-3. Furthermore, the order of steps S1 to S15 above may be changed as necessary.

3: Effects of the embodiment According to the above description, the second conference management device 20 communicates with the first conference management device 10 as an external device that manages the first web conference held in the real space, and manages the second web conference held in the virtual space VS. The second conference management device 20 includes a first image generation unit 214 as a first generation unit, a second image generation unit 216 as a second generation unit, and a communication control unit 218 as a communication unit. The first image generation unit 214 generates first image information PI1 indicating a two-dimensional image obtained by capturing an image of a conference room in the virtual space VS with a first camera M1 as a first virtual camera. The second image generation unit 216 generates second image information indicating a three-dimensional image obtained by capturing an image of a conference room in the virtual space VS in which a virtual object indicating a two-dimensional image related to the first web conference output from the first conference management device 10 as an external device is arranged with a second camera M2 as a second virtual camera. The communication control unit 218 transmits the first image information PI1 to the first conference management device 10 serving as an external device, and transmits the second image information PI2 to one or more terminal devices 30-3 to 30-4 participating in the second Web conference.

The second conference management device 20 has the above configuration, so that both users of a web conference in real space and users of a web conference in virtual space VS can participate in a web conference. For example, when a user of a web conference in virtual space VS participates in the web conference using an avatar, conventionally, a user who has not yet generated an avatar could not participate in the web conference. With the second conference management device 20 according to the embodiment, a user who has not yet generated an avatar can participate in a web conference in virtual space VS in which an avatar is used, as a user of a web conference in real space.

Furthermore, according to the above description, the second conference management device 20 includes a first voice acquisition unit 215 as a first acquisition unit, and a second voice acquisition unit 217 as a second acquisition unit. The first voice acquisition unit 215 acquires the first voice in the first web conference. The second voice acquisition unit 217 acquires the second voice in the second web conference. The communication control unit 218 transmits the first voice to one or more terminal devices 30-1 to 30-2, and transmits the second voice to the first conference management device 10 as an external device.

The second conference management device 20 has the above configuration, so that users of a web conference in real space and users of a web conference in virtual space VS can converse using voice.

Furthermore, according to the above description, the second conference management device 20 includes a movement control unit 213. The movement control unit 213 moves at least one of the first camera M1 as the first virtual camera and the second camera M2 as the second virtual camera in the virtual space VS.

Since the second conference management device 20 has the above configuration, images of the web conference from any viewpoint and any angle are displayed on the terminal device 30-1 used by user U1 of the web conference in the real space, and on the terminal device 30-2 used by user U2 of the web conference in the real space. Similarly, images of the web conference from any viewpoint and any angle are displayed on the terminal device 30-3 used by user U3 of the web conference in the virtual space VS, and on the terminal device 30-4 used by user U4 of the web conference in the virtual space VS.

Furthermore, according to the above explanation, the second conference management device 20 includes an avatar control unit 212. The avatar control unit 212 displays one or more avatars A3 to A4 in one-to-one correspondence with users U3 to U4 of one or more terminal devices 30-3 to 30-4 in the conference room in the virtual space VS.

Since the second conference management device 20 has the above configuration, user U3 of the web conference in the virtual space VS can participate in the web conference using avatar A3 corresponding to himself. Similarly, user U4 of the web conference in the virtual space VS can participate in the web conference using avatar A3 corresponding to himself.

4: Modifications The present disclosure is not limited to the above-described embodiments. Specific modifications are exemplified below. Two or more of the following examples may be combined.

4-1: Modification 1
In the above embodiment, the control program PR2 is stored in the storage device 22 of the second conference management device 20, but the control program PR2 may be manufactured or sold separately. The control program PR2 may be provided to a purchaser, for example, by distributing a computer-readable recording medium such as a flash ROM on which the control program PR2 is written, or by distributing the control program PR2 by downloading it via a telecommunications line.

4-2: Modification 2
In the above embodiment, the space generation unit 211, the avatar control unit 212, the movement control unit 213, the first image generation unit 214, the first voice acquisition unit 215, the second image generation unit 216, the second voice acquisition unit 217, and the communication control unit 218 are all software modules. However, any one, any two, or all of the space generation unit 211, the avatar control unit 212, the movement control unit 213, the first image generation unit 214, the first voice acquisition unit 215, the second image generation unit 216, the second voice acquisition unit 217, and the communication control unit 218 may be hardware modules. The hardware modules are, for example, DSPs (Digital Signal Processors), ASICs (Application Specific Integrated Circuits), PLDs (Programmable Logic Devices), and FPGAs (Field Programmable Gate Arrays). Even if any one, any two, or all of the space generation unit 211, the avatar control unit 212, the movement control unit 213, the first image generation unit 214, the first audio acquisition unit 215, the second image generation unit 216, the second audio acquisition unit 217, and the communication control unit 218 are hardware modules, the same effect as in the above embodiment can be achieved.

5: Others (1) In the above-described embodiment, storage device 12, storage device 22, storage device 32A, and storage device 32B are exemplified by ROM and RAM, but the storage devices may be flexible disks, magneto-optical disks (e.g., compact disks, digital versatile disks, Blu-ray (registered trademark) disks), smart cards, flash memory devices (e.g., cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppy (registered trademark) disks, magnetic strips, databases, servers, or other suitable storage media.

(2) In the above-described embodiments, the information, signals, etc. described may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

(3) In the above-described embodiment, the input and output information, etc. may be stored in a specific location (e.g., memory) or may be managed using a management table. The input and output information, etc. may be overwritten, updated, or added to. The output information, etc. may be deleted. The input information, etc. may be transmitted to another device.

(4) In the above-described embodiment, the determination may be made based on a value (0 or 1) represented using one bit, a Boolean value (true or false), or a comparison of numerical values (e.g., a comparison with a predetermined value).

(5) The order of the process steps, sequences, flow charts, etc. illustrated in the above-described embodiments may be changed as long as it is not inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order and are not limited to the particular order presented.

(6) Each function illustrated in Figures 1 to 9 is realized by any combination of at least one of hardware and software. Furthermore, there are no particular limitations on the method of realizing each functional block. That is, each functional block may be realized using one device that is physically or logically coupled, or may be realized using two or more devices that are physically or logically separated and connected directly or indirectly (e.g., using wires, wirelessly, etc.) and these multiple devices. A functional block may be realized by combining the one device or the multiple devices with software.

(7) The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

Software, instructions, information, etc. may also be transmitted and received via a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using at least one of wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and/or wireless technologies (such as infrared, microwave), then at least one of these wired and wireless technologies is included within the definition of a transmission medium.

(8) In each of the above forms, the terms "system" and "network" are used interchangeably.

(9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, may be expressed using relative values from a predetermined value, or may be expressed using other corresponding information.

(10) In the above-described embodiment, the first conference management device 10, the second conference management device 20, and the terminal devices 30-1 to 30-4 may be mobile stations (MS). A mobile station may also be referred to by those skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other suitable term. In addition, in this disclosure, the terms "mobile station", "user terminal", "user equipment (UE)", "terminal", etc. may be used interchangeably.

(11) In the above-mentioned embodiments, the terms "connected" and "coupled" or any variation thereof refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, "connected" may be read with "access". As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and light (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

(12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

(13) The terms "determining" and "determining" as used in this disclosure may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), and considering ascertaining as "judging" or "determining". Also, "determining" and "determining" may include considering receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, and accessing (e.g., accessing data in memory) as "judging" or "determining". Additionally, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to have been "judged" or "decided." In other words, "judgment" and "decision" can include considering some action to have been "judged" or "decided." Additionally, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

(14) In the above embodiments, when the terms "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive or.

(15) In this disclosure, where articles have been added by translation, such as a, an, and the in English, this disclosure may include that the noun following these articles is in the plural.

(16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." In addition, the term may mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

(17) Each aspect/embodiment described in this disclosure may be used alone, in combination, or switched depending on the execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being an explicit notification, but may be performed implicitly (e.g., not notifying the specific information).

Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended as an illustrative example and does not have any limiting meaning on the present disclosure.

1...conference system, 10...first conference management device, 11...processing device, 12...storage device, 13...input device, 14...communication device, 15...display, 20...second conference management device, 21...processing device, 22...storage device, 23...input device, 24...communication device, 25...display, 30...terminal device, 31A, 31B...processing device, 32A, 32B...storage device, 33...input device, 34...communication device, 35...sound collection device, 36...speaker, 37...imaging device, 38, 38A, 38B...display, 111...communication control unit, 211...space generation unit, 212...avatar control unit, 212-1... Avatar generation unit, 212-2...motion control unit, 213...movement control unit, 214...first image generation unit, 215...first voice acquisition unit, 216...second image generation unit, 217...second voice acquisition unit, 218...communication control unit, 311...communication control unit, 312...display control unit, A3, A4...avatar, C1, C2...chair, M1...first camera, M2...second camera, PI1...first image information, PI2...second image information, PR1, PR2, PR3...control program, R1, R2...real image, RI1...first imaging information, SI1...first voice information, SI2...second voice information, U1, U2, U3, U4...user

Claims

A management device that communicates with an external device that manages a first Web conference held in a real space and manages a second Web conference held in a virtual space,
a first generation unit that generates first image information indicating a two-dimensional image obtained by capturing an image of the conference room in the virtual space with a first virtual camera;
a second generation unit that generates second image information indicating a three-dimensional image obtained by capturing an image of the conference room in the virtual space with a second virtual camera;
a communication unit that transmits the first image information to the external device and transmits the second image information to one or more terminal devices participating in the second web conference;
Equipped with
A virtual object showing a two-dimensional image related to the first web conference output from the external device is placed in the conference room in the virtual space.
Management device.
a first acquisition unit that acquires a first voice in the first web conference;
A second acquisition unit that acquires a second voice in the second web conference,
The communication unit transmits the first sound to the one or more terminal devices and transmits the second sound to the external device.
The management device according to claim 1 .
a movement control unit that moves the first virtual camera, the second virtual camera, or both the first virtual camera and the second virtual camera in the virtual space,
The management device according to claim 1 .
and an avatar control unit that displays, in a conference room in the virtual space, one or more avatars that correspond one-to-one to users of the one or more terminal devices.
The management device according to claim 1 .