JP2024052610A - COMMUNICATION MANAGEMENT SERVER, COMMUNICATION SYSTEM, COMMUNICATION MANAGEMENT METHOD, AND PROGRAM - Google Patents

Abstract

[Problem] The present disclosure aims to make it easier for people participating in an event remotely to communicate with people at the event. [Solution] A communication management server 3 of the present disclosure establishes voice communication between a robot R1 capable of inputting and outputting voice in real space and capable of autonomously moving to a position indicated by real space position information and a student terminal in the real space (S32), creates a virtual space image showing an icon related to a person in the real space and an icon related to the robot R1 in the real space (S51), transmits the virtual space image to the student terminal 9f (S52), receives changed virtual space position information based on the movement of the icon related to the robot R1 relative to the virtual space image from the student terminal 9f (S54), and transmits changed real space position information corresponding to the changed virtual space position information to the robot R1 (S55). [Selected Figure] Fig. 21

Description

This disclosure relates to a communication management server, a communication system, a communication management method, and a program.

In recent years, with the spread of remote work, it has become common to conduct distance learning by connecting the school's communication terminal with each student's communication terminal. Furthermore, due to measures against virus infection and the promotion of new ways of working, there is a growing need to be able to participate in distance learning from home, etc. using a communication terminal, in addition to actually attending school.

It has also been proposed that students attending classes and students participating in remote classes can communicate with each other via a communicative robot that exists in the real world, such as a classroom (see Patent Document 1).

However, participants in events such as distance learning face the problem that mobile devices such as robots are not necessarily located near the people they want to communicate with, making it difficult to communicate with them.

This disclosure was made in consideration of the above-mentioned problems, and aims to make it easier for people participating in an event remotely to communicate with those on-site.

The invention according to claim 1 is a communication management server having a communication establishment unit that establishes voice communication between a mobile device capable of inputting and outputting voice in real space and capable of autonomously moving to a position indicated by real space position information and a specified communication terminal in the real space, an image creation unit that creates a virtual space image showing an icon related to a person in the real space and an icon related to the mobile device in the real space, a transmission unit that transmits the virtual space image to the specified communication terminal, and a reception unit that receives changed virtual space position information based on the movement of the icon related to the mobile device relative to the virtual space image from the specified communication terminal, and the transmission unit transmits changed real space position information corresponding to the changed virtual space position information to the mobile device.

This disclosure makes it easier for people participating in an event remotely to communicate with people on-site.

1 is an overall configuration diagram of a communication system according to an embodiment of the present invention; This is a diagram showing the situation of a classroom in real space. FIG. 2 is a hardware configuration diagram of the communication management system. FIG. 2 is a diagram illustrating the hardware configuration of a teacher terminal, a robot terminal, and a student terminal. FIG. 2 is a hardware configuration diagram of the earphones with a microphone. FIG. 2 is a diagram illustrating a hardware configuration of the wide-angle imaging device. FIG. 1 is a diagram illustrating an example of an image of a wide-angle imaging device in use. 1A and 1B are diagrams illustrating an outline of a process for creating an equirectangular projection image and a celestial sphere image from an image captured by a wide-angle imaging device. 1A and 1B are diagrams illustrating an outline of a process for creating an equirectangular projection image and a celestial sphere image from an image captured by a wide-angle imaging device. FIG. 13 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere image is a three-dimensional solid sphere. FIG. 11 is a diagram showing an example of a three-dimensional perspective view of FIG. 10 . 11 is a diagram illustrating an example of a relationship between predetermined area information and an image of a predetermined area. FIG. 2 is a hardware configuration diagram of a vehicle device. FIG. 2 is a diagram illustrating the functional configuration of the communication system. FIG. 4 is a conceptual diagram of a location information management table. FIG. 11 is a sequence diagram showing a login process of each terminal and a robot. 11 is a sequence showing a process of starting audio communication and audio-video communication, and a process of registering attendance. FIG. 13 is a diagram showing an image of student attendance registration. 11 is a sequence showing a process for displaying a virtual space image. A figure showing icons indicating the current positions of the teacher and robot, icons indicating the initial positions of each student, and a virtual space image including video from the robot. FIG. 13 is a diagram showing a virtual space image after the robot icon has been moved. FIG. 13 shows the situation of the classroom in the real space after the robot has moved. FIG. 13 shows an image of a remote student displayed on a robot terminal. FIG. 11 is a sequence diagram showing a process for performing dedicated voice communication. FIG. 13 is a diagram showing a virtual space image in a state where the icon of a party with which dedicated voice communication is to be performed is selected. FIG. 2 is a diagram illustrating a hardware configuration of a head mounted display.

[Outline of the communication system]
An outline of a communication system 1 will be described with reference to Fig. 1. Fig. 1 is a diagram showing the overall configuration of a communication system according to this embodiment.

The communication system 1 is constructed from a communication management server 3, a teacher terminal 5, a microphone-equipped earphone 6, a student terminal 9, and a telepresence robot (hereafter referred to as the "robot") R.

The robot R is mainly composed of a robot terminal 7, a wide-angle imaging device 8, and a vehicle device 10 that moves the robot R. In this embodiment, the robot R moves autonomously, but it does not have to move autonomously. The robot R is an example of a moving device, and moving devices also include those that move in the air, such as drones, and those that move underwater, such as submarine-type radio-controlled vehicles. The vehicle device 10 is an example of a propulsion device. The robot R is equipped with environmental sensors such as a temperature sensor, a humidity sensor, an oxygen sensor, or a carbon dioxide sensor, and is also equipped with a lighting device that illuminates the area around the robot R.

The communication management server 3, teacher terminal 5, student terminal 9, and robot terminal 7 of robot R can communicate via a communication network 100 such as the Internet. The communication form may be wired or wireless. In FIG. 1, the teacher terminal 5, student terminal 9, and robot terminal 7 are shown to communicate wirelessly. In addition, the microphone-equipped earphones 6 can communicate in close range by pairing with the teacher terminal 5. The communication management server 3 may be constructed of multiple servers.

Note that the teacher terminal 5 and the student terminal 9 are examples of communication terminals.

[Real space situation]
Next, the situation of a classroom as a site in real space (real world) will be described with reference to FIG. 2. FIG. 2 is a diagram showing the situation of a classroom in real space. In FIG. 2, in a classroom in real space, four students S1 to S4 are seated at their respective desks D1 to D4, which have been determined in advance, and a teacher T1 is teaching a class. Since students S5 and S6 are not attending the class in the classroom, but are participating in the remote class from a remote location such as their home, the classroom has desks D5 and D6 for students S5 and S6, but students S5 and S6 are not present. Note that teachers T1 and T2 are examples of "teacher T", students S1 to S6 are examples of "student S", desks D1 to D6 are examples of "desk D", and robots R1 and R2 are examples of "robot R". The teacher terminals 5a and 5b are an example of a "teacher terminal 5", the earphones with microphone 6a and 6b are an example of an "earphones with microphone 6", the robot terminals 7a and 7b are an example of a "robot terminal 7", and the student terminals 9e and 9f are an example of a "student terminal 9". Furthermore, the wide-angle imaging devices 8a and 8b are an example of a "wide-angle imaging device 8", and the vehicle devices 10a and 10b are an example of a "vehicle device 10".

Although FIG. 2 illustrates a classroom, the present invention is not limited to this and may be an exhibition or seminar venue. The number of students, desks, and teachers are also not limited to these. A class is an example of an "event," and events also include exhibitions, lectures, and the like.

Furthermore, as shown in Figure 2, in the classroom, robot R1 is positioned in front of the podium as its initial position and takes pictures of the entire classroom.

Teacher T1 has a teacher terminal 5a and can operate it at any time. Note that teacher terminal 5a is an example of teacher terminal 5 in FIG. 1.

The robot R1 is composed of a robot terminal 7a, a wide-angle imaging device 8a, and a vehicle device 10a. The robot terminal 7a is an example of the robot terminal 7 in FIG. 1. The wide-angle imaging device 8a is an example of the wide-angle imaging device 8 in FIG. 1. The vehicle device 10a is an example of the vehicle device 10 in FIG. 1.

[Hardware configuration]
Next, the hardware configuration of the servers, terminals, and the like that constitute the communication system 1 will be described with reference to FIGS.

<Hardware configuration of communication system>
FIG. 3 is a diagram showing the hardware configuration of the communication management system.

As shown in FIG. 3, the communication management server 3 has a typical computer configuration, and includes, for example, a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, an SSD (Solid State Drive) 304, and a display 308. The communication management server 3 also includes a network I/F 306, a keyboard 311, a pointing device 312, a media I/F 307, and a bus 310. Note that an HDD (Hard Disk Drive) may be used instead of an SSD.

The CPU 301 is a computing device that realizes each function of the communication management server 3 by reading programs and data stored in, for example, the ROM 302, the SSD 304, etc., and executing processing. The ROM 302 is a non-volatile memory that stores in advance programs and the like used to start the CPU 301, such as IPL. The RAM 303 is a volatile memory that is used as a work area for the CPU 301, etc.

SSD344 is a storage device that stores, for example, an OS, application software programs, and various data. Display 3058 is a display means that displays various information such as a cursor, menu, window, text, or image.

The network I/F 306 is a communication interface for performing data communication using the communication network 2. The keyboard 311 is an example of an input means for accepting input operations such as characters, numbers, and various instructions by a system administrator or the like. The pointing device 312 accepts operations such as the selection and execution of various instructions by the system administrator or the like, the selection of a processing target, and cursor movement. Note that a mouse or the like may also be used.

The media I/F 307 controls the reading and writing (storing) of data from and to a recording medium 306 such as a memory card. It electrically connects the above components and transmits address signals, data signals, and various control signals.

<Hardware for teacher terminals, robot terminals, and student terminals>
FIG. 4 is a diagram showing the hardware configuration of the teacher terminal, the robot terminal, and the student terminal.

As shown in FIG. 4, the teacher terminal 5 includes a CPU 501, a ROM 502, a RAM 503, an EEPROM 504, a CMOS sensor 505, an image sensor I/F 506, an acceleration/orientation sensor 507, a media I/F 509, and a GPS receiver 511.

Of these, the CPU 501 controls the operation of the entire teacher terminal 5. The ROM 502 stores the CPU 501 and programs used to drive the CPU 501, such as the IPL. The RAM 503 is used as a work area for the CPU 501. The EEPROM 504 reads or writes various data, such as programs for the teacher terminal, under the control of the CPU 501. The CMOS (Complementary Metal Oxide Semiconductor) sensor 505 is a type of built-in imaging means that captures an image of a subject (mainly a self-portrait) and obtains image data under the control of the CPU 501. Note that instead of a CMOS sensor, an imaging means such as a CCD (Charge Coupled Device) sensor may also be used. The imaging element I/F 506 is a circuit that controls the operation of the CMOS sensor 505. The acceleration/direction sensor 507 is various sensors such as an electronic magnetic compass or gyrocompass that detects geomagnetism, and an acceleration sensor. The media I/F 509 controls the reading or writing (storing) of data from a recording medium 508, such as a flash memory. The GPS receiver 511 receives GPS signals from GPS satellites. The GPS receiver 511 can also receive signals indicating indoor positions, such as IMES (Indoor MEssaging System).

The teacher terminal 5 also includes a long-distance communication circuit 512, a CMOS sensor 513, an image sensor I/F 514, a microphone 515, a speaker 516, an audio input/output I/F 517, a display 518, an external device connection I/F 519, a short-distance communication circuit 520, an antenna 520a of the short-distance communication circuit 520, and a touch panel 521.

Among these, the long-distance communication circuit 512 is a circuit that communicates with other devices via the communication network 100. The CMOS sensor 513 is a type of built-in imaging means that captures an image of a subject under the control of the CPU 501 to obtain image data. The image sensor I/F 514 is a circuit that controls the driving of the CMOS sensor 513. The microphone 515 is a built-in circuit that converts sound into an electrical signal. The speaker 516 is a built-in circuit that converts an electrical signal into physical vibrations to generate sounds such as music and voice. The sound input/output I/F 517 is a circuit that processes the input and output of sound signals between the microphone 515 and the speaker 516 under the control of the CPU 501. The display 518 is a type of display means such as liquid crystal or organic EL (Electro Luminescence) that displays images of subjects and various icons. The external device connection I/F 519 is an interface for connecting various external devices. The short-distance communication circuit 520 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The touch panel 521 is a type of input means that allows the user to operate the teacher terminal 5 by pressing the display 518.

The teacher terminal 5 also includes a bus line 510. The bus line 510 is an address bus, a data bus, etc., for electrically connecting each component such as the CPU 401 shown in FIG. 4.

The robot terminal 7 and the student terminal 9 have the same configuration as the teacher terminal 5, so their explanation will be omitted.

<Hardware configuration of earphones with microphone>
FIG. 5 is a diagram showing the hardware configuration of the earphone with microphone.

As shown in FIG. 5, the earphones with microphone 6 include a microphone 615, a speaker 616, an audio input/output I/F 617, a short-range communication circuit 620, an antenna 620a of the short-range communication circuit 520, and a button 621.

The microphone 615 is a built-in circuit that converts sound into an electrical signal. The speaker 616 is a built-in circuit that converts the electrical signal into physical vibrations to produce sound such as music or voice. The sound input/output I/F 617 is a circuit that processes the input and output of sound signals between the microphone 515 and the speaker 516. The short-range communication circuit 520 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The button 621 is a type of operating means that allows the user to operate the microphone-equipped earphone 6.

<Hardware configuration of wide-angle imaging device>
The hardware configuration of the wide-angle imaging device 8 will be described with reference to Fig. 6. Fig. 6 is a hardware configuration diagram of the wide-angle imaging device 8. In the following, the wide-angle imaging device 8 is an omnidirectional (all-directional) wide-angle imaging device 8 using two imaging elements, but the number of imaging elements may be two or more. In addition, the wide-angle imaging device 8 does not necessarily have to be a device dedicated to omnidirectional imaging, and a normal digital camera, a smartphone, or the like may be provided with substantially the same functions as the wide-angle imaging device 8 by attaching an omnidirectional imaging unit to the device.

As shown in FIG. 6, the wide-angle imaging device 8 is also connected to an imaging unit 801, an image processing unit 804, an imaging control unit 805, a microphone 808, a sound processing unit 809, a CPU 811, a ROM 812, an SRAM 813, a DRAM 814, an operation unit 815, a network I/F 816, a short-range communication circuit 817, an antenna 817a, an electronic compass 818, a gyro sensor 819, and an acceleration sensor 820.

Of these, the imaging unit 801 includes wide-angle lenses (so-called fisheye lenses) 802a and 802b each having an angle of view of 180° or more for forming a hemispherical image, and two imaging elements 803a and 803b provided corresponding to each wide-angle lens. The imaging elements 803a and 803b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image by the fisheye lenses 802a and 802b into image data of an electrical signal and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands and parameters necessary for the operation of the imaging elements are set.

The imaging elements 803a and 803b of the imaging unit 801 are each connected to the image processing unit 804 via a parallel I/F bus. Meanwhile, the imaging elements 803a and 803b of the imaging unit 801 are connected to a serial I/F bus (such as an I2C bus) separately from the imaging control unit 805. The image processing unit 804, the imaging control unit 805, and the sound processing unit 809 are connected to the CPU 811 via a bus 810. In addition, the ROM 812, the SRAM 813, the DRAM 814, the operation unit 815, the network I/F 816, the short-range communication circuit 817, and the electronic compass 818 are also connected to the bus 810.

The image processing unit 804 takes in the image data output from the image sensors 803a and 803b via the parallel I/F bus, performs a predetermined process on each piece of image data, and then synthesizes the image data to create equirectangular projection image data as shown in Figure 8(c).

The imaging control unit 805 generally sets commands and the like in the registers of the imaging elements 803a and 803b using the I2C bus, with the imaging control unit 805 as the master device and the imaging elements 803a and 803b as slave devices. Necessary commands and the like are received from the CPU 811. The imaging control unit 805 also uses the I2C bus to retrieve status data and the like from the registers of the imaging elements 803a and 803b, and send it to the CPU 811.

The imaging control unit 805 also instructs the imaging elements 803a and 803b to output image data when the shutter button on the operation unit 815 is pressed. Some wide-angle imaging devices 8 have a preview display function or a function that supports video display on a display. In this case, the image data is output from the imaging elements 803a and 803b continuously at a predetermined frame rate (frames/minute).

As described below, the imaging control unit 805 also functions as a synchronization control means that cooperates with the CPU 811 to synchronize the output timing of image data from the imaging elements 803a and 803b. In this embodiment, the wide-angle imaging device 8 is not provided with a display, but a display unit may be provided.

The microphone 808 converts the recorded sound into audio data. The sound processing unit 809 takes in the audio data output from the microphone 808 via the I/F bus and performs a specified process on the audio data.

The CPU 811 controls the overall operation of the wide-angle shooting device 8 and executes necessary processing. The ROM 812 stores various programs for the CPU 811. The SRAM 813 and DRAM 814 are work memories that store programs executed by the CPU 811 and data in the middle of processing. In particular, the DRAM 814 stores image data in the middle of processing by the image processing unit 804 and data of a processed equirectangular projection image.

The operation unit 815 is a general term for operation buttons such as the shutter button. The operator operates the operation unit 815 to input various imaging modes, imaging conditions, and the like.

The network I/F 816 is a general term for an interface circuit (such as a USB I/F) with external media such as an SD card or a personal computer. The network I/F 816 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 814 is recorded on external media via the network I/F 816, or transmitted to the outside (such as a device) via the network I/F 816 as necessary.

The short-range communication circuit 817 communicates with the outside (such as the robot terminal 7) by short-range wireless communication technology such as Wi-Fi, NFC, or Bluetooth (registered trademark) via an antenna 817a provided in the wide-angle shooting device 8. This short-range communication circuit 817 can also transmit data of the equirectangular projection image to the outside.

The electronic compass 818 calculates the direction of the wide-angle imaging device 8 from the Earth's magnetism and outputs the direction information. This direction information is an example of related information (metadata) that conforms to the Exchangeable image file format (Exif), and is used for image processing such as image correction of the captured image. The related information also includes data such as the capture date and time of the image and the data size of the image data.

The gyro sensor 819 detects changes in angle (roll angle, pitch angle, yaw angle) that accompany the movement of the wide-angle imaging device 8. The changes in angle are an example of related information (metadata) according to Exif, and are used for image processing such as image correction of captured images.

The acceleration sensor 820 detects acceleration in three axial directions. The attitude of the wide-angle imaging device 8 (angle with respect to the direction of gravity) is detected based on the detected acceleration. Having both the gyro sensor 819 and the acceleration sensor 820 improves the accuracy of image correction.

(About spherical images)
Next, the usage of the wide-angle imaging device 8 will be described with reference to Fig. 7. Fig. 7 is an image diagram of the wide-angle imaging device 8 in use. As shown in Fig. 7, the wide-angle imaging device 8 is used, for example, by a person holding it to capture surrounding subjects and scenery. In this case, two hemispherical images can be obtained by capturing images of subjects around the person using the imaging element 803a and the imaging element 803b shown in Fig. 6, respectively.

Next, an outline of the process for creating an equirectangular projection image EC and a celestial sphere image CE from an image captured by the wide-angle imaging device 8 will be described with reference to Figures 8 and 9. Note that Figure 8(a) shows a hemispherical image (front side) captured by the wide-angle imaging device 8, Figure 8(b) shows a hemispherical image (rear side) captured by the wide-angle imaging device 8, and Figure 8(c) shows an image expressed by equirectangular projection (hereinafter referred to as an "equirectangular projection image"). Figure 9(a) is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and Figure 9(b) shows a celestial sphere image.

As shown in FIG. 8(a), the image obtained by the image sensor 803a becomes a hemispherical image (front side) curved by the fisheye lens 802a described below. Also, as shown in FIG. 8(b), the image obtained by the image sensor 803b becomes a hemispherical image (rear side) curved by the fisheye lens 802b. The hemispherical image (front side) and the hemispherical image (rear side) flipped 180 degrees are then combined by the wide-angle imaging device 8 to create an equirectangular projection image EC as shown in FIG. 8(c).

By using OpenGL ES (Open Graphics Library for Embedded Systems), the equirectangular projection image is pasted to cover the sphere as shown in FIG. 9A, and a celestial sphere image CE as shown in FIG. 9B is created. In this way, the celestial sphere image CE is represented as an image in which the equirectangular projection image EC faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the celestial sphere image CE may be a still image or a video.

As described above, the spherical image CE is an image pasted to cover the spherical surface, which gives a sense of incongruity to humans when viewed. Therefore, by displaying a specific area of the spherical image CE (hereinafter, equivalent to the above-mentioned "specific area image") as a planar image with little curvature, it is possible to display the image in a way that does not give a sense of incongruity to humans. This will be described with reference to Figs. 10 and 11. The specific area image may be a video or a still image.

10 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is a three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of the operator who views the omnidirectional image CE displayed as a three-dimensional sphere. FIG. 11(a) is a three-dimensional perspective view of FIG. 10, and FIG. 11(b) is a diagram showing a predetermined area image when displayed on a display. In FIG. 11(a), the omnidirectional image shown in FIG. 10 is represented by a three-dimensional sphere CS. If the omnidirectional image CE generated in this way is a sphere CS, as shown in FIG. 11, the virtual camera IC is located inside the omnidirectional image CE. The predetermined area T in the omnidirectional image CE is the imaging area of the virtual camera IC, and is specified by the predetermined area information indicating the imaging direction and the angle of view of the virtual camera IC in the three-dimensional virtual space including the omnidirectional image CE. The zoom of the predetermined area T can also be expressed by moving the virtual camera IC closer to or farther away from the omnidirectional image CE. The predetermined area image Q is an image of a predetermined area T in the omnidirectional image CE. Therefore, the predetermined area T can be specified by the angle of view α and the distance f from the virtual camera IC to the omnidirectional image CE (see FIG. 12).

The predetermined area image Q shown in FIG. 11(a) is displayed on a predetermined display as an image of the imaging area of the virtual camera IC, as shown in FIG. 11(b). The image shown in FIG. 11(b) is a predetermined area image represented by the initial (default) predetermined area information. In the following, the imaging direction (ea, aa) and angle of view (α) of the virtual camera IC are used for explanation. Note that the predetermined area T may be represented by the imaging area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, instead of the angle of view α and distance f.

In addition, when the virtual viewpoint of the virtual camera IC is moved (also called "changed") from the state shown in FIG. 11(a) to the right (left when looking at the drawing) as shown in FIG. 11(c), the predetermined area T in the omnidirectional image CE is moved to the predetermined area T' accordingly, and the predetermined area image Q displayed on the predetermined display is changed to the predetermined area image Q'. As a result, the image shown in FIG. 11(b) is changed to the image shown in FIG. 11(d) and displayed on each display 518 of the destination teacher terminal 5 and student terminal 9.

Next, the relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. 12. FIG. 12 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. 12, "ea" indicates an elevation angle, "aa" indicates an azimuth angle, and "α" indicates an angle of view. That is, the posture of the virtual camera IC is changed so that the gaze point of the virtual camera IC indicated by the imaging direction (ea, aa) becomes the center point CP of the predetermined area T, which is the imaging area of the virtual camera IC. As shown in FIG. 12, the center point CP when the diagonal angle of view of the predetermined area T represented by the angle of view α of the virtual camera IC is α becomes the (x, y) parameter of the predetermined area information. f is the distance from the virtual camera IC to the center point CP. L is the distance between any vertex of the predetermined area T and the center point CP (2L is the diagonal line). In FIG. 12, the trigonometric function shown in the following (Equation 1) generally holds.
(L/f)=tan(α/2) (Equation 1)
<Hardware configuration of vehicle device>
13 is a diagram showing a hardware configuration of the vehicle device 10. The vehicle device 10 includes, for example, a CPU 1001, a ROM 1002, a RAM 1003, an external device connection I/F 1019, a short-range communication circuit 1020, an antenna 1020a of the short-range communication circuit 1020, a wheel drive unit 1021, and a steering unit 1022.

The CPU 1001 is a computing device that realizes each function of the vehicle device 10 by executing programs stored in the ROM 1002 or the like. The ROM 1002 is a non-volatile memory that stores the programs of the vehicle device 10, etc. The ROM 1002 may be, for example, a rewritable non-volatile memory such as a flash ROM. The RAM 1003 is a volatile memory that is used as a work area for the CPU 1001, etc.

The external device connection I/F 1019 is a wired communication interface for connecting to and communicating with a robot terminal 7, etc.

The short-range communication circuit 1020 is, for example, a wireless communication interface for performing wireless communication using the same wireless communication method as the robot terminal 7, etc.

The wheel drive unit 1021 is an example of a drive device that drives wheels to move the vehicle device 10. The wheel drive unit 1021 includes, for example, a motor.

The steering unit 1022 is an example of a steering device that steers the vehicle device 10 that moves by the wheel drive unit 1021. The steering unit 1022 may, for example, change the direction or inclination of the wheels, or may change the direction of the vehicle device 10 (robot R) by controlling the number of rotations or speed of the left and right wheels.

[Functional configuration of the communication system]
Next, the functional configuration of the communication system 1 will be described with reference to Fig. 14. Fig. 14 is a diagram showing the functional configuration of the communication system 1.

<Communication management server>
As shown in FIG. 14, the communication management server 3 has a transmission/reception unit 31, an authentication unit 35, a position setting unit 36, an image creation unit 37, and a participation registration unit 38. Each of these units is a function or means realized by any of the components shown in FIG. 3 operating according to an instruction from the CPU 301 in accordance with a program deployed from the SSD 304 to the RAM 303. The communication management server 3 also has a storage unit 40 constructed by the ROM 302, RAM 303, or SSD 304 shown in FIG. 3. The storage unit 40 stores position correspondence information (matching information) indicating the correspondence between positions in the real space and positions in the virtual space. The storage unit 40 also stores seat position information indicating the seat position in the real space of each student S.

(Location information management table)
The storage unit 40 stores a location information management table as shown in FIG. 15. In the location information management table, each piece of information indicating the name of a person or robot (for a person, the name, etc.), login ID, password, icon ID, icon image, real space position information, virtual space position information, participation destination, and dedicated voice communication partner is managed in association with each other. Note that the first half of the information (name, login ID, password, icon ID, and icon image) is information that is rarely changed, and the second half of the information (real space position information, virtual space position information, participation destination, and dedicated voice communication partner) is information that is likely to be changed depending on the situation in the classroom in the real space and the operation of the student S participating in the remote class. Note that the icon image is merely shown diagrammatically, and actually, the icon data or the URL in the storage unit 40 in which the icon is stored, etc. are managed. It can also be said that the icon is not only related to a specific person, but also related to the communication terminal (teacher terminal 5, student terminal 9, etc.) used by the same specific person.

"Participation destination" is information for identifying, for example, the icon t1 of teacher T1 and the robot R1 (here, the icon ID of robot R1) associated with the icon of each student S in FIG. 20. "Exclusive voice communication partner" is information for identifying the partner (here, the icon ID) with which teacher T communicates by voice when the voice cannot be heard by other students.

(Functional configuration)
The transmitting/receiving unit 31 performs data communication with other terminals (devices).

The authentication unit 35 performs login authentication for other terminals (devices), etc.

The position setting unit 36 performs processing such as storing and setting, in the "virtual space position information" field of FIG. 15, virtual space terminal (device) position information indicating a position in virtual space corresponding to the real space terminal position information, based on position correspondence information (matching information) indicating the correspondence relationship between positions in real space and positions in virtual space. Conversely, the position setting unit 36 performs processing such as storing and setting, in the "real space position information" field, real space position information indicating a position in real space corresponding to the virtual space terminal (device) position information, based on the position correspondence information.

The image creation unit 37 creates the attendance registration image shown in FIG. 18 and virtual space images such as FIG. 20, which will be described later.

The participation registration unit 38 registers teacher T (or teacher T's communication terminal) and remote student S (student S's communication terminal) to participate in a remote lesson in a virtual space.

The communication management server 3 may be constructed from multiple servers. In this case, each field of the location information management table is divided and stored according to the role of each server.

<Teacher's terminal>
14, the teacher terminal 5 has a communication unit 50, a transmission/reception unit 51, a reception unit 52, a position acquisition unit 53, and a display control unit 54. Each of these units is a function or means realized by any of the components shown in FIG. 3 operating in response to an instruction from the CPU 501 in accordance with a program loaded from the EEPROM 504 onto the RAM 503.

The communication unit 50 communicates with the microphone-equipped earphones 6 through processing by the short-range communication circuit 520, etc.

The transmitter/receiver unit 51 communicates data with other terminals (devices) via the communication network 100.

The reception unit 52 receives operations from a person (such as teacher T).

The location acquisition unit 53 acquires location information indicating an outdoor or indoor location through processing by the GPS receiving unit 511, etc.

The display control unit 54 displays various images on the display 518 of the teacher terminal 5.

<Robot terminal>
14, the robot R (robot terminal 7) has a transmission/reception unit 71, a reception unit 72, a position acquisition unit 73, a display control unit 74, and a movement control unit 78. Each of these units is a function or means realized by any of the components shown in FIG. 3 operating in response to an instruction from the CPU 501 in accordance with the program loaded from the EEPROM 504 onto the RAM 503.

The transmitter/receiver unit 71 communicates data with other terminals (devices) via the communication network 100.

The reception unit 72 receives operations from a person (such as teacher T).

The location acquisition unit 73 acquires location information indicating an outdoor or indoor location through processing by the GPS receiving unit 511, etc.

The display control unit 74 displays various images on the display 518 of the robot terminal 7.

The movement control unit 78 performs control to drive the vehicle device 10 and move the robot R in response to instructions from the communication management server 3.

<Student device>
14, the student terminal 9 has a transmission/reception unit 91, a reception unit 92, and a display control unit 94. Each of these units is a function or means realized by any of the components shown in FIG. 3 operating in response to an instruction from the CPU 501 in accordance with a program loaded from the EEPROM 504 onto the RAM 503.

The transmitter/receiver unit 71 communicates data with other terminals (devices) via the communication network 100.

The reception unit 72 receives operations from people (such as student S).

The display control unit 74 displays various images on the display 518 of the student terminal 9.

[Communication system processing or operation]
Next, the processing or operation of the communication system will be described with reference to FIGS.

<Login process>
First, the login process of each terminal and robot will be described with reference to Fig. 16. Fig. 16 is a sequence diagram showing the login process of each terminal and robot. In the case of robot R, the robot terminal 7 actually performs the login. Also, the processes of the microphone-equipped earphone 6a, the teacher terminal 5a, the robot R1 (robot terminal 7a), the communication management server 3, and the student terminal 9f will be described.

S11: The reception unit 72 of the robot R1 receives a login operation (such as input of a login ID and password) from the teacher T1, and the transmission/reception unit 71 sends a login request for video and audio communication to the communication management server 3. This login request also includes the login ID and password of the robot R1. Furthermore, the login request also includes real-space device position information indicating the position of the robot R1 in real space (here, in the classroom). The real-space device position information of the robot R1 is information indicating the position of the robot R1 in real space acquired by the position acquisition unit 73 of the robot R1. As a result, the transmission/reception unit 31 of the communication management server 3 receives the login request from the robot R1.

S12: In the communication management server 3, the authentication unit 35 performs login authentication based on whether the set of login ID and password received by the transmission/reception unit 31 in process S11 matches the set of login ID and password managed in the position information management table. If the login authentication is successful, the position setting unit 36 stores the real space device position information received in process S11 in the "real space position information" field of the record that includes the login ID and password used for this authentication. Note that the timing of transmission and reception of the real space device position information is not limited to process S11 and can be any time before the start of process S51 described below. The authentication result is returned to the robot terminal 7a.

S13: Before logging in, pair the microphone-equipped earphones 6a with the teacher terminal a5 to establish a short-range wireless connection between the microphone-equipped earphones 6a and the teacher terminal 5a.

S14: The reception unit 52 of the teacher terminal 5a receives a login operation (such as input of a login ID and password) from the teacher T1, and the transmission/reception unit 51 transmits a login request for the virtual space and voice communication to the communication management server 3. This login request also includes the login ID and password of the teacher T1. The login request also includes real-space terminal position information indicating the position of the teacher terminal 5a in the real space. The real-space terminal position information of the teacher terminal 5a is information indicating the position of the teacher terminal 5a in the real space (here, in the classroom) acquired by the position acquisition unit 53 of the teacher terminal 5a. As a result, the transmission/reception unit 31 of the communication management server 3 receives the login request from the teacher terminal 5a. Each time the teacher terminal 5a moves, the real-space terminal position information after the movement is transmitted from the teacher terminal 5a to the communication management server 3.

S15: In the communication management server 3, the authentication unit 35 performs login authentication based on whether the login ID and password pair received by the transmission/reception unit 31 in process S12 matches the login ID and password pair managed in the location information management table. If the login authentication is successful, the location setting unit 36 stores the real-space location information received in process S12 in the "real-space location information" field of the record that includes the login ID and password used for this authentication. Note that the timing of transmission and reception of real-space terminal location information is not limited to process S12 and can be any time before process S51, which will be described later, is started. The authentication result is returned to the teacher terminal 5a.

S16: The participation registration unit 38 registers the login ID "L31" of the robot R1 registered in process S12 in the "Participation destination" field of the record identified by the login ID received in process S14, and teacher T1 thereby participates in the remote lesson in the classroom in which robot R1 is present. As a result, teacher T1 not only attends the classroom lesson in the real world, but also participates in the remote lesson in the virtual world.

S17: The reception unit 92 of the student terminal 9f receives a login operation (such as input of a login ID and password) from the student S6, and the transmission/reception unit 91 transmits a login request for the communication management server 3's services to the communication management server 3. This login request also includes the login ID and password of the student terminal 9f. As a result, the transmission/reception unit 31 of the communication management server 3 receives the login request from the student terminal 9f.

S18: In the communication management server 3, the authentication unit 35 performs login authentication based on whether the login ID and password pair received by the transmission/reception unit 31 in step S17 matches the login ID and password pair managed in the location information management table. If the login authentication is successful, the location setting unit 36 registers predetermined virtual space location information in the "virtual space location information" field of the record containing the login ID and password used for this authentication. Examples of this virtual space location information being predetermined include when the position of the student's desk in the classroom is fixed and therefore the corresponding virtual space position is fixed, or when each student S registers their own seat before class. The authentication result is returned to the student terminal 9f. In the following, the explanation will be continued for the case where all of the above login authentications are successful.

S19: The participation registration unit 38 participates in the remote lesson in the classroom in which the robot R1 is located by registering the login ID "L31" of the robot R1 registered in process S12 in the "Participation destination" field of the record identified by the login ID received in process S17. Note that FIG. 15 shows a case in which not only the student S6 but also the student S5 using the student terminal 9e participates in the remote lesson in the classroom in which the robot R1 is located.

<Audio communication, audio-video communication, and attendance registration>
Next, the process of starting voice communication and audio-video communication and the process of registering attendance will be described with reference to Fig. 17 and Fig. 18. Fig. 17 shows a sequence diagram of the process of starting voice communication and audio-video communication and the process of registering attendance.

S31: The transmitter/receiver 31 of the communication management server 3 references the location information management table and establishes an audio communication session between the teacher terminal 5 (teacher terminal 5a in this case) and the student terminal 9 (student terminals 9e and 9f in this case) in which the login ID "L31" of the same robot R1 is managed in the "Participation Destination" field. In addition, because a short-range wireless connection has been established between the microphone-equipped earphone 6 and the teacher terminal 5a by the above-mentioned process S13, audio communication can be performed between the microphone-equipped earphone 6a, the teacher terminal 5a, and the student terminal 9e.

S32: The transmission/reception unit 31 of the communication management server 3 refers to the location information management table, and establishes a video and audio communication session between the robot R1 and the student terminal 9 (here, student terminals 9e and 9f) in which the login ID "L31" of the robot R1 is managed in the "Participation destination" field. This allows the robot R1 and the student terminals 9e and 9f to perform video and audio communication with each other via the communication management server 3. For example, when the robot R1 transmits video and audio data of a classroom, the video of the classroom is displayed on the display 518 of the student terminals 9e and 9f, and audio from within the classroom is output. On the other hand, when the student terminals 9e and 9f transmit data of the video of the faces of students S5 and S6 and the audio data of students S5 and S6, the robot R1 displays the video of the faces of students S5 and S6 on the display 518 of the robot terminal 7a and outputs the audio of students S5 and S6 (see FIG. 23).

S33: The reception unit 52 of the teacher terminal 5a receives a request for an attendance registration image from teacher T1. Then, the transmission/reception unit 51 transmits the request for the attendance registration image to the communication management server 3. As a result, the transmission/reception unit 31 of the communication management server 3 receives the request for the attendance registration image.

S34: The image creation unit 37 reads the seat position information stored in the memory unit 40 and creates an attendance registration image.

S35: The transmitting/receiving unit 31 transmits data of the attendance registration image created by the image creation unit 37 to the teacher terminal 5a. As a result, the transmitting/receiving unit 51 of the teacher terminal 5a receives the data of the attendance registration image.

S36: As shown in FIG. 18, in the teacher terminal 5a, the display control unit 54 causes a screen 110 showing an attendance registration image to be displayed on the display 518 of the teacher terminal 5a. FIG. 18 is a diagram showing a student attendance registration image. FIG. 18 shows the layout of the classroom, with icons α1 to α6 showing the predetermined positions of student desks and student names. In addition, a "Register" button 119 for registering attendance is displayed in the lower right corner of the attendance registration image.

Here, the reception unit 52 receives the results of attendance at the classroom in real space by teacher T1 selecting each icon with cursor c1. FIG. 18 shows a state in which students S1 to S4 are attending the classroom in real space, while students S5 and S6 are not. Then, when the reception unit 52 receives teacher T1 pressing the "Register" button, the transmission/reception unit 51 transmits attendance information indicating the attendance received from teacher T1 to the communication management server 3. This attendance information includes student identification information that identifies the student who attended. The student identification information is the student's name, the student's login ID, or the icon ID of the student's icon. As a result, the transmission/reception unit 31 of the communication management server 3 receives the attendance information.

S37: In the communication management server 3, the position setting unit 36 registers predetermined virtual space position information in the "virtual space position information" field of the record in the position information management table that is identified by the student identification information. Examples of this virtual space position information being predetermined include when the positions of students' desks in a classroom are fixed, or when each student registers their own seat before class.

As a result, as shown in Figure 2, students S1 to S4 are attending a classroom class in the real world, while students S5 and S6 are participating in a remote class in the virtual world.

Display of Image Space Images
Next, the process of displaying an image space image will be described with reference to Fig. 19 to Fig. 23. Fig. 19 shows a sequence illustrating the process of displaying a virtual space image.

S51: In the communication management server 3, the position setting unit 36 stores and sets the virtual space terminal information corresponding to the real space terminal position information stored in the "real space position information" field in the "virtual space position information" field of the record of the teacher T1 in the position information management table based on the position correspondence information (matching information) indicating the correspondence between the position in the real space and the position in the virtual space. Similarly, the position setting unit 36 stores and sets the virtual space terminal information corresponding to the real space terminal position information stored in the "real space position information" field in the "virtual space position information" field of the record of the robot R1 in the position information management table based on the position correspondence information (matching information). As a result, the movements of the teacher T1 and the robot R1 in the real space are reflected in the movements of the icons t1 and r1 in the virtual space. Furthermore, in this embodiment, the movement of the icon r1 of the robot R1 in the virtual space is reflected in the movement of the robot R1 in the real space.

In addition, as shown in FIG. 20, the image creation unit 37 creates a virtual space image by mapping icon images corresponding to the virtual space position information based on the seat position information and the virtual space position information managed in the position information management table onto a layout map m1 that is a schematic representation of a classroom in real space.

S52: The transmission/reception unit 31 transmits data of the virtual space image created by the image creation unit 37 to the student terminal 9f. As a result, the transmission/reception unit 91 of the student terminal 9f receives the data of the virtual space image.

S53: Next, on the student terminal 9f, the display control unit 94 causes the display 518 to display a screen 120 showing a virtual space image as shown in FIG. 20.

Figure 20 shows a virtual space image including icons indicating the current positions of the teacher and robot, icons indicating the initial positions of each student, and video from the robot. As shown in Figure 20, the virtual space image shows a layout map m1 that simplifies a classroom in real space, an icon t1 of teacher T1, an icon r1 of robot R1, icons s1-s6 of each student S1-S6, and icons d1-d6 of each desk D1-D6. Shown here are the icon t1 of teacher T1 at its current position in real space, and the icon r1 of robot R at its current position in real space. Also shown are the icons s1-s6 and d1-d6, whose positions in virtual space have been determined in advance.

Furthermore, on the right side of screen 120, a separate screen (such as a pop-up screen) 121 is displayed, which shows the current image of the classroom in real space sent from robot R1 by the audio-visual communication of process 32, and audio from robot R1 is also being output.

In addition, teacher terminal 5a of teacher T1 and student terminal 9e of student S5 and student terminal 9f of student S6 can perform voice communication in process S31. Therefore, teacher T1 can use earphones with a microphone 6a to explain the classroom situation in real space to students S5 and S6, and student S5 can use student terminal 9e and student S6 can use student terminal 9f to ask teacher T1 questions about the lesson in real space.

The robot R1 can transmit audio data from the classroom to the remote student terminals 9e, 9f through audiovisual communication, but this also picks up noise and conversations around the robot R1. In contrast, when using the above-mentioned audiovisual communication, the teacher T1 does not speak to the robot R1, but speaks through the microphone-equipped earphones 6, so that the students S5, S6 can hear the teacher T1's voice clearly.

S54: In the student terminal 9e, the reception unit 92 receives an operation from the student S6 to move the icon r1 using the cursor c2 as shown in FIG. 21. The transmission/reception unit 91 then transmits the changed virtual space position information resulting from the movement of the icon r1 to the communication management server 3. This virtual space position information includes the icon ID "r31" of the robot R1. As a result, the transmission/reception unit 31 of the communication management server 3 receives the changed virtual space position information.

The position setting unit 36 overwrites the changed virtual space position information in the "virtual space position information" field in the position information management table that corresponds to the icon ID received in process S54. As a result, the image creation unit 37 creates the latest virtual space image using the overwritten changed virtual space position information. Furthermore, the position setting unit 36 uses the position correspondence information (matching information) to overwrite the changed real space position information that corresponds to the changed virtual space position information in the "real space position information" field of the same record. In this way, processes S51 to S54 are repeated, and the virtual space image displayed on the student terminal 9e is updated.

S55: The transmitter/receiver 31 transmits the changed real-space position information overwritten in step S54 to the robot R1. As a result, the transmitter/receiver 71 of the robot R1 receives the changed real-space position information.

S56: In the robot R1, the movement control unit 78 controls the movement of the vehicle device 10 based on the changed real-space position information received in process S55 so that the robot R1 moves to a position in the classroom in the real space indicated by the real-space position information. As a result, as shown in FIG. 22, in the classroom in the real space, the robot R1 moves to the vicinity of the student S3. For example, when a student S6 participating in a remote class wants to talk with a student S3 attending a class in the classroom in the real space, as shown in FIG. 21, the student S6 moves the icon r1 of the robot R1 to the vicinity of the icon s3 of the student S3 on the virtual space screen of the student terminal 9f. As a result, as shown in FIG. 22, the robot R in the classroom in the real space moves to the vicinity of the student S3, so that the student S3 in the classroom and the remote student S6 can talk through the robot R1. In this case, as shown in FIG. 23, the remote students S5 and S6 are displayed on the robot terminal 7a of the robot R1 by the video and audio communication of process 32, so that the student S3 can talk with the student S6 with a feeling close to the real space.

<Dedicated voice communication>
Next, the process of performing dedicated voice communication will be described with reference to Fig. 24 and Fig. 25. Fig. 24 is a sequence diagram showing the process of performing dedicated voice communication. Fig. 25 is a diagram showing a virtual space image in a state where an icon of a party with which to perform dedicated voice communication is selected.

S71: In the student terminal 9a, the reception unit 92 receives an operation from the student S6 to select the icon t1 of the teacher T1 using the cursor c2 as shown in FIG. 25. The transmission/reception unit 91 then transmits information indicating the selection of the icon t1 to the communication management server 3 as a request to establish a dedicated voice session. This information includes the icon ID "s26" of the student S6 who made the selection and the icon ID "t11" of the icon t1 of the selected teacher T1. As a result, the transmission/reception unit 31 of the communication management server 3 receives the information indicating the selection of the icon.

S72: The transmission/reception unit 31 of the communication management server 3 stores and manages the login ID "L26" in the "dedicated voice communication partner" field of the record of the icon ID "t11" in the location information management table, and stores and manages the login ID "L11" in the "dedicated voice communication partner" field of the record of the icon ID "s26" in the location information management table. As a result, the transmission/reception unit 91 establishes a dedicated voice communication session between the teacher terminal 5a, which uses the login ID "L11" that is mutually managed as a dedicated voice communication partner, and the student terminal 9f, which uses the login ID "L26" that is mutually managed as a dedicated voice communication partner. In addition, since the microphone-equipped earphone 6a and the teacher terminal 5a are connected by short-distance wireless connection by the above-mentioned process S13, dedicated voice communication can be performed between the microphone-equipped earphone 6a, the teacher terminal 5a, and the student terminal 9f. This allows student S6 to consult with teacher T1 without the other students S5 and others hearing the audio content, and conversely, teacher T1 can reply to student S6 without the other students S5 and others hearing the audio content.

In FIG. 25, student S6 selects the icon t1 of teacher T1, but this is not limited to this. For example, student S6 may select the icon s5 of another remote student S5 and hold a discussion with student S5 via dedicated voice communication.

In addition, at least one of the teacher terminal 5 and the student terminal 9 may be a head mounted display (HMD). Here, the hardware configuration of the HMD will be explained using FIG. 26. FIG. 26 is a hardware configuration diagram of a head mounted display.

As shown in FIG. 26, the HMD is a computer that includes a CPU 201, a ROM 202, a RAM 203, an external device connection I/F 205, a display 207, an operation unit 208, a media I/F 209, a bus line 210, a speaker 212, an electronic compass 218, a gyro sensor 219, and an acceleration sensor 220.

Of these, the CPU 201 controls the operation of the entire HMD. The ROM 202 stores programs used to drive the CPU 201, such as the IPL. The RAM 203 is used as a work area for the CPU 201.

The external device connection I/F 205 is an interface for connecting various external devices. In this case, the external devices are the communication management server 3, the earphones with microphone 6, etc.

The display 207 is a type of display unit, such as a liquid crystal or organic EL (Electro Luminescence) display, that displays various images.

The operation unit 208 is an input means for selecting and executing various instructions, selecting the processing target, moving the cursor, etc., such as various operation buttons, a power switch, physical buttons, and an eye gaze operation circuit that detects and operates the user's gaze.

The media I/F 209 controls the reading and writing (storing) of data to a recording medium 209m such as a flash memory. The recording medium 209m includes DVDs and Blu-ray Discs (registered trademarks).

Speaker 212 is a circuit that converts electrical signals into physical vibrations to produce sounds such as music and voice.

The electronic compass 218 calculates the direction of the HMD from the Earth's magnetic field and outputs the direction information.

The gyro sensor 219 is a sensor that detects changes in angle (roll angle, pitch angle, yaw angle) that accompany the movement of the HMD.

The acceleration sensor 220 is a sensor that detects acceleration in three axial directions.

The bus line 210 is an address bus, a data bus, etc., for electrically connecting each component such as the CPU 201.

In this case, each functional configuration of the teacher terminal 5 and the student terminal 9 (see FIG. 14) is realized by instructions from the CPU 201 according to the program stored in the RAM 253.

In this case, the position acquisition unit 53 of the teacher terminal 5 acquires position information indicating an indoor position through processing of the electronic compass 218, the gyro sensor 219, and the acceleration sensor 220. In addition, in order for the position acquisition unit 53 to acquire position information indicating an outdoor position, the HMD may have a configuration similar to that of the GPS receiver unit 511 in FIG. 4. Furthermore, the communication unit 50 of the teacher terminal 5 performs wireless communication with the microphone-equipped earphones 6 through processing of the external device connection I/F 205, etc.

[Main Effects of the Embodiments]
As described above, according to this embodiment, as shown in Fig. 21, when a student S6 participating in a remote class moves the icon r1 of the robot R1 on the virtual space image, the robot R1 in the real space moves from the position shown in Fig. 2 to the position in Fig. 22 corresponding to the movement of the icon r1. This makes it easier for the student S6 participating in an event such as a remote class to communicate with the student S3 in the classroom where the event is held.

Furthermore, as described above, the teacher T can talk to the remote student S using the microphone-equipped earphones 6 without speaking directly to the robot R, so that the remote student S can hear the clear voice of the student S, rather than hearing the voice of the robot R mixed with surrounding noise and conversations, etc.

Also, as shown in FIG. 25, when a student S6 participating in an event such as a distance learning class selects the icon t1 of the person with whom he or she wants to talk directly (here, teacher T1), a dedicated voice communication session separate from the voice communication session by process S31 is established (process S72), and the student S6 can talk to teacher T1 about matters that he or she does not want other students S5, etc. to hear.

〔supplement〕
Each function in the above-described embodiment can be realized by one or more processing circuits. Here, the "processing circuit" in the present embodiment includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and devices such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), a SOC (System on a chip), a GPU, and a conventional circuit module designed to execute each of the above-described functions.

Furthermore, each of the above programs may be distributed by being recorded on a (non-temporary) recording medium.

Each CPU 201, 301, 501, 801, 1001 may be single or multiple.

R Robot (an example of a moving device)
1 Communication system 3 Communication management server 5 Teacher terminal (an example of another communication terminal)
6. Earphones with a microphone (an example of an audio input/output device)
7 Robot terminal 8 Wide-angle imaging device 9 Student terminal 9e Student terminal (an example of another communication terminal)
9f Student terminal (an example of a predetermined communication terminal)
10 Vehicle device 31 Transmitting/receiving unit (an example of a transmitting unit, an example of a receiving unit, an example of a voice communication establishment unit, an example of a communication establishment unit)
35 Authentication unit 36 Position setting unit 37 Image creation unit 38 Participation registration unit 40 Storage unit

JP 2021-136679 A

Claims (10)

a communication establishment unit that establishes voice communication between a mobile device capable of inputting and outputting voice in a real space and capable of autonomously moving to a position indicated by real space position information and a predetermined communication terminal in the real space;
an image creation unit that creates a virtual space image showing an icon related to a person in the real space and an icon related to the mobile device in the real space;
A transmission unit that transmits the virtual space image to the predetermined communication terminal;
a receiving unit that receives, from the predetermined communication terminal, virtual space position information after a change based on a movement of an icon related to the moving device with respect to the virtual space image;
having
the transmitting unit transmits, to the mobile device, changed real space position information corresponding to the changed virtual space position information;
Communications management server. the communication establishment unit establishes, in addition to the voice communication, a video/audio communication that enables a video communication between the mobile device and the predetermined communication terminal;
The mobile device is capable of taking pictures, and the pictures obtained by taking pictures are transmitted from the mobile device to the predetermined communication terminal.
The communication management server according to claim 1 . The communication management server according to claim 1, further comprising a participation registration unit that, if a login is successful in response to a login request from the specified communication terminal, registers the specified communication terminal to participate in a virtual space event corresponding to the real space event in which the mobile device exists. The communication management server according to claim 3, wherein the image creation unit creates the virtual space image showing icons relating to the specified communication terminals that have been registered by the participation registration unit to participate in the event in the virtual space, and icons relating to people who attended the event in the real space. The communication management server according to claim 3, further comprising an audio communication establishment unit that establishes audio communication between the specified communication terminal in the real space and another communication terminal capable of audio communication with an audio input/output device in the real space. the image creation unit creates the virtual space image showing an icon related to the predetermined communication terminal and an icon related to the other communication terminal as icons related to people in the real space;
the voice communication establishment unit accepts an operation from the predetermined communication terminal to associate an icon related to the predetermined communication terminal with an icon related to the other communication terminal, thereby establishing dedicated voice communication between the predetermined communication terminal and the other communication terminal;
The communication management server according to claim 5. A communication management server according to any one of claims 1 to 6;
The moving device;
A communication system having the above configuration. The communication system according to claim 7, wherein the mobile device displays the video transmitted from the specified communication terminal through the video and audio communication established by the communication management server. a communication establishment process for establishing voice communication between a mobile device capable of inputting and outputting voice in a real space and capable of autonomously moving to a position indicated by the real space position information and a predetermined communication terminal in the real space;
an image creation process for creating a virtual space image showing an icon related to a person in the real space and an icon related to the mobile device in the real space;
a transmission process of transmitting the virtual space image to the predetermined communication terminal;
a receiving process for receiving, from the predetermined communication terminal, virtual space position information after change based on a movement of an icon related to the moving device with respect to the virtual space image;
having
the transmission process includes a process of transmitting, to the mobile device, changed real space position information corresponding to the changed virtual space position information;
How to manage communications. A program for causing a computer to execute the method according to claim 9.

Images