JP7400231B2 - Communication systems, information processing devices, communication methods and programs - Google Patents

Description

The present invention relates to a communication system, an information processing device, a communication method, and a program.

In a video conferencing system in which multiple participants at multiple locations hold a conference using a display, there is a known technology for displaying, for example, avatars that virtually represent each participant on the display as information about the participants in the video conference. It is being

In order to reduce the amount of data transmitted over the network and prevent it from becoming difficult for participants to grasp the status of the meeting, a video conferencing system that displays avatars is designed to A system for controlling screen display has been disclosed (see Patent Document 1).

However, the technique described in Patent Document 1 has a problem in that display control cannot be performed unless attribute information of participants is registered in advance.

The present invention has been made in view of the above-mentioned problems, and the present invention provides a communication system and information system that can control the display of participant information even if attribute information is not registered before the start of a video conference. The purpose is to provide processing devices, communication methods, and programs.

In order to solve the above-mentioned problems and achieve the objects, the present invention provides a communication system in which a plurality of communication terminals can conduct a video conference by transmitting and receiving audio data via a network. an imaging section that obtains video data of the above participants; a detection section that detects the participants from the video data obtained by the imaging section; and an audio input section that inputs audio data uttered by the participants. an extraction unit that extracts attribute information of the participant from the audio data input by the audio input unit; and an extraction unit that extracts attribute information of the participant from the audio data input by the audio input unit; Regarding participant information, a first control unit that controls display on a first communication terminal, and the attribute information for participants who use a second communication terminal that communicates with the first communication terminal via the network. a request unit that transmits an instruction to request an utterance to the second communication terminal; and a request unit that transmits an instruction to request an utterance from the second communication terminal to a participant using the second communication terminal, according to the instruction received from the request unit. A second control unit that performs a process of prompting the user to utter the attribute information .

According to the present invention, display control of participant information can be performed even if attribute information is not registered before the start of a video conference.

FIG. 1 is a schematic configuration diagram of a communication system according to an embodiment. FIG. 2 is a diagram illustrating an example of the hardware configuration of a communication terminal (video conference terminal) according to the embodiment. FIG. 3 is a diagram showing an example of the hardware configuration of the communication terminal (electronic whiteboard) according to the embodiment. FIG. 4 is a diagram illustrating an example of the hardware configuration of the management system and program providing system according to the embodiment. FIG. 5 is a diagram illustrating an example of the software configuration of the communication terminal according to the embodiment. FIG. 6 is a diagram illustrating an example of the configuration of functional blocks of the communication system according to the embodiment. FIG. 7 is a diagram showing an example of an authentication management table. FIG. 8 is a diagram showing an example of a terminal management table. FIG. 9 is a diagram showing an example of a group management table. FIG. 10 is a diagram showing an example of a session management table. FIG. 11 is a diagram showing sessions established for transmitting and receiving content data and various types of management information in the communication system according to the embodiment. FIG. 12 is a sequence diagram illustrating an example of a preparation stage process including an authentication process for a communication terminal to start a call in the communication system according to the embodiment. FIG. 13 is a diagram showing a display example of the destination list. FIG. 14 is a sequence diagram illustrating an example of a process for requesting the start of a call in the communication system according to the embodiment. FIG. 15 is a sequence diagram illustrating an example of processing for permitting a request to start a call in the communication system according to the embodiment. FIG. 16 is a diagram illustrating a display example of a start request reception screen. FIG. 17 is a flowchart illustrating an example of the flow of avatar generation processing in the communication system according to the embodiment. FIG. 18 is a diagram illustrating the display operation of the avatar. FIG. 19 is a diagram illustrating the display operation of the avatar. FIG. 20 is a flowchart illustrating an example of the flow of audio direction identification processing of the communication terminal according to the embodiment. FIG. 21 is a flowchart illustrating an example of the flow of attribute information acquisition processing in the communication system according to the embodiment. FIG. 22 is a diagram illustrating an operation for acquiring attribute information. FIG. 23 is a diagram showing an example of a face/avatar/attribute correspondence table. FIG. 24 is a flowchart illustrating an example of the flow of avatar control processing in the communication system according to the embodiment. FIG. 25 is a diagram illustrating the operation of changing the arrangement of avatars based on attribute information. FIG. 26 is an explanatory diagram illustrating the correspondence between speech directions and speakers. FIG. 27 is a diagram showing an example of a speaker/speech direction correspondence table. FIG. 28 is a sequence diagram illustrating an example of the flow of the overall operation of the communication system according to the embodiment. FIG. 29 is a diagram illustrating an example of a screen that prompts the acquisition of attribute information through voice recognition. FIG. 30 is a diagram illustrating an example of a software configuration when using a web application of a communication terminal according to an embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of a communication system, an information processing device, a communication method, and a program according to the present invention will be described in detail below with reference to the drawings. Further, the present invention is not limited to the following embodiments, and the constituent elements in the following embodiments include those that can be easily conceived by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. is included. Furthermore, various omissions, substitutions, changes, and combinations of constituent elements can be made without departing from the gist of the following embodiments.

Furthermore, computer software refers to programs related to computer operations and other information used for processing by a computer that is equivalent to a program (hereinafter, computer software will be referred to as software). Application software is a general term for software used to perform specific tasks among software categories. On the other hand, an operating system (OS) is software that controls a computer and allows application software and the like to use computer resources. The operating system performs basic computer management and control, such as controlling input and output, managing hardware such as memory and hard disks, and managing processes. Application software operates using functions provided by the operating system. A program is a set of instructions to a computer that are combined to produce a single result. Also, what is said to be similar to a program refers to something that cannot be called a program because it is not a direct command to a computer, but has properties similar to a program in that it specifies computer processing. For example, a data structure (a logical structure of data expressed by mutual relationships between data elements) corresponds to a program.

[Embodiment]
(Overall configuration of communication system)
FIG. 1 is a schematic configuration diagram of a communication system according to an embodiment. An outline of the configuration of a communication system 1 according to this embodiment will be described with reference to FIG. 1.

As shown in FIG. 1, the communication system 1 includes a plurality of communication terminals 10aa, 10ab, 10ba, 10bb, 10ca, 10cb, 10da, 10db, displays 120aa, 120ba, 120ca, 120da for some communication terminals, It includes PCs (Personal Computers) 30ab and 30cb connected to some communication terminals, a management system 50, and a program provision system 90, and is constructed to be able to communicate with each other via the communication network 2. .

In addition, although communication terminals 10aa, 10ab, 10ba, 10bb, 10ca, 10cb, 10da, and 10db are shown in FIG. 1, when any communication terminal among these is shown or collectively referred to, it is simply referred to as "communication terminal 10. ”. Further, the plurality of communication terminals 10 included in the communication system 1 shown in FIG. 1 is an example, and the number of communication terminals 10 may be different.

Further, although displays 120aa, 120ba, 120ca, and 120da are shown in FIG. 1, when any of these displays is shown or collectively referred to, they are simply referred to as "display 120." Furthermore, the plurality of displays 120 included in the communication system 1 shown in FIG. 1 is an example, and the number of displays 120 may be different.

The communication terminal 10 is a terminal that transmits and receives various information to and from other devices. The communication terminal 10 establishes a session with another communication terminal 10, and performs a telephone call by transmitting and receiving content data including audio data and image data (video data) in the established session. Thereby, in the communication system 1, a video conference between the plurality of communication terminals 10 is realized. Note that the communication terminal 10 is any one of a dedicated device (video conference terminal) to which a display (display 120 described above) is connected, an electronic whiteboard (interactive whiteboard: IWB), a desktop PC, a notebook PC, a smartphone, a tablet terminal, etc. It may be.

The display 120 is a display device that displays images of participants received by the connected communication terminal 10 via the communication network 2. The display 120 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

For example, the PC 30 is connected to a communication terminal 10 and transmits a screen image to be shared with another communication terminal 10 to the communication terminal 10 and distributes it via the management system 50, thereby sharing the screen image with the other communication terminal 10. This is an information processing device that is placed in a shared state. Note that the PC 30 is not limited to a PC such as a desktop PC or a notebook PC, but may be an information processing device such as a smartphone or a tablet terminal.

The management system 50 is a computer that centrally manages the communication terminals 10. The management system 50 establishes a session between the communication terminals 10, thereby realizing a video conference using a telephone call or the like between the communication terminals 10. When receiving session start request information from a predetermined communication terminal 10, the management system 50 establishes a session between the communication terminal 10 that transmitted the start request information (start request terminal) and the destination terminal, and starts the video conference. start. Therefore, the management system 50 relays content data between the plurality of communication terminals 10 using the established session. Although the management system 50 establishes sessions between the plurality of communication terminals 10, the relay of content data may actually be performed by another relay device. In this embodiment, in order to simplify the explanation, it will be assumed that the management system 50 establishes a session and relays content data.

The program providing system 90 includes an auxiliary storage device (HDD (Hard Disk Drive), etc.) that stores terminal programs for realizing various functions or means in the communication terminal 10, and provides terminal programs to the communication terminal 10. This is a computer that provides programs (such as communication application A to be described later). The program providing system 90 also stores programs in the auxiliary storage device for making the management system 50 etc. implement various functions or means, and can send the corresponding programs to the management system 50 etc.

As shown in FIG. 1, the communication network 2 is constructed to include, for example, LANs (Local Area Networks) 2a to 2d, private lines 2ab and 2cd, and the Internet 2i. Note that the communication network 2 is not limited to the configuration shown in FIG. 1, and may include other network devices, and may have locations where wireless communication is performed in addition to wired communication. .

LANs 2a to 2d and private lines 2ab and 2cd include routers 70a to 70d, 70ab and 70cd, respectively. The routers 70a to 70d, 70ab, and 70cd are network devices that select optimal routes for communication data.

The communication terminals 10 (10aa, 10ab, . . . ) and the router 70a are communicably connected via the LAN 2a. Furthermore, the communication terminals 10 (10ba, 10bb, . . . ) and the router 70b are communicably connected via the LAN 2b. Further, the LAN 2a, the LAN 2b, and the router 70ab are communicably connected by a dedicated line 2ab, and are constructed within the region A.

On the other hand, the communication terminals 10 (10ca, 10cb, . . . ) and the router 70c are communicably connected via the LAN 2c. Further, the communication terminals 10 (10da, 10db, . . . ) and the router 70d are communicably connected via the LAN 2d. Further, LAN2c, LAN2d, and router 70cd are communicably connected by dedicated line 2cd, and are constructed within area B.

The networks in region A and region B are communicably connected via the Internet 2i by routers 70ab and 70cd, respectively.

Furthermore, the management system 50 and the program providing system 90 are communicably connected to each communication terminal 10 via the Internet 2i. Note that the management system 50 and the program providing system 90 may be installed in area A or area B, or may be installed in an area other than these.

In addition, in FIG. 1, four sets of numbers shown near each communication terminal 10, management system 50, each router 70, and program providing system 90 are simplified representations of IP (Internet Protocol) addresses in general IPv4. It is shown in For example, it is assumed that the IP address of the communication terminal 10aa is "1.2.1.3". Note that IPv6 may be used instead of IPv4, but to simplify the explanation, IPv4 will be used in the explanation.

Note that the configuration of the communication system 1 shown in FIG. 1 is an example, and is not limited to this configuration. That is, the number of each device and system shown in FIG. 1 is not limited to the number shown in FIG. In addition, although FIG. 1 shows the network configuration of two regions, regions A and B, the network may be within the same region, or it may be a structure in which three or more regions are connected by a network. Good too.

(Hardware configuration of communication terminal)
FIG. 2 is a diagram illustrating an example of the hardware configuration of a communication terminal (video conference terminal) according to the embodiment. FIG. 3 is a diagram showing an example of the hardware configuration of the communication terminal (electronic whiteboard) according to the embodiment. First, with reference to FIG. 2, the details of the hardware configuration when the communication terminal 10 according to this embodiment is a video conference terminal will be described.

As shown in FIG. 2, the communication terminal 10 according to the present embodiment includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, and an auxiliary storage device 105. It includes a media drive 107 and an input device 108.

CPU 101 is an arithmetic device that controls the overall operation of communication terminal 10 . The ROM 102 is a non-volatile storage device that stores programs for the communication terminal 10 (such as communication application A to be described later). The RAM 103 is a volatile storage device used as a work area for the CPU 101.

The auxiliary storage device 105 is a nonvolatile storage device such as an HDD or an SSD (Solid State Drive) that stores various data such as image data, audio data, and video data. The media drive 107 is a device that controls reading and writing of data to and from the medium 106, such as a flash memory, under the control of the CPU 101. Media 106 is a storage device that is detachable from communication terminal 10 . Note that the medium 106 is not limited to a flash memory as long as it is a nonvolatile memory that reads and writes data under the control of the CPU 101, and may also be an EEPROM (Electrically Erasable and Programmable ROM) or the like.

The input device 108 is a device for inputting various information such as a mouse or a keyboard, or a button such as an operation button or a power button.

The communication terminal 10 also includes a network I/F 111, an image sensor I/F 113, an audio input/output I/F 116, a USB (Universal Serial Bus) I/F 117, and a display I/F 119.

The network I/F 111 is an interface for communicating data using the communication network 2. The network I/F 111 is, for example, a NIC (Network Interface Card) that is compliant with TCP (Transmission Control Protocol)/IP (Internet Protocol).

The image sensor I/F 113 is an interface for transmitting image data to and from the camera 112 that images a subject and obtains image data under the control of the CPU 101. The camera 112 includes a lens and a solid-state image sensor that converts light into electric charge to digitize an image (video) of a subject. The camera 112 is connected to the image sensor I/F 113 by a cable 112c. As the solid-state image sensor, a CMOS (Complementary Metal Oxide Semiconductor), a CCD (Charge Coupled Device), or the like is used.

The audio input/output I/F 116 inputs and outputs audio signals (audio data) to and from the smart speaker 114, which has a microphone 114a for inputting audio and a speaker 114b (output device) for outputting audio, under the control of the CPU 101. It is an interface to process. The smart speaker 114 is a device that includes a microphone 114a constituted by a microphone array, and specifies the direction of the sound by performing sound processing on the sound input from each microphone. Note that the smart speaker 114 does not necessarily need to be a smart speaker as long as it is a device that is equipped with a microphone array and can specify the direction of sound. Smart speaker 114 is connected to audio input/output I/F 116 via cable 114c.

The USB I/F 117 is a USB standard interface for connecting with an external device (for example, a PC, etc.) and performing data communication.

The display I/F 119 is an interface for transmitting image data to an external display 120 (display device) under the control of the CPU 101. Display 120 is connected to display I/F 119 by cable 120c. The cable 120c may be a cable for analog RGB (VGA) signals, or may be a cable for component video, such as HDMI (registered trademark) (High-Definition Multimedia Interface) or DVI (Digital Video Interactive) signal. It may be a cable for

The above-mentioned CPU 101, ROM 102, RAM 103, auxiliary storage device 105, media drive 107, input device 108, network I/F 111, image sensor I/F 113, audio input/output I/F 116, USB I/F 117, and display I/F 119, They are communicably connected to each other by bus lines 110 such as an address bus and a data bus.

Note that the hardware configuration of the communication terminal 10, which is a video conference terminal, shown in FIG. 2 is an example, and may include components other than those shown in FIG. Further, the camera 112 and the smart speaker 114 may be integrally provided in the communication terminal 10, or at least one of the camera 112 and the smart speaker 114 may be a separate external device. You can. Furthermore, although the display 120 is shown as a display that is attached externally to the communication terminal 10 in FIG. 2, it is not limited to this, and may be provided integrally with the communication terminal 10.

Next, with reference to FIG. 3, the details of the hardware configuration when the communication terminal 10 according to this embodiment is an electronic whiteboard will be described.

As shown in FIG. 3, the communication terminal 10 according to the present embodiment includes a CPU 201, a ROM 202, a RAM 203, an SSD 204, a network I/F 205, and an external device connection I/F 206.

The CPU 201 is a calculation device that controls the operation of the communication terminal 10 as a whole. The ROM 202 is a nonvolatile storage device that stores the CPU 201 and programs used to drive the CPU 201 such as an IPL (Initial Program Loader). The RAM 203 is a volatile storage device used as a work area for the CPU 201.

The SSD 204 is a nonvolatile storage device that stores various data such as image data, audio data, operation data, and programs for the communication terminal 10. Note that a nonvolatile storage device such as an HDD may be used instead of the SSD 204.

The network I/F 205 is an interface for communicating data using the communication network 2. The network I/F 205 is, for example, a NIC based on TCP/IP.

The external device connection I/F 206 is an interface such as a USB standard for connecting various external devices. In this case, the external devices include, for example, the USB memory 230, the smart speaker 240, and the camera 260.

The smart speaker 240 has a microphone 241 that inputs audio, and a speaker 242 (output device) that outputs audio. The smart speaker 240 is a device that is equipped with a microphone 241 constituted by a microphone array, and can specify the direction of the sound by performing sound processing on the sound input from each microphone. Note that the device does not necessarily need to be a smart speaker as long as it is equipped with a microphone array and can specify the direction of sound.

The camera 260 includes a lens and a solid-state image sensor that converts light into electric charges to digitize an image (video) of a subject. As the solid-state image sensor, CMOS, CCD, etc. are used.

The communication terminal 10 also includes a capture device 211, a GPU (Graphics Processing Unit) 212, a display controller 213, a display 214 (display device), a sensor controller 215, a contact sensor 216, an electronic pen controller 217, It includes a power switch 222 and a selection switch 223.

The capture device 211 is a device that displays video information as a still image or a moving image on the display of the external PC 270.

The GPU 212 is an arithmetic unit specialized for image processing. The display controller 213 is a controller that controls and manages screen display in order to output the output image from the GPU 212 to the display 214 or the like.

The sensor controller 215 is a controller that controls processing of the contact sensor 216. The contact sensor 216 is a sensor that inputs and detects coordinates using an infrared cutoff method. This method of inputting and detecting coordinates is such that two light receiving and emitting devices installed at both ends of the upper side of the display 214 emit a plurality of infrared rays in parallel to the display 214. This is a method of receiving light that is reflected by a reflecting member and returns along the same optical path as the light emitted by the light receiving element. The contact sensor 216 detects that the electronic pen 290, the user's hand H, etc. are in contact with the display 214. The contact sensor 216 outputs the ID of the infrared rays emitted by the two light receiving and emitting devices blocked by the object to the sensor controller 215, and the sensor controller 215 identifies the coordinate position that is the contact position of the object.

Note that the contact sensor 216 is not limited to the infrared cutoff type, but can also be a capacitive type touch panel that identifies the touch position by detecting changes in capacitance, or a touch sensor that identifies the touch position by voltage changes between two opposing resistive films. Various detection means may be used, such as a resistive film type touch panel that detects the electromagnetic induction generated when a touch object contacts the display section, and an electromagnetic induction type touch panel that specifies the contact position.

The electronic pen controller 217 is a controller that communicates with the electronic pen 290 to determine whether or not the display 214 has been touched with the tip of the pen or the butt of the pen. Note that the electronic pen controller 217 may determine whether or not a portion of the electronic pen 290 that the user grips and other portions of the electronic pen are touched, in addition to the pen tip and pen tail of the electronic pen 290.

The power switch 222 is a switch for switching the power of the communication terminal 10 on and off. The selection switch 223 is, for example, a group of switches for adjusting the brightness, shade, etc. of the display on the display 214.

The above-mentioned CPU 201, ROM 202, RAM 203, SSD 204, network I/F 205, external device connection I/F 206, capture device 211, GPU 212, sensor controller 215, electronic pen controller 217, power switch 222, and selection switch 223 are an address bus and data They are communicably connected to each other by a bus line 210 such as a bus.

Note that the hardware configuration of the communication terminal 10, which is an electronic whiteboard, shown in FIG. 3 is an example, and may include components other than those shown in FIG. Further, the camera 260 and the smart speaker 240 may be integrally provided in the communication terminal 10, or at least one of the camera 260 and the smart speaker 240 may be a separate external device. You can.

(Hardware configuration of management system and program provision system)
FIG. 4 is a diagram illustrating an example of the hardware configuration of the management system and program providing system according to the embodiment. The details of the hardware configuration of the management system 50 and the program providing system 90 will be described with reference to FIG. 4.

First, the hardware configuration of the management system 50 will be described with reference to FIG. 4. As shown in FIG. 4, the management system 50 includes a CPU 301, a ROM 302, a RAM 303, an auxiliary storage device 305, a media drive 307, a display 308, a network I/F 309, a keyboard 311, a mouse 312, A DVD (Digital Versatile Disc) drive 314 is provided.

The CPU 301 is a calculation device that controls the operation of the management system 50 as a whole. ROM 302 is a nonvolatile storage device that stores programs for management system 50. RAM 303 is a volatile storage device used as a work area for CPU 301.

The auxiliary storage device 305 is a storage device such as an HDD or an SSD that stores various data such as an authentication management DB 5001, a terminal management DB 5002, a group management DB 5003, and a session management DB 5004, which will be described later. The media drive 307 is a device that controls reading and writing of data to and from a recording medium 306 such as a flash memory under the control of the CPU 301 .

The display 308 is a display device configured with a liquid crystal, an organic EL, or the like that displays various information such as a cursor, menu, window, characters, or images. The network I/F 309 is an interface for communicating data using the communication network 2. The network I/F 309 is, for example, a NIC based on TCP/IP.

The keyboard 311 is an input device for selecting letters, numbers, various instructions, moving a cursor, and the like. The mouse 312 is an input device for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like.

The DVD drive 314 is a device that controls reading and writing of data to and from a DVD 313 such as a DVD-ROM or a DVD-R (Digital Versatile Disk Recordable), which is an example of a removable storage medium.

The above-described CPU 301, ROM 302, RAM 303, auxiliary storage device 305, media drive 307, display 308, network I/F 309, keyboard 311, mouse 312, and DVD drive 314 can communicate with each other via bus lines 310 such as an address bus and a data bus. It is connected to the.

Note that the hardware configuration of the management system 50 shown in FIG. 3 is an example, and it is not necessary to include all the components shown in FIG. 3, or it may include other components.

Note that since the program providing system 90 has the same hardware configuration as the above-described management system 50, a description thereof will be omitted. However, a program for the program providing system 90 for controlling the program providing system 90 is recorded in the ROM 302 .

(Software configuration of communication terminal)
FIG. 5 is a diagram illustrating an example of the software configuration of the communication terminal according to the embodiment. The details of the software configuration of the communication terminal 10 according to this embodiment will be described with reference to FIG. 5.

A communication application A is installed on the communication terminal 10 as a client application. Here, the application means application software. As shown in FIG. 5, an OS (Operating System) 1020 and a communication application A operate on a work area 1010 of the RAM 103 (RAM 203) of the communication terminal 10.

The OS 1020 is basic software that provides basic functions and manages the entire communication terminal 10. The communication application A is an application that operates under the control of the OS 1020 and is used to communicate (call) with other communication terminals 10 .

Note that examples of the communication protocol of the communication application A include SIP (Session Initiation Protocol), H.323, IRC (Internet Relay Chat), Jingle, and the like.

(Configuration of functional blocks of communication system)
FIG. 6 is a diagram illustrating an example of the configuration of functional blocks of the communication system according to the embodiment. The configuration of functional blocks of the communication system 1 according to this embodiment will be described with reference to FIG. 6.

<Configuration of functional blocks of communication terminal>
As shown in FIG. 6, the communication terminal 10 includes a communication section 11, an operation input reception section 12, an imaging section 13, a display control section 14 (an example of a second control section), an audio input section 15, and an audio input section 15. It has an output section 16 (an example of a second control section), a storage/readout section 17, a storage section 18, an authentication request section 19, and a speech direction specifying section 20 (first specifying section).

The communication unit 11 is a functional unit that transmits and receives various data to and from other communication terminals 10 or each system via the communication network 2. The communication unit 11 is realized by execution of a communication application A, which is software, by the CPU 101 shown in FIG. 2 (CPU 201 shown in FIG. 3), and by the network I/F 111 (network I/F 205).

The communication unit 11 indicates the status of each communication terminal 10 as a destination terminal candidate from the management system 50 before the communication terminal 10 establishes a session with another communication terminal 10 and starts a video conference by telephone call. Start receiving each status information. Here, the term "destination terminal candidate" refers to other communications that the communication terminal 10 uses as a destination candidate for use by a video conference partner, that is, a user (participant) who is a video conference partner who can be specified as a session partner. This is the terminal 10. That is, the communication terminal 10 cannot establish a session and cannot conduct a video conference with a communication terminal that has not been set in advance as a candidate for a destination terminal.

In addition, the status information includes the operating status of each communication terminal 10 (online or offline status), and if online, detailed status such as whether it is on a call or on standby (hereinafter referred to as communication status). ). In addition, the status information includes not only the operating status and communication status of each communication terminal 10, but also information such as whether the cable is disconnected from the communication terminal 10, whether audio can be output but not images, or whether audio is not input. Although it may indicate various states such as muted (muted), in the following, a case where the operating state and the communication state are shown will be described as an example.

The communication unit 11 transmits start request information to the management system 50 when the communication terminal 10 operates as a start request terminal. Here, the start request information is information requesting the start of a session used for a video conference. Specifically, the start request information includes information indicating that a start is requested, the terminal ID of the start requesting terminal that is the source of the start request information, and the terminal ID of the destination terminal that is the other party of the session. . The terminal ID is information for identifying the communication terminal 10, and may be stored in the communication terminal 10 in advance or may be determined by being input directly into the communication terminal 10 by the user.

The operation input receiving unit 12 is a functional unit that receives various inputs from the user. The operation input receiving unit 12 is realized by the input device 108 shown in FIG. 2 (the contact sensor 216, the power switch 222, and the selection switch 223 shown in FIG. 3).

For example, when the user turns on the power button of the operation input receiving unit 12 as the input device 108 shown in FIG. 2, the power of the communication terminal 10 is turned on. Further, when the user turns off the power from the on state, the communication unit 11 transmits status information to the management system 50 to the effect that the power of the communication terminal 10 is turned off, and then is completely turned off. Thereby, the management system 50 can grasp that the communication terminal 10 has changed from being powered on to being powered off.

The image capturing unit 13 is a functional unit that captures an image of a subject and obtains image data obtained by capturing the image. The imaging unit 13 executes communication application A, which is software, by the CPU 101 shown in FIG. Realized.

The display control unit 14 is a functional unit that controls the display of image data and the like on the display 120 (display 214). The display control unit 14 is realized by execution of communication application A, which is software, by the CPU 101 shown in FIG. 2 (CPU 201 shown in FIG. 3).

For example, before the communication terminal 10 as a video conference request source starts a video conference call with the communication terminal 10 as a desired destination, the display control unit 14 displays the destination terminal candidates received by the communication unit 11. A destination list including candidate names of each destination terminal is displayed on the display 120 (display 214), reflecting the status information of the destination terminal.

The audio input unit 15 is a functional unit that inputs the audio signal after the participant's (speaker's) audio collected by the microphone array of the microphone 114a (microphone 241) is converted into an audio signal. The audio input unit 15 is realized by the CPU 101 shown in FIG. 2 (the CPU 201 shown in FIG. 3) executing the communication application A, which is software, and the audio input/output I/F 116 (external device connection I/F 206).

The audio output unit 16 is a functional unit that outputs an audio signal to the speaker 114b (speaker 242) and causes the speaker 114b (speaker 242) to output audio. The audio output unit 16 is realized by execution of a communication application A, which is software, by the CPU 101 shown in FIG. 2 (CPU 201 shown in FIG. 3), and by the audio input/output I/F 116 (external device connection I/F 206).

The storage/readout unit 17 is a functional unit that stores various data in the storage unit 18 and performs processing to read out various data stored in the storage unit 18. For example, content data received when making a call to a destination terminal is overwritten and stored in the storage unit 18 each time the content data is received. Among these, an image is displayed on the display 120 (display 214) using the image data before being overwritten, and sound is output from the speaker 114b (speaker 242) using the audio data before being overwritten. The storage/reading unit 17 is realized by execution of communication application A, which is software, by the CPU 101 shown in FIG. 2 (CPU 201 shown in FIG. 3).

The authentication request unit 19 causes the communication unit 11 to log in to the management system 50 via the communication network 2 when the communication terminal 10 is powered on or when the operation input reception unit 12 accepts an authentication request operation. This is a functional unit that transmits authentication request information indicating that authentication of the communication terminal 10 is requested, and the current IP address of the communication terminal 10. The authentication request unit 19 is realized by execution of communication application A, which is software, by the CPU 101 shown in FIG. 2 (CPU 201 shown in FIG. 3).

The speech direction specifying unit 20 is a functional unit that specifies the speech direction (speech direction) based on the audio signal input by the audio input unit 15. Specifically, the audio signal input by the audio input unit 15 includes each audio signal input from each microphone included in the microphone array of the microphone 114a (microphone 241), and the speech direction identifying unit 20 The direction of the sound is determined by performing sound processing on the sound signal. The speech direction specifying unit 20 is realized by execution of a communication application A, which is software, by the CPU 101 shown in FIG. 2 (CPU 201 shown in FIG. 3).

Note that at least one of the display control section 14, authentication request section 19, and speech direction identification section 20 described above is realized by a hardware circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). It may also be used as a

Moreover, each functional unit of the communication terminal 10 shown in FIG. 6 shows the function conceptually, and is not limited to such a configuration. For example, a plurality of functional units illustrated as independent functional units in the communication terminal 10 shown in FIG. 6 may be configured as one functional unit. On the other hand, the function of one functional unit of the communication terminal 10 shown in FIG. 6 may be divided into a plurality of functions and configured as a plurality of functional units.

<Configuration of functional blocks of management system>
As shown in FIG. 6, the management system 50 includes a communication unit 51, an authentication unit 52, a status management unit 53, a terminal extraction unit 54, a terminal status acquisition unit 55, a session control unit 56, and a storage/readout unit 56. It has a section 57 and a storage section 58. The management system 50 further includes a face detection unit 61 (detection unit), a comparison unit 62, a generation unit 63, an attribute information request unit 64 (request unit), an association unit 65 (second identification unit), It has a text conversion section 66, an extraction section 67, a registration section 68, and a display control section 69 (first control section, control section). The storage unit 58 is realized by the auxiliary storage device 305 shown in FIG. 4, and stores an authentication management DB 5001, a terminal management DB 5002, a group management DB 5003, and a session management DB 5004, as shown in FIG. Each table managed in each DB stored in the storage unit 58 will be described below.

<<Authentication management table>>
FIG. 7 is a diagram showing an example of an authentication management table.

The storage unit 58 stores an authentication management DB 5001 including an authentication management table shown in FIG. In the authentication management table, a password is managed in association with a user ID that identifies a user (participant) using the communication terminal 10 that performs login authentication. Here, the password is information used to authenticate login. For example, the authentication management table shown in FIG. 7 shows that the password associated with the user ID "A_10aa" is "aaaa".

Note that the user ID is identification information such as letters, symbols, numbers, or various marks used to uniquely identify the user who uses the communication terminal 10. For example, the user ID is identification information such as an e-mail address used by the user. You can.

<<Terminal management table>>
FIG. 8 is a diagram showing an example of a terminal management table.

The storage unit 58 stores a terminal management DB 5002 including a terminal management table shown in FIG. In the terminal management table, for each terminal ID of each communication terminal 10, the terminal name, the user ID of the user who logged in to each communication terminal 10, the operating status of each communication terminal 10, the communication status with other communication terminals 10, and each The IP address of the communication terminal 10 is associated and managed.

Here, the operating state includes online, which is a state in which the power is turned on and communication is possible or in progress, and offline, which is a state in which the power is not turned on and communication is not possible. Furthermore, the communication status includes, for example, "Calling", "Ringing", "Accepted", "Busy", and "None". "Calling" indicates a state in which another communication terminal 10 is called, that is, a state in which start request information for establishing a session used for a video conference is sent to another communication terminal 10 and a response is waited for. show. "Ringing" indicates a state of being called by another communication terminal 10, that is, a state of receiving start request information from another communication terminal 10 and not completing a response to the received start request information. "Accepted" indicates a state in which a permission response has been completed in response to start request information from another communication terminal 10, but session establishment has not been completed, and a state in which permission has been granted in response to start request information sent by the own terminal. Indicates that the reception of the response has been completed, but the session establishment has not been completed. "Busy" indicates a state in which a session with another communication terminal 10 has been established and a call is being made by communicating content data in a video conference. “None” indicates that the communication terminal 10 is not communicating with other communication terminals 10 and is on standby.

For example, in the terminal management table shown in FIG. 8, the communication terminal 10ad with the terminal ID "10ad" has the terminal name "Japan Tokyo Office AD Terminal" and the user ID of the logged-in user is "C_10ad". The operating state is "online" and the communication state is "Ringing", which indicates that another communication terminal 10 is calling, and the IP address of this communication terminal 10ad is "1.2.1.6". It is shown.

<<Group management system>>
FIG. 9 is a diagram showing an example of a group management table.

The storage unit 58 stores a group management DB 5003 including a group management table shown in FIG. In the group management table, for each video conference group registered in advance in the management system 50, the terminal IDs of the communication terminals 10 included in the group are managed. That is, in the group management table, a group ID for identifying a group and a terminal ID of a communication terminal 10 included in the group are managed in association with each other.

For example, in the group management table shown in FIG. 9, a group with a group ID of "G002" is shown to include communication terminals 10 with terminal IDs of "10ac", "10ca", and "10cb".

<<Session management table>>
FIG. 10 is a diagram showing an example of a session management table.

The storage unit 58 stores a session management DB 5004 including a session management table shown in FIG. In the session management table, for each session ID for identifying the session in which content data is communicated between the communication terminals 10, the terminal ID of the session start request terminal and the other party in the start request information for establishing the session, The terminal ID of the designated destination terminal is associated and managed.

For example, in the session management table shown in FIG. 10, the session identified by the session ID "se1" is between the start requesting terminal (communication terminal 10aa) with the terminal ID "10aa" and the destination terminal (communication terminal 10aa) with the terminal ID "10db". This indicates that the connection has been established with the communication terminal 10db).

Although the information managed in each table shown in FIGS. 7 to 10 is in table format, the information is not limited to this, and as long as each piece of managed information can be associated, it can be in table format. It is not limited to.

Returning to FIG. 6, the description of the functional blocks of the management system 50 will be returned.

The communication unit 51 is a functional unit that transmits and receives various data to and from the communication terminal 10 or other systems via the communication network 2. The communication unit 51 is realized by the execution of a program by the CPU 301 and the network I/F 309 shown in FIG.

The authentication unit 52 uses the user ID and password included in the authentication request information received via the communication unit 51 as a search key, searches the authentication management table (see FIG. 7) in the storage unit 58, and searches the authentication management table. This is a functional unit that performs user authentication by determining whether the same user ID and password are managed in the same user ID and password. The authentication unit 52 is realized by executing a program by the CPU 301 shown in FIG.

The status management unit 53 is a functional unit that manages the operating status and communication status of the terminal management table shown in FIG. In order to manage the operating state of the communication terminal 10 that has requested login authentication, the state management unit 53 stores the terminal ID of this communication terminal 10 and the user of the participant who is logged in to the communication terminal 10 in the terminal management table. The ID, the operating state of the communication terminal 10, and the IP address of the communication terminal 10 are stored and managed in association with each other.

When the communication terminal 10 changes from the off state to the on state by the user's operation of the input device 108 (power switch 222), the state management unit 53 controls the state management unit 53 based on the information sent from the communication terminal 10 to turn on the power. , updates the operating status of the terminal management table from offline to online. In addition, when the communication terminal 10 changes from the on state to the off state by the user's operation of the input device 108 (power switch 222), the state management unit 53 responds to information sent from the communication terminal 10 indicating that the power is to be turned off. Based on this, the operating status of the terminal management table is updated from online to offline.

The state management unit 53 is realized by executing a program by the CPU 301 shown in FIG.

The terminal extraction unit 54 searches the group management table shown in FIG. 9 using the terminal ID of the target terminal to be processed, such as the communication terminal 10 that has requested login authentication, as a search key, and can make a call to the target terminal. That is, it is a functional unit that reads the terminal ID of a destination terminal candidate (communication terminal 10 of the same group) with which a session can be established. The terminal extraction unit 54 is realized by executing a program by the CPU 301 shown in FIG.

The terminal status acquisition unit 55 is a functional unit that searches the terminal management table shown in FIG. 8 using the terminal ID as a search key, and reads out the operating status and communication status for each terminal ID. Thereby, the terminal status acquisition unit 55 can acquire the operating status and communication status of a candidate destination terminal that can communicate with the communication terminal 10 that has made the login authentication request. The terminal status acquisition unit 55 is realized by executing a program by the CPU 301 shown in FIG.

The session control unit 56 is a functional unit that stores and manages the generated session ID, the terminal ID of the start requesting terminal, and the terminal ID of the destination terminal in association with each other in the session management table shown in FIG. Session control unit 56 performs control for establishing a session between communication terminals 10. The session control unit 56 is realized by executing a program by the CPU 301 shown in FIG.

The storage/reading unit 57 is a functional unit that stores information in various tables in the storage unit 58 and performs processing to read information from various tables stored in the storage unit 58. The storage/reading unit 57 is realized by executing a program by the CPU 301 shown in FIG.

The face detection unit 61 detects the facial image of the participant in the image from the video data received by the communication unit 51, digitizes facial features, and calculates feature values (hereinafter sometimes referred to as face detection information). ). For example, features such as the eyes, eyebrows, nose, and mouth of the detected face are digitized to obtain feature values, and IDs (identification information) such as "M4" and "EL2" are assigned to each feature value. Further, the position of each face part is determined as a feature point, and the position, distance between the feature points, etc. are also included as feature values in the face detection information. The face detection unit 61 also calculates the center coordinates of the image of the participant's face detected in the video data. Although the face detection unit 61 detects the face image of the participant, it is not limited to this, and the detection target may be the upper body including the face of the participant, etc., which cannot be used to identify the participant. Any part that is possible is fine. The face detection unit 61 is realized by executing a program by the CPU 301 shown in FIG.

The comparison unit 62 refers to a face/avatar/attribute correspondence table (see FIG. 23) stored in the storage unit 58, which will be described later, and determines whether the face detection information acquired by the face detection unit 61 corresponds to a face/avatar/attribute. This is a functional unit that compares whether or not it matches the face recognition information registered in the table. Furthermore, in order for the comparison unit 62 to determine that the face detection information and the face recognition information in the face/avatar/attribute correspondence table match, the face detection information and the face recognition information do not necessarily have to match completely. However, if they approximate to a certain degree, they may be determined to match. The comparison unit 62 is realized by executing a program by the CPU 301 shown in FIG.

The generation unit 63 represents the participant corresponding to the face detection information based on the avatar information corresponding to the face recognition information determined by the comparison unit 62 to match the face detection information in the face/avatar/attribute correspondence table. This is a functional unit that generates avatars. Here, avatar information is information used to generate an avatar, for example, information for generating an avatar that resembles a person based on a person's face image, or information on the face image itself. It may be. Alternatively, the avatar information does not necessarily have to be information for generating an avatar that is recognizable as the participant's face, but can be used to generate an avatar that can replace each participant (for example, an animal avatar, etc.) It may be information of Alternatively, the avatar information may be type information indicating which information is used to generate an avatar from among several existing pieces of information for generating an avatar. The generation unit 63 is realized by executing a program by the CPU 301 shown in FIG.

The attribute information requesting unit 64 is a functional unit that transmits an instruction to the communication terminal 10 to request a participant to speak about attribute information. Here, the attribute information is information indicating the attributes of the participants, such as the company name and affiliation to which the participants in the video conference belong, and the positions and names of the participants. The attribute information request unit 64 is realized by executing a program by the CPU 301 shown in FIG.

The association unit 65 is a functional unit that associates the speech direction of the participant's voice received via the communication unit 51 with the face image detected by the face detection unit 61, that is, the participant. Note that the method of associating audio and facial images (i.e., participants) by the associating unit 65 is not limited to the above-mentioned operation; The voice and the participant may be associated with each other based on the participant's lip movements indicated by the video data and the timing of the audio output of the audio data. The association unit 65 is realized by executing a program by the CPU 301 shown in FIG.

The text conversion unit 66 is a functional unit that converts voice data received via the communication unit 51 into text using known voice recognition technology. The text converting unit 66 is realized by executing a program by the CPU 301 shown in FIG.

The extraction unit 67 is a functional unit that extracts keywords that match registered words that are registered in advance as candidates indicating attribute information from the text converted into text by the text conversion unit 66. The registered words may be stored in the storage unit 58 in advance, for example. For example, as shown in the face/avatar/attribute correspondence table in FIG. 23, which will be described later, if company name, affiliation, position, and name are registered as attribute information, the registered words include the registration related to the company name. A registered word, a registered word related to affiliation, a registered word related to position, and a registered word related to name may be prepared. Furthermore, the attributes handled as attribute information are not limited to company name, department, position, and name, and may include other attributes (for example, role, age, etc.). The extraction unit 67 is realized by executing a program by the CPU 301 shown in FIG.

The registration unit 68 uses the keyword extracted by the extraction unit 67 as attribute information of the participant corresponding to the audio data converted into text by the text conversion unit 66, and performs face recognition using the comparison unit 62 in the face/avatar/attribute correspondence table. This is a functional unit that registers in association with face detection information of a participant that is determined to match the information. The registration unit 68 is realized by executing a program by the CPU 301 shown in FIG.

The display control unit 69 is a functional unit that performs display control for the avatar generated by the generation unit 63 based on attribute information corresponding to the avatar. Specifically, the display control unit 69 arranges the avatars of participants in the video conference who belong to the same company based on the attribute information so that they are in the same row, and arranges them in order of job title. Then, video data is generated. Further, the display control unit 69 may display participants at each base of the video conference in the same display area, or may display participants in separate display areas for each base. Further, the display control unit 69 may arrange the avatars in the order of their positions, or alternatively, display the position names near the avatars. At this time, the display control unit 69 may further display the name, affiliation, etc. near the avatar. In this way, by arranging the positions in order and displaying the position names, etc., it is possible to at least know which participants are superior to other participants (avatars) at bases other than one's own base. The position of each participant can be understood, and the meeting can proceed smoothly.

Note that when the display control unit 69 generates video data by changing the arrangement of avatars based on attribute information, the closer the avatar is placed in the video data, the higher the sound pressure of the corresponding audio data. You may perform processing to raise the level. This makes it possible to enhance the sense of realism in a video conference even when the video data displays an avatar.

In addition, when the display control unit 69 receives audio data from the communication terminal 10 via the communication unit 51 and also receives information on the speaking direction, the display control unit 69 displays the state in which the avatar corresponding to the speaking direction is speaking. Generate the video data shown. With this, when a participant is looking at a display displaying an avatar of a participant at the other site, it is possible to recognize whether that participant (avatar) is speaking. Here, the state in which the avatar is uttering audio means, for example, if the avatar has a mouth, the state in which the mouth is moving in accordance with the output of audio data, or the state in which the avatar is in a state in which the mouth is moving in accordance with the output of audio data. Examples include a state in which the user appears to be speaking by moving the button up and down. Furthermore, the display control unit 69 may reflect the participant's facial expression, line of sight, etc. on the corresponding avatar in real time, not only when the participant is speaking.

Further, when including an avatar in the video data, the display control unit 69 may use a real image background or a virtual background as the background image. Furthermore, when using the background of the real image as the background image, the display control unit 69 may display only the avatar clearly and blur the background of the real image. By displaying a virtual background or blurring the background of a real image, participants in a video conference can participate in a video conference without worrying about the state of the conference room or other room they are in. Therefore, even if confidential information is included in the background, it is possible to suppress the prevention of leakage of the information.

The display control unit 69 is realized by executing a program by the CPU 301 shown in FIG. In this way, by using the video data of the avatar rather than the video data of the participant's real image by the display control unit 69, it is possible to transfer the data at a lower frame rate than in the case of the real image, thereby reducing the amount of data communication. can be reduced, and the specification requirements for the display can be lowered.

Note that the above-mentioned authentication section 52, state management section 53, terminal extraction section 54, terminal state acquisition section 55, session control section 56, face detection section 61, comparison section 62, generation section 63, attribute information request section 64, association At least one of the section 65, the text conversion section 66, the extraction section 67, the registration section 68, and the display control section 69 may be realized by a hardware circuit such as ASIC or FPGA.

Furthermore, the functions of each functional unit of the management system 50 shown in FIG. 6 are conceptually shown, and the structure is not limited to this. For example, a plurality of functional units illustrated as independent functional units in the management system 50 shown in FIG. 6 may be configured as one functional unit. On the other hand, the function of one functional section of the management system 50 shown in FIG. 6 may be divided into a plurality of parts and configured as a plurality of functional parts.

Further, the functional units included in the communication terminal 10 shown in FIG. It's good. Further, the functional units included in the communication terminal 10 and the management system 50 may be realized by devices other than the communication terminal 10 and the management system 50. For example, the detection of a participant's face by the face detection unit 61 of the management system 50 in the avatar generation process, the text conversion of voice data by the text conversion unit 66 of the management system 50 in the attribute information acquisition process, etc. are performed by the management system 50 and not by the management system 50. It may be a function that the terminal 10 has. Further, the authentication process by the authentication unit 52 of the management system 50 may be performed by a device other than the communication terminal 10 and the management system 50.

(Status of sending and receiving content data and various management information)
FIG. 11 is a diagram showing sessions established for transmitting and receiving content data and various types of management information in the communication system according to the embodiment. A session established for transmitting and receiving content data and various management information in the communication system 1 will be described with reference to FIG. 11.

As shown in FIG. 11, in the communication system 1, a management information session sei is established for transmitting and receiving various types of management information between the start request terminal, destination terminal A, and destination terminal B via the management system 50. Established. Further, a content data session sed for transmitting and receiving image data, audio data, etc. is established between the start request terminal, destination terminal A, and destination terminal B via the management system 50. That is, the content data session sed is a session that is directly used in the video conference. Note that this session concept is just an example; for example, an image data session may be divided by resolution.

(Transmission and reception processing of each management information in the preparation stage before the communication terminal starts a call)
FIG. 12 is a sequence diagram illustrating an example of a preparation stage process including an authentication process for a communication terminal to start a call in the communication system according to the embodiment. FIG. 13 is a diagram showing a display example of the destination list. With reference to FIGS. 12 and 13, the transmission and reception process of each piece of information in the preparation stage before the communication terminal 10aa starts a call will be described. Note that FIG. 12 shows a process in which various types of management information are transmitted and received by the management information session sei.

<Step S21>
First, when the user of the communication terminal 10aa turns on the power by operating the input device 108 shown in FIG. do.

<Step S22>
Then, the authentication request unit 19 of the communication terminal 10aa sends authentication request information indicating a login authentication request from the communication unit 11 to the management system 50 via the communication network 2, triggered by the above-mentioned power-on of the communication terminal 10aa. The IP address of the communication terminal 10aa is transmitted. This authentication request information includes a terminal ID for identifying the communication terminal 10aa, which is the own terminal as the start requesting terminal, a user ID of a user logging into the communication terminal 10aa, and a password. The terminal ID, user ID, and password are data read from the storage section 18 by the storage/readout section 17 of the communication terminal 10aa and sent to the communication section 11. Further, when authentication request information is transmitted from the communication terminal 10aa to the management system 50, the management system 50, which is the receiving side, can grasp the IP address of the communication terminal 10aa, which is the sending side.

<Step S23>
Next, the authentication unit 52 of the management system 50 searches the authentication management table (see FIG. 7) using the user ID and password included in the authentication request information received via the communication unit 51 as a search key, and performs authentication. User authentication is performed by determining whether the same user ID and password are managed in the management table.

<Step S24-1>
If the authentication unit 52 determines that the login authentication request is from the communication terminal 10 that has legitimate usage authority, the state management unit 53 of the management system 50 writes the following information in the terminal management table (see FIG. 8). The user ID and the IP address of the communication terminal 10aa are stored in association with each record indicated by the terminal ID and terminal name of the communication terminal 10aa. As a result, the terminal ID "10aa" of the communication terminal 10aa is managed in association with the user ID "A_10aa" and the IP address "1.2.1.3" in the terminal management table.

<Step S24-2>
Subsequently, the state management unit 53 sets the operating state "online" and the communication state "None" of the communication terminal 10aa, and adds the operating state to the record indicated by the terminal ID and terminal name of the communication terminal 10aa in the terminal management table. and the communication status are stored in association with each other. As a result, the terminal ID "10aa" of the communication terminal 10aa is managed in association with the operating state "online" and the communication state "None" in the terminal management table.

<Step S25>
Then, the communication unit 51 of the management system 50 transmits authentication result information indicating the user authentication result obtained by the authentication unit 52 to the start requesting terminal (communication Terminal 10aa). In the present embodiment, the following description will be continued on the assumption that the communication terminal 10aa has been authenticated by the authentication unit 52 as a terminal used by a user who has legitimate usage authority.

<Step S26>
When the communication terminal 10aa receives authentication result information indicating that the terminal is authenticated to be used by a user with legitimate usage authority, the communication unit 11 transmits the information to the management system 50 via the communication network 2. , sends destination list request information indicating that a destination list is requested.Thereby, the communication unit 51 of the management system 50 receives the destination list request information.

<Step S27>
Next, the terminal extraction unit 54 of the management system 50 searches the group management table (see FIG. 9) using the terminal ID "10aa" of the start requesting terminal (communication terminal 10aa) as a search key, and the start requesting terminal makes a call. In other words, the terminal ID of the destination terminal candidate belonging to the same group as the start requesting terminal (here, the group with group ID "G001") is extracted. Furthermore, the terminal extraction unit 54 searches the terminal management table using the extracted terminal ID as a search key, and extracts the terminal name corresponding to this terminal ID, that is, the terminal name of the destination terminal candidate. Here, the respective terminal IDs ("10ab", "10ac", "10db") of destination terminal candidates (communication terminals 10ab, 10ac, 10db) corresponding to the terminal ID "10aa" of the start requesting terminal (communication terminal 10aa) are ) and the corresponding terminal names (“Japan Tokyo Office AB Terminal”, “Japan Tokyo Office AC Terminal”, “USA Washington Office DB Terminal”) are extracted.

<Steps S28, S29>
Next, the communication unit 51 of the management system 50 transmits destination list information including the terminal ID and terminal name of the destination terminal candidates extracted by the terminal extraction unit 54 to the start requesting terminal (communication terminal 10aa). As a result, in the start requesting terminal (communication terminal 10aa), the communication section 11 receives the destination list information, and the storage/reading section 17 stores the destination list information in the storage section 18.

In this manner, in this embodiment, the management system 50 centrally manages the destination list information of all communication terminals 10, instead of managing destination list information at each communication terminal 10. As a result, even if a new communication terminal 10 is included in the communication system 1 or a communication terminal 10 already included in the communication system 1 is excluded, each communication The effort of changing destination list information on the terminal 10 side can be saved.

<Step S30>
In addition, the terminal status acquisition unit 55 of the management system 50 searches the terminal management table using the terminal ID (“10ab”, “10ac”, “10db”) of the destination terminal candidate extracted by the terminal extraction unit 54 as a search key. search for. Then, the terminal state acquisition unit 55 reads out the corresponding operating state and communication state for each terminal ID of the destination terminal candidate, thereby obtaining the operating state and communication state of each of the destination terminal candidates (communication terminals 10ab, 10ac, 10db). Get the status.

<Step S31>
Next, the communication unit 51 sends status information including the terminal ID, which is the search key used in step S30, and the operating status and communication status of the corresponding destination terminal candidate, to the start requesting terminal via the communication network 2. Send to. Specifically, the communication unit 51 transmits state information including the terminal ID "10ab" as a search key and the operation state "offline" of the destination terminal candidate (communication terminal 10ab) to the start requesting terminal (communication terminal 10ab). Terminal 10aa). Note that when the operating state is "offline", the state information does not include the communication state. The communication unit 51 also provides status information for all destination terminal candidates, such as status information including the terminal ID “10ac”, the operation status “online” of the destination terminal candidate (communication terminal 10ac), and the communication status “None”. Each piece of information is transmitted to the start request terminal (communication terminal 10aa).

<Step S32>
Next, the storage/reading unit 17 of the start request terminal (communication terminal 10aa) sequentially stores the status information received from the management system 50 in the storage unit 18. Therefore, by receiving the status information of the destination terminal candidates, the start requesting terminal (communication terminal 10aa) can obtain the current operating status and communication status of each of the destination terminal candidates with which the call can be made. .

<Step S33>
Next, the display control unit 14 of the start requesting terminal (communication terminal 10aa) determines the operating status of the destination terminal candidate based on the destination list information stored in the storage unit 18 and the status information of the destination terminal candidate. Create a destination list that reflects the communication status. Then, the display control unit 14 displays a destination list as shown in FIG. 13 at a predetermined timing on the display 120aa shown in FIG.

As shown in FIG. 13, the destination list displayed on the display 120aa includes a terminal ID 1100-2 of destination terminal candidates, a terminal name 1100-3, icons 1100-4a and 1100-4b reflecting status information, etc. including. The icons include an offline icon 1100-4a that indicates that a call cannot be made offline, and a call enabled icon 1100-4b that indicates that a call is possible online. Note that there may be a call-in-progress icon or the like that indicates that the call is in progress online.

If the operating state of the destination terminal candidate is "online" and the communication state is "None," the display control unit 14 assigns a call enabled icon 1100-4b to this destination terminal candidate. Further, when the operating state of the destination terminal candidate is "offline", the display control unit 14 assigns an offline icon 1100-4a to this destination terminal candidate. Note that if the operation status of the destination terminal candidate is "online" and the communication status is other than "None", the display control unit 14 may assign a busy icon to this destination terminal candidate. good.

Note that in other communication terminals 10, similarly to step S21, when the user turns on the power through the input device 108 shown in FIG. , processing similar to the processing of steps S22 to S33 described above is performed.

(Processing when a communication terminal requests the start of communication with another communication terminal)
FIG. 14 is a sequence diagram illustrating an example of a process for requesting the start of a call in the communication system according to the embodiment. Processing when the communication terminal 10 requests the start of communication with another communication terminal 10 will be described with reference to FIG. 14. Note that FIG. 14 shows processing in which various types of management information are transmitted and received entirely by the management information session sei.

In FIG. 14, an example will be described in which the communication terminal 10aa, which was permitted to log in in FIG. 12, transmits start request information, that is, an example in which the communication terminal 10aa operates as a start request terminal. The communication terminal 10aa serving as the start request terminal determines that among the destination terminal candidates, the operating state is "online" and the communication state is "None" based on the state information of the destination terminal candidates received in step S31 of FIG. It is possible to make a telephone conversation with at least one communication terminal 10 among the communication terminals 10 that are ``. For example, the start requesting terminal (communication terminal 10aa) is a communication terminal whose operating state is "online" and whose communication state is "none" according to the state information received in step S31 of FIG. 12 among the destination terminal candidates. You can talk to 10db. Therefore, below, a case will be described in which the user of the start requesting terminal (communication terminal 10aa) selects to start a call with the destination terminal (communication terminal 10db).

It is assumed that before the process shown in FIG. 14 is started, the destination list shown in FIG. 13 is displayed on the display 120aa of the communication terminal 10aa as the start requesting terminal. The user of the start requesting terminal can then select a desired communication partner (destination terminal) from the destination list.

<Step S41>
First, the user of the start request terminal operates the input device 108 of the communication terminal 10aa to select a destination terminal (communication terminal 10db).

<Step S42>
Then, the communication unit 11 of the communication terminal 10aa sends the start request information including the terminal ID "10aa" of the start requesting terminal (communication terminal 10aa) and the terminal ID "10db" of the destination terminal (communication terminal 10db) to the start requesting terminal. It is sent to the management system 50 along with the IP address of. As a result, the communication unit 51 of the management system 50 receives the start request information and grasps the IP address "1.2.1.3" of the start request terminal (communication terminal 10aa) that is the sender. .

<Step S43>
Then, the state management unit 53 selects the terminal in the terminal management DB 5002 based on the terminal ID "10aa" of the start requesting terminal (communication terminal 10aa) and the terminal ID "10db" of the destination terminal (communication terminal 10db) included in the start request information. In the management table, change the communication status field of the record that includes the terminal ID "10aa" and the terminal ID "10db". Specifically, the state management unit 53 changes the communication state of the record including the terminal ID "10aa" in the terminal management table to "Calling". Similarly, the state management unit 53 changes the communication state of the record including the terminal ID "10db" in the terminal management table to "Ringing".

<Step S44>
Then, the session control unit 56 of the management system 50 uses a session for identifying a session (content data session sed) for executing communication with the destination terminal requested by the start requesting terminal (communication terminal 10aa). Generate ID "se1". After generating the session ID, the session control unit 56 stores the session ID "se1" in the session management table (see FIG. 10).

<Step S45>
Next, the session control unit 56 enters the terminal ID "10aa" of the start requesting terminal into the fields of the start requesting terminal terminal ID and the destination terminal terminal ID of the record including the session ID "se1" in the session management table. ”, and the terminal ID “10db” of the destination terminal is stored and managed.

<Step S46>
Next, the communication unit 51 transmits the session ID generated by the session control unit 56 to the start requesting terminal (communication terminal 10aa) via the communication network 2.

<Step S47>
The communication unit 51 also transmits start request information including the terminal ID "10aa" of the start request terminal and the session ID "se1" and the IP address of the management system 50 to the destination terminal. As a result, the destination terminal (communication terminal 10db) receives the start request information and also learns the IP address "1.1.1.2" of the management system 50.

(Processing when the destination terminal receives a response permitting the start of communication with the start requesting terminal)
FIG. 15 is a sequence diagram illustrating an example of processing for permitting a request to start a call in the communication system according to the embodiment. FIG. 16 is a diagram illustrating a display example of a start request reception screen. Referring to FIGS. 15 and 16, the user of the destination terminal that has received the start request information operates the input device 108 to indicate permission to start communication (establishment of a session) with the start request terminal. The processing when the response is accepted will be explained.

<Step S51>
At the start of the transmission/reception process shown in FIG. 15, a start request reception screen 1200-1 (see FIG. 16) indicating that start request information has been received is displayed on the display 214 of the destination terminal (communication terminal 10db).

The start request reception screen 1200-1 shown in FIG. 16 indicates that start request information has been received, and allows the user to specify whether or not to permit the start of communication (establishment of a session) with the start request terminal. It is an interface. The user can confirm that the start request has been received by viewing the start request reception screen 1200-1. The start request acceptance screen 1200-1 includes a "Yes" button 1200-2 for allowing the establishment of a session, and a "No" button 1200-3 for selecting not to permit the establishment of the session.

<Step S52>
When the “Yes” button 1200-2 is pressed by operating the input device 108 of the destination terminal (communication terminal 10db), the operation input reception unit 12 starts communication with the start requesting terminal (communication terminal 10aa). A response indicating that session establishment is permitted is accepted.

<Step S53>
Next, the communication unit 11 of the destination terminal transmits start response information including the terminal ID “10db” of the destination terminal, the terminal ID “10aa” of the start requesting terminal, and the session ID “se1” to the management system 50. .

<Step S54>
When the communication unit 51 of the management system 50 receives the start response information, the state management unit 53 performs terminal management based on the terminal ID “10aa” of the start requesting terminal and the terminal ID “10db” of the destination terminal included in the start response information. In the table, the communication status field portion of the record containing the terminal ID "10aa" and the terminal ID "10db" is changed. Specifically, the state management unit 53 changes the communication state of the record including the terminal ID "10aa" in the terminal management table to "Accepted". Similarly, the status management unit 53 also changes the communication status of the record that includes the terminal ID "10db" in the terminal management table to "Accepted".

<Step S55>
Next, the communication unit 51 transmits start response information including the terminal ID "10db" of the destination terminal (communication terminal 10db) and the session ID "se1" to the start request terminal (communication terminal 10aa).

<Step S56>
Upon receiving this start response information, the start request terminal establishes a session by transmitting the session ID "se1" to the management system 50 using the communication unit 11.

<Step S57>
On the other hand, the destination terminal establishes a session by transmitting the session ID "se1" to the management system 50 using the communication unit 11.

Note that in the operation of extracting the terminal ID for identifying the communication terminals 10 of the same group in FIG. 12 described above and the operation of establishing a session between the communication terminal 10aa and the communication terminal 10db shown in FIGS. may be an operation using the user ID (for example, e-mail address, etc.) of the user who is logged in to each communication terminal, instead of using the terminal ID.

(Avatar generation process)
FIG. 17 is a flowchart illustrating an example of the flow of avatar generation processing in the communication system according to the embodiment. FIGS. 18 and 19 are diagrams illustrating the display operation of the avatar. The flow of processing for generating avatars of participants participating in a video conference in the management system 50 will be described with reference to FIGS. 17 to 19.

<Step S61>
The face detection unit 61 of the management system 50 detects facial images of participants in the video data from the communication terminals 10 participating in the video conference received by the communication unit 51, and determines facial characteristics. The feature values obtained by quantifying the values are obtained as face detection information. Then, the face detection unit 61 calculates the center coordinates of the image of the participant's face detected in the video data. Then, the process moves to step S62.

<Step S62>
The comparison unit 62 of the management system 50 refers to a face/avatar/attribute correspondence table (described later) stored in the storage unit 58 (see FIG. 23, described later), and determines whether the face detection information acquired by the face detection unit 61 is Compare whether it matches the face recognition information registered in the face/avatar/attribute correspondence table. Then, the process moves to step S63.

<Step S63>
As a result of the comparison between the face detection information and the face recognition information by the comparison unit 62, if they match (step S63: Yes), the process moves to step S65, and if they do not match (step S63: No), the process moves to step S64.

<Step S64>
If there is no face recognition information that matches the face detection information acquired by the face detection unit 61 in the face/avatar/attribute correspondence table, the comparison unit 62 uses the face detection information as new face recognition information and uses the face/avatar as new face recognition information. -Register in the attribute correspondence table. At this point, in the face/avatar/attribute correspondence table, the record of newly registered face recognition information has no corresponding (associated) avatar information and attribute information registered therein. Then, the process moves to step S65.

<Step S65>
The generation unit 63 of the management system 50 generates face recognition information that matches the face detection information detected by the face detection unit 61 (if it is newly registered as face recognition information in step S64) in the face/avatar/attribute correspondence table. , the face recognition information) is registered. If the associated avatar information is registered (step S65: Yes), the process moves to step S66, and if it is not registered (step S65: No), the process moves to step S67.

<Step S66>
The generation unit 63 acquires avatar information corresponding to the face recognition information determined by the comparison unit 62 to match the face detection information from the face/avatar/attribute correspondence table, and based on the avatar information, generates the face recognition information. Generate an avatar representing the corresponding participant. Then, the display control unit 69 of the management system 50 performs display control on the avatar (an example of participant information) generated by the generation unit 63 based on attribute information corresponding to the avatar.

For example, as shown in FIG. 18(a), at site a, there are participants A to D who participate in a video conference using communication terminal 10a, and at site b, there are participants A to D who participate in the video conference using communication terminal 10b. When it is assumed that there are participants E and F, the display control unit 69 displays all participants at each base participating in the video conference to the communication terminals 10a and 10b, as shown in FIG. 18(b). (Avatar) may be generated and transmitted as if they are in the same location. Alternatively, as shown in FIGS. 19(a) and 19(b), the display control unit 69 may display the avatars generated by the generation unit 63 in such a way that only participants (avatars) from bases other than the own base are displayed. It is also possible to generate video data and transmit it to the communication terminals 10a and 10b. In the example shown in FIG. 19(b), the display control unit 69 displays the video data displayed on the display 214 of the communication terminal 10b of the base b by the participation of a base (here, base a) other than its own base (here, base b). It is also possible to generate video data that shows only persons A to D.

In addition, when the attribute information of the participants is registered in the face/avatar/attribute correspondence table, the display control unit 69 refers to the attribute information and selects participants who belong to the same company among the participants of the video conference. The video data may be generated and transmitted so that the avatars are arranged in the same row and displayed in the order of their positions. Further, the display control unit 69 may arrange the avatars in the order of their positions, or alternatively, display the position names near the avatars. At this time, the display control unit 69 may further display the name, affiliation, etc. near the avatar. In this way, by arranging the positions in order and displaying the position names, etc., it is possible to at least know which participants are superior to other participants (avatars) at bases other than one's own base. The position of each participant can be understood, and the meeting can proceed smoothly. Here, if attribute information corresponding to the participant is not registered in the face/avatar/attribute correspondence table, attribute information acquisition processing, which will be described later with reference to FIG. 21, is executed.

Further, when including an avatar in the video data, the display control unit 69 may use a real image background or a virtual background as the background image. Furthermore, when using the background of the real image as the background image, the display control unit 69 may display only the avatar clearly and blur the background of the real image. By displaying a virtual background or blurring the background of a real image, participants in a video conference can participate in a video conference without worrying about the state of the conference room or other room they are in. Therefore, even if confidential information is included in the background, it is possible to suppress the prevention of leakage of the information.

<Step S67>
In the face/avatar/attribute correspondence table, the face recognition information that matches the face detection information detected by the face detection unit 61 (if the face recognition information is newly registered as face recognition information in step S64, the face recognition information) If the avatar information has not been registered, the generation unit 63 generates new avatar information. For example, it is assumed that default avatar information is stored in the storage unit 58, and the generation unit 63 may use the default avatar information as the new avatar information. Then, the generation unit 63 generates an avatar representing the participant corresponding to the face recognition information based on the newly generated avatar information. The generation unit 63 then registers the generated new avatar information in association with the face recognition information in the face/avatar/attribute correspondence table. The display control of the avatar by the display control unit 69 is similar to the operation described in step S66 above.

The avatar generation process by the management system 50 is executed by the flow of steps S61 to S67 described above.

(Speech direction identification process)
FIG. 20 is a flowchart illustrating an example of the flow of audio direction identification processing of the communication terminal according to the embodiment. Referring to FIG. 20, the flow of speech direction identification processing in the communication terminal 10 will be explained.

<Step S71>
Participants in a video conference speak to communicate with participants at other locations. Then, the process moves to step S72.

<Step S72>
Then, the microphone 114a (microphone 241) of the communication terminal 10 used by the participant who spoke collects the voice spoken by the microphone array and converts it into an audio signal, and the voice input unit 15 of the communication terminal 10 receives the voice. Input (acquire) the signal. Then, the process moves to step S73.

<Step S73>
The speech direction specifying unit 20 of the communication terminal 10 specifies the speech direction of the voice by performing voice processing on the voice signal input by the voice input unit 15. Then, the process moves to step S74.

<Step S74>
The communication unit 11 of the communication terminal 10 receives audio data (audio signal) input by the audio input unit 15, video data captured by the imaging unit 13, and information on the speech direction specified by the speech direction identification unit 20. It is sent to the management system 50. Then, the speech direction specifying process ends.

The speech direction specifying process by the communication terminal 10 is executed through the flow of steps S71 to S74 described above.

(Attribute information acquisition processing)
FIG. 21 is a flowchart illustrating an example of the flow of attribute information acquisition processing in the communication system according to the embodiment. FIG. 22 is a diagram illustrating an operation for acquiring attribute information. FIG. 23 is a diagram showing an example of a face/avatar/attribute correspondence table. The flow of attribute information acquisition processing by the communication system 1 will be described with reference to FIGS. 21 to 23. Note that when the video conference is started, the attribute information of the participants used when the display control unit 69 of the management system 50 displays the avatar is registered in the face/avatar/attribute correspondence table, as explained in FIG. 17 above. Therefore, it is assumed that the attribute information requesting unit 64 transmits an instruction to request speech regarding the attribute information to the communication terminal 10 at the base of the participant whose attribute information is not registered. In this case, upon receiving the instruction via the communication unit 11, the display control unit 14 of the communication terminal 10 causes the display 120 (display 214) to display a message prompting the user to speak the attribute information, for example. In addition to displaying a message prompting the user to utter the attribute information, for example, the audio output unit 16 of the communication terminal 10 may cause the speaker 114b (speaker 242) to output a voice prompting the user to utter the attribute information.

<Step S81>
Participants of the video conference who confirm the message prompting them to speak the attribute information displayed on the display 120 (display 214) of the communication terminal 10 speak attribute information such as their name, the name of the company to which they belong, their affiliation, and position. . In the example shown in FIG. 22, if a message prompting participants E and F at site b to speak their attribute information is displayed, participant E will respond to the message "AAA company, technology" that includes his or her own attribute information. This is BBB from Company XXX.'', and Participant F is shown uttering, ``This is YYY, the manager of XXX Company,'' which includes his own attribute information. The example in FIG. 22 shows a state in which the participant at base b is speaking, but during the avatar generation process of the management system 50 described above, the attribute information of the participant at base a is If it is confirmed that the attribute information is not registered in the correspondence table, the management system 50 also sends an instruction to the communication terminal 10 at the base a to request speech regarding the attribute information. The audio input unit 15 of the communication terminal 10 inputs (obtains) audio data of the voices uttered by the participants, which are collected by the microphone 114a (microphone 241). Furthermore, the speech direction specifying unit 20 of the communication terminal 10 specifies the voice direction (speech direction) based on the voice data input by the voice input unit 15. Then, the communication unit 11 of the communication terminal 10 transmits the voice data input by the voice input unit 15 and the speech direction information specified by the speech direction specifying unit 20 to the management system 50. The communication unit 51 of the management system 50 receives voice data and speech direction information transmitted from the communication terminal 10. Then, the process moves to step S82.

<Step S82>
The text conversion unit 66 of the management system 50 converts the voice data received by the communication unit 51 into text using known voice recognition technology. Then, the process moves to step S83.

<Step S83>
The extraction unit 67 of the management system 50 compares the text converted into text by the text conversion unit 66 with registered words registered in advance in the storage unit 58 as candidates indicating attribute information. Then, the process moves to step S84.

<Step S84>
As a result of the comparison between the text and the registered word by the extraction unit 67, if there is a matching keyword (step S84: Yes), the process moves to step S85, and if there is no matching keyword (step S84: No), the process moves to step S87. .

<Step S85>
As a result of comparing the text converted into text by the text conversion unit 66 with the registered words, the extraction unit 67 extracts the keyword if the text has a keyword that matches the registered word. For example, when there are attributes such as company name, affiliation, position, and name as attribute information as in the face/avatar/attribute correspondence table shown in FIG. 23, the extraction unit 67 extracts keywords corresponding to each attribute from the text. Extract. Then, the process moves to step S86.

<Step S86>
The registration unit 68 of the management system 50 compares the keywords extracted by the extraction unit 67 as participant attribute information corresponding to the audio data converted into text by the text conversion unit 66 in the face/avatar/attribute correspondence table. The face detection information of the participant determined by the unit 62 to match the face recognition information is registered in association with the face detection information of the participant corresponding to the speaking direction. For example, in the face/avatar/attribute correspondence table shown in FIG. 23, the attribute information for participants whose face recognition information is "XXX" (M4) and "YYY" (EL2) is the company name "AAA" and the affiliation "BBB." ”, title “CCC”, and name “DDD” are registered. Then, the attribute information acquisition process ends.

<Step S87>
As a result of comparing the text converted into text by the text conversion unit 66 with registered words, there is no keyword matching the registered word in the text, so the attribute information request unit 64 of the management system 50 determines that the attribute information is registered. An instruction is sent to the communication terminal 10 via the communication unit 51 to request the utterance of the attribute information to the participants who have not yet done so. Then, upon receiving the instruction via the communication unit 11, the display control unit 14 of the communication terminal 10 causes the display 120 (display 214) to display a message prompting the user to speak the attribute information, for example. Then, the process returns to step S81.

Attribute information acquisition processing by the communication system 1 is executed through the flow of steps S81 to S87 described above.

(Avatar control processing)
FIG. 24 is a flowchart illustrating an example of the flow of avatar control processing in the communication system according to the embodiment. FIG. 25 is a diagram illustrating the operation of changing the arrangement of avatars based on attribute information. FIG. 26 is an explanatory diagram illustrating the correspondence between speech directions and speakers. FIG. 27 is a diagram showing an example of a speaker/speech direction correspondence table. The flow of avatar control processing by the communication system 1 will be described with reference to FIGS. 24 to 27.

<Step S91>
When the avatar generation process shown in FIG. 17 is executed, the display control unit 69 performs display control of the avatar generated by the generation unit 63 of the management system 50 based on the attribute information corresponding to the avatar. Video data is generated. Details of the display control by the display control unit 69 are as described in step S66 of FIG. 17. Here, the communication terminal 10 at the own base receives display-controlled video data about the avatar generated for the participant at the other base from the management system 50, and displays the video data on the display 120 (display 214). It is assumed that there is

For example, as shown in Figure 25(a), the participants at site a are the section manager of company A, the participants at site b are the general manager of company B and the section manager of company B, and the participants at site c are the person in charge of company A. In the case of a person in charge of company B, video data is displayed on the display 120 (display 214) of the communication terminal 10 at base a, for example, as shown in FIG. 25(b). That is, the video data shown in FIG. 25(b) uses the display control unit 69 to ensure that the participants at each base (here, bases b and c, which are the other bases) are in the same location and belong to the same company. The display is controlled so that the participants (the general manager, section manager, and person in charge of Company B) are arranged in the same row and in the order of their positions.

Then, the process moves to step S92.

<Step S92>
When a participant at the other site speaks, the communication unit 11 of the communication terminal 10 at the other site uses the audio data input by the audio input unit 15 , the video data captured by the imaging unit 13 , and the speech direction identification unit 20 to identify the speech direction. The information about the direction of speech is sent to the management system 50. The communication unit 51 of the management system 50 receives audio data, video data, and speech direction information of the other party's base from the communication terminal 10 of the other party's base. Then, the process moves to step S93.

<Step S93>
The association unit 65 of the management system 50 associates the speech direction of the participant's voice received via the communication unit 51 with the face image detected by the face detection unit 61, that is, the participant. Specifically, the association unit 65 refers to a speaker/speech direction correspondence table that associates coordinates (face center coordinates) as shown in FIG. 27 and utterance directions, which is stored in advance in the storage unit 58. Then, it is specified to which coordinates (face center coordinates) the speech direction received by the communication unit 51 corresponds. Here, for example, as shown in FIG. 26, if the participants at the partner base are A to D, the face detection unit 61 has calculated the center coordinates of the face images of the participants A to D, respectively. Then, the matching unit 65 identifies the center coordinates that match (or can be considered to match) the specified coordinates among the center coordinates of the face images of the participants at the partner base calculated by the face detection unit 61, and Identify the participant's avatar corresponding to the face image with coordinates. Then, the process moves to step S94.

<Step S94>
The display control unit 69 of the management system 50 reflects the behavior of the participant's avatar identified by the association unit 65 so that it is speaking in accordance with the audio data received by the communication unit 51. Generate video data. The specific reflection operation is as described above. Then, the communication unit 51 of the management system 50 transmits the audio data received from the communication terminal 10 at the other site and the video data generated by the display control unit 69 to the communication terminal 10 at the own site. When the display control unit 14 of the communication terminal 10 at its own site receives audio data and video data from the management system 50 via the communication unit 11, the display control unit 14 displays the video data on the display 120 (display 214), and outputs the audio data to the audio output unit 16. outputs the audio data as audio.

The avatar control process by the communication system 1 is executed through the flow of steps S91 to S94 described above. In this way, when a participant at the other site speaks, the speaking direction is specified, and which participant is speaking from that speaking direction is identified, so the speaking action can be directed to the avatar of the participant. can be reflected. As a result, participants at their own site who are viewing video data that includes the avatar of the other site can recognize which participant is speaking through the avatar, so in a video conference using the avatar. It also allows for smooth communication.

(Overall operation of communication system)
FIG. 28 is a sequence diagram illustrating an example of the flow of the overall operation of the communication system according to the embodiment. FIG. 29 is a diagram illustrating an example of a screen that prompts the acquisition of attribute information through voice recognition. With reference to FIGS. 28 and 29, the overall operation flow of the communication system 1 will be described in general. In addition, in FIG. 28, the communication terminal at the own site is assumed to be the communication terminal 10a (second communication terminal), and the communication terminal at the other site is assumed to be the communication terminal 10b (first communication terminal).

<Steps S101, S102>
In order to start a video conference with a participant at the other site, the participant at the own site performs a start operation (for example, selecting the communication terminal 10b as the destination terminal) via the input device 108 of the communication terminal 10a. operation). As a result, the operations shown in FIGS. 14 and 15 described above are executed, and a session is established between communication terminal 10a and communication terminal 10b.

<Step S103>
It is assumed that after the session is established, the communication unit 11 of the communication terminal 10a transmits the audio data input by the audio input unit 15 and the video data captured by the imaging unit 13 to the management system 50.

<Step S104>
When the management system 50 receives audio data and video data from the communication terminal 10a, it executes the avatar generation process shown in FIG. 17. As a result, video data showing the participant at the own base as an avatar is displayed on the display 120 (display 214) of the communication terminal 10b at the other base.

<Step S105>
In the avatar generation process of step S104, if the attribute information of at least one of the participants at the own base is not registered in the face/avatar/attribute correspondence table, the attribute information requesting unit 64 of the management system 50: An instruction to request the participant whose attribute information is not registered to speak the attribute information is transmitted to the communication terminal 10a via the communication unit 51.

<Step S106>
When the display control unit 14 of the communication terminal 10a receives the instruction via the communication unit 11, it causes the display 120 (display 214) to display a message prompting the user to utter the attribute information, for example. For example, FIG. 29 shows an example in which a message such as "Please introduce yourself. Company, affiliation, position, name" is displayed on the display 214 of the communication terminal 10a as a message prompting for attribute information.

<Step S107>
After confirming the message prompting them to speak the attribute information displayed on the display 120 (display 214) of the communication terminal 10a, the participants of the video conference self-introduce their attribute information such as their name, the name of the company they belong to, their affiliation, and position. utter as. Then, the microphone 114a (microphone 241) of the communication terminal 10a used by the participant who spoke collects the voice spoken by the microphone array and converts it into an audio signal, and the voice input unit 15 of the communication terminal 10a receives the voice. Input (acquire) the signal.

<Steps S108, S109>
The speech direction specifying unit 20 of the communication terminal 10a specifies the speech direction of the voice by performing voice processing on the voice signal input by the voice input unit 15. The communication unit 11 of the communication terminal 10 receives audio data (audio signal) input by the audio input unit 15, video data captured by the imaging unit 13, and information on the speech direction specified by the speech direction identification unit 20. The information is sent to the management system 50. Note that the information on the direction of speech is not limited to being transmitted to the management system 50 in step S109, but the communication terminal 10a identifies the direction of speech each time a speech is made in video conference communication, and the information on the direction of speech is transmitted to the other party. It is transmitted to the communication terminal 10b at the base.

<Step S110>
When the management system 50 receives audio data, video data, and speech direction information from the communication terminal 10a, the attribute information acquisition process shown in FIG. 21 is executed. As a result, the attribute information of the participant at the own base is registered in the face/avatar/attribute correspondence table.

<Steps S111, S112>
Then, when an avatar is generated by executing the avatar generation process and the attribute information of the participant at the own base is registered by the attribute information acquisition process, a correspondence relationship between the avatar and the attribute information is established, and the management system When the audio data, video data, and speech direction information are transmitted to the communication terminal 10b, the avatar control process shown in FIG. Sent.

The overall operation of the communication system 1 is performed through the flow of steps S101 to S112 described above.

As described above, in the communication system 1 according to the present embodiment, if attribute information of a participant is not registered at the time of starting a video conference, the management system 50 sends attribute information to the communication terminal 10 used by the participant. The communication terminal 10 transmits an instruction to request the user to speak the attribute information, and the communication terminal 10 performs an operation (for example, displaying a message or outputting a voice) to prompt the user to speak the attribute information. Then, the management system 50 converts the audio data input at the communication terminal 10 into text, extracts keywords that match the registered words, and registers them as attribute information of the participant who spoke. As a result, even if attribute information is not registered before the start of a video conference, it is possible to obtain attribute information, so display control of participant information (for example, avatars, real images) can be performed based on the attribute information. It can be carried out. Furthermore, by controlling the display of participant information based on the attribute information in this way, it is possible to understand the participants' positions, etc., and the meeting can proceed smoothly. Note that display control of participant information based on attribute information does not necessarily require displaying an avatar to represent the participant; for example, displaying a real image of the participant (an example of participant information) It may also be possible to perform control such as displaying attribute information nearby.

Further, in the communication system 1 according to the present embodiment, the management system 50 detects the faces of participants from the video data captured by the communication terminal 10, and provides registered face recognition information that matches the detected face detection information. An avatar representing the participant is generated from the avatar information corresponding to the participant. By using video data of avatars representing participants in this way, it is possible to transfer data at a lower frame rate than when using real images, which reduces the amount of data communication and reduces spec requirements for displays. Can be lowered.

Furthermore, in the communication system 1 according to the present embodiment, the direction of speech of a participant who speaks at the communication terminal 10 at each base is identified, and the face image of the participant detected by the management system 50 is associated with the direction of speech. to identify the avatar of the participant who made the utterance. Then, the management system 50 generates video data reflecting the avatar's uttering action in accordance with the voice data of the utterance, and transmits it to the communication terminal 10 at the other party's base. As a result, participants at the other site who view video data that includes an avatar from their own site can recognize which participant is speaking through the avatar, so even in video conferences using avatars. , it is possible to realize smooth communication.

Note that during a video conference, there is a possibility that participants may move their seats within the conference room where they are based, and in this case, the direction of the participants' speech will also change. In this case, for example, the face detection unit 61 detects the participant's face and calculates the center coordinates of the detected face image at regular intervals, or the movement of the participant is detected from the video data. This can be handled by executing the following steps. As a result, in the above-mentioned avatar control process, the association unit 65 refers to the speaker/speech direction correspondence table and specifies which coordinates (face center coordinates) the changed speech direction corresponds to. It is possible to specify the participant's avatar corresponding to the face image having the center coordinates.

Furthermore, the software configuration of the communication terminal 10 according to the embodiment described above is not limited to the configuration shown in FIG. 5 described above, and may be, for example, the configuration shown in FIG. 30. FIG. 30 is a diagram illustrating an example of a software configuration when using a web application of a communication terminal according to an embodiment. Although the above-described FIG. 5 describes the operation of the communication application A executed by the communication terminal 10, similar processing can also be realized by a Web application. A Web application operates by cooperation between a program based on JavaScript (registered trademark) and a program on a Web server that runs on a browser, and is used by a user on a browser. That is, as shown in FIG. 30, the communication terminal 10 downloads a program WA (HTML (HyperText Markup Language) + JavaScript (registered trademark) + CSS, etc.) from the management system 50 and executes it on the browser 1040. The browser 1040 operates under the control of the OS 1020. The communication terminal 10 can utilize the services provided by the management system 50 by exchanging data with the management system 50 using a protocol such as HTTP (Hypertext Transfer Protocol) or HTTPS (Hypertext Transfer Protocol Secure). In such a usage pattern, there is no need to download the communication application A to the communication terminal 10 in advance.

Moreover, each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" refers to a processor that is programmed to execute each function using software, such as a processor implemented using an electronic circuit, or an ASIC or DSP (Digital It includes devices such as a signal processor), an FPGA, an SoC (system on a chip), a GPU (graphics processing unit), and a conventional circuit module.

Further, in the above-described embodiment, when at least one of the functional units of the communication terminal 10 and the management system 50 is realized by executing a program, the program is provided by being incorporated in a ROM or the like in advance. Furthermore, the programs executed by the communication terminal 10 and the management system 50 according to the above-described embodiments can be stored as files in an installable or executable format on a CD-ROM (Compact Disc Read Only Memory) or a flexible disk (FD). , a CD-R (Compact Disk-Recordable), a DVD, or an SD card. Further, the program executed by the communication terminal 10 and the management system 50 according to the above-described embodiment is stored on a computer connected to a network such as the Internet, and is provided by being downloaded via the network. Good too. Furthermore, the programs executed by the communication terminal 10 and the management system 50 according to the above-described embodiments may be configured to be provided or distributed via a network such as the Internet. Further, the programs executed by the communication terminal 10 and the management system 50 according to the above-described embodiment have a module configuration including at least one of the above-mentioned functional units, and the actual hardware includes the CPU described above. By reading a program from the storage device and executing it, each of the above-mentioned functional units is loaded onto the main storage device and generated.

1 Communication system 2 Communication network 2a to 2d LAN
2ab, 2cd Dedicated line 2i Internet 10, 10a, 10aa, 10ab, 10b, 10ba, 10bb, 10ca, 10cb, 10da, 10db Communication terminal 11 Communication section 12 Operation input reception section 13 Imaging section 14 Display control section 15 Audio input section 16 Audio output unit 17 Storage/readout unit 18 Storage unit 19 Authentication request unit 20 Speech direction identification unit 30ab, 30cb PC
50 Management System 51 Communication Unit 52 Authentication Unit 53 State Management Unit 54 Terminal Extraction Unit 55 Terminal Status Acquisition Unit 56 Session Control Unit 57 Storage/Reading Unit 58 Storage Unit 61 Face Detection Unit 62 Comparison Unit 63 Generation Unit 64 Attribute Information Request Unit 65 Correlation unit 66 Text conversion unit 67 Extraction unit 68 Registration unit 69 Display control unit 70a to 70d, 70ab, 70cd Router 90 Program provision system 101 CPU
102 ROM
103 RAM
105 Auxiliary storage device 106 Media 107 Media drive 108 Input device 110 Bus line 111 Network I/F
112 Camera 112c Cable 113 Image sensor I/F
114 Smart speaker 114a Microphone 114b Speaker 114c Cable 115 Speaker 115c Cable 116 Audio input/output I/F
117 USB I/F
119 Display I/F
120aa, 120ba, 120ca, 120da Display 120c Cable 201 CPU
202 ROM
203 RAM
204 SSD
205 Network I/F
206 External device connection I/F
210 Bus line 211 Capture device 212 GPU
213 Display controller 214 Display 215 Sensor controller 216 Contact sensor 217 Electronic pen controller 222 Power switch 223 Selection switch 230 USB memory 240 Smart speaker 241 Microphone 242 Speaker 260 Camera 270 PC
290 Electronic pen 301 CPU
302 ROM
303 RAM
305 Auxiliary storage device 306 Recording media 307 Media drive 308 Display 309 Network I/F
310 Bus line 311 Keyboard 312 Mouse 313 DVD
314 DVD drive 315 USB I/F
1010 Work area 1020 OS
1040 Browser 1100-2 Terminal ID
1100-3 Terminal name 1100-4a Offline icon 1100-4b Call ready icon 1200-1 Start request reception screen 1200-2 "Yes" button 1200-3 "No" button 5001 Authentication management DB
5002 Terminal management DB
5003 Group management DB
5004 Session management DB
A Communication app WA program

Japanese Patent Application Publication No. 2010-093583

Claims (15)

A communication system that enables video conferencing by multiple communication terminals transmitting and receiving audio data via a network,
an imaging unit that obtains video data of one or more participants in the video conference;
a detection unit that detects the participant from the video data obtained by the imaging unit;
an audio input unit that inputs audio data uttered by the participant;
an extraction unit that extracts attribute information of the participant from the audio data input by the audio input unit;
a first control unit that controls display on a first communication terminal of the participant information detected by the detection unit based on the attribute information extracted by the extraction unit;
a request unit that transmits to the second communication terminal an instruction requesting a participant using a second communication terminal that communicates with the first communication terminal via the network to utter the attribute information;
a second control unit that performs a process of prompting a participant using the second communication terminal to utter the attribute information in the second communication terminal according to the instruction received from the request unit;
A communication system with The communication system according to claim 1 , wherein the second control unit displays a message indicating the instruction on a display device of the second communication terminal to a participant using the second communication terminal. The communication system according to claim 1 , wherein the second control unit causes an output device of the second communication terminal to output a voice indicating the instruction to a participant using the second communication terminal. further comprising a storage unit that associates and stores the participant's facial recognition information and attribute information;
The detection unit detects the face of the participant from the video data and stores the detected information in the storage unit as face recognition information,
When the attribute information is stored in the storage unit in association with the face recognition information of the participant detected by the detection unit, the first control unit controls information about the participant based on the attribute information. is displayed on the first communication terminal,
When the attribute information is not stored in the storage unit in association with the face recognition information of the participant detected by the detection unit, the request unit transmits an instruction to request the utterance to the second communication terminal. The communication system according to any one of claims 1 to 3, wherein the communication system transmits a message to a user. The audio input unit inputs audio data uttered by a participant using the second communication terminal after the request unit sends an instruction to request the utterance to the second communication terminal,
The extraction unit extracts attribute information of the participant from the audio data of the participant using the second communication terminal input by the audio input unit,
The communication system further includes a registration unit that registers attribute information of a participant using the second communication terminal extracted by the extraction unit in the storage unit in association with face recognition information corresponding to the participant. The communication system according to claim 4, comprising: further comprising a generation unit that generates an avatar representing the participant detected by the detection unit,
The first control unit controls display on the first communication terminal of an avatar corresponding to the participant generated by the generation unit as information about the participant detected by the detection unit. The communication system according to any one of items 1 to 5 . The communication system according to claim 6 , wherein the first control unit changes the arrangement of the avatar corresponding to the attribute information based on the attribute information. Among the avatars displayed on the first communication terminal, the first control unit increases the sound pressure level of the voice data of the participant corresponding to the avatar that is placed closer to the avatar. The communication system according to claim 7 , wherein the communication system outputs the output from the output device of the first communication terminal. The communication system according to claim 6 , wherein the first control unit displays the attribute information near the avatar corresponding to the attribute information. a first identifying unit that identifies, based on the audio data input by the audio input unit, the speaking direction of the participant who uttered the audio data;
a second identifying unit that identifies a participant corresponding to the speaking direction among the participants detected by the detecting unit based on the position of the participant detected by the detecting unit on the video data; ,
It further has
The first control unit performs display control on the avatar of the participant identified by the second identification unit so as to indicate that the avatar of the participant is speaking the audio of the audio data in accordance with the audio data. The communication system according to any one of items 6 to 9 . According to any one of claims 1 to 10 , the first control unit divides the information of the participants detected by the detection unit by base of the participant and displays the information on the first communication terminal. Communication system as described. The communication according to any one of claims 1 to 10 , wherein the first control unit displays information on the participants detected by the detection unit on the first communication terminal as if they were at the same base. system. An information processing device that relays transmission and reception of audio data between multiple communication terminals that conduct a video conference,
a detection unit that detects the participant from video data obtained by an imaging unit that has obtained video data of one or more participants of the video conference;
an extraction unit that extracts attribute information of the participant from audio data input by an audio input unit that inputs audio data uttered by the participant;
a first control unit that controls display on a first communication terminal of the participant information detected by the detection unit based on the attribute information extracted by the extraction unit;
a requesting unit that transmits to the second communication terminal an instruction requesting a participant using a second communication terminal communicating with the first communication terminal to utter the attribute information;
a second control unit that performs a process of prompting a participant using the second communication terminal to utter the attribute information in the second communication terminal according to the instruction received from the request unit;
An information processing device having: A communication method that realizes a video conference by having multiple communication terminals transmit and receive audio data via a network,
an imaging step of obtaining video data of one or more participants in the video conference;
a detection step of detecting the participant from the video data;
a voice input step of inputting voice data uttered by the participant;
an extraction step of extracting attribute information of the participant from the input audio data;
a first control step of controlling display on a first communication terminal of the detected participant information based on the extracted attribute information;
a requesting step of transmitting to the second communication terminal an instruction requesting a participant using a second communication terminal that communicates with the first communication terminal via the network to utter the attribute information;
a second control step of performing, in the second communication terminal, a process of prompting a participant using the second communication terminal to utter the attribute information according to the instruction;
A communication method having A computer that relays the transmission and reception of audio data between multiple communication terminals conducting a video conference.
a detection step of detecting the participant from video data obtained by an imaging unit that has obtained video data of one or more participants of the video conference;
an extraction step of extracting attribute information of the participant from the audio data input by an audio input unit that inputs the audio data uttered by the participant;
a first control step of controlling display on a first communication terminal of the detected participant information based on the extracted attribute information;
a requesting step of transmitting, to the second communication terminal, an instruction requesting a participant using a second communication terminal communicating with the first communication terminal to utter the attribute information;
a second control step of performing, in the second communication terminal, a process of prompting a participant using the second communication terminal to utter the attribute information according to the instruction;
A program to run.

Images