JP5403137B2 - Network communication system - Google Patents

Description

  The present invention relates to a network communication system in which a plurality of communication devices installed at locations distant from each other are connected to a network to communicate audio and video with each other.

  Conventionally, various systems for performing a conference between remote locations such as a head office, a branch office, and a branch office have been disclosed. In such a conference system, as shown in Patent Document 1 and Patent Document 2, in order to give a sense of realism by the conference, speaker detection is performed, and the video of the currently speaking speaker is displayed, for example, so as to be emphasized. Processing such as making it happen.

JP-A-4-339484 Japanese Patent No. 2648192

  In the conference system as described above, it is necessary to manage video data, audio data, and speaker information of each conference device. For this reason, in Patent Document 1, a central control device is provided in a network, and communication control of all data and information is performed by the central control device. Further, in Patent Document 2, among a plurality of conference apparatuses connected to a network, only one conference apparatus that gives a transmission right to the transmission apparatus and obtains transmission permission transmits data and information to other apparatuses. .

  However, when a centralized control device is used, since data concentrates on the centralized control device, the centralized control device must be made large-scale and complicated. Further, when the transmission right is used, data can be transmitted only by a single device having the transmission right, so that flexibility is lacking.

  Therefore, an object of the present invention is to provide a network communication system that can acquire and use speaker information of a plurality of conference apparatuses such as sound emitting and collecting apparatuses almost in real time without constituting a large-scale system. There is to do.

  The present invention includes a plurality of sound emission and collection devices that are connected to an audio data communication network that communicates audio data, each of which emits and collects sound, and a communication control server that performs connection control between the plurality of sound emission and collection devices. The plurality of sound emission and collection devices each include a speaker detection means, and the communication control server controls speaker detection information acquisition for each of the plurality of sound emission and collection devices. When the plurality of sound emission and collection devices accept the speaker detection information acquisition control, the speaker detection information acquired by the speaker detection means is transmitted to the communication control server, and the voice data is transmitted from the communication control server. It is characterized in that direct communication is performed between a plurality of sound emitting and collecting devices without going through a control server.

  In this configuration, the communication control server does not perform audio data communication control between the sound emitting and collecting devices, but only performs connection control such as connection start between the sound emitting and collecting devices. Apart from such connection control, the communication control server performs acquisition control of speaker detection information for each sound emission and collection device. When each sound emission and collection device accepts the speaker detection information acquisition control, it acquires the speaker position information around its own device including the presence or absence of the speaker and transmits it to the communication control server. When the communication control server accepts the speaker position information, the speaker position information of each sound emitting and collecting apparatus connected to the network is collectively managed. At this time, the communication control server does not need to perform communication control of voice data that requires a high communication speed even though the amount of data is large. Therefore, it is possible to acquire and manage speaker position information with a simple configuration and simple processing. It can be carried out.

  Further, the speaker detection means of the sound emission and collection device of the present invention generates the speaker detection information when the level of the sound collection data input to the sound emission and collection device is a predetermined threshold or more. And

In addition, the communication control server of the network communication system according to the present invention generates a talk state data that associates the acquired talker detection information with the sound emission and collection device that has transmitted the talker detection information. A person information management unit is provided.
In this configuration, the communication control server can recognize the presence or absence of a speaker in each sound emission and collection device.

  The communication control server of the network communication system according to the present invention stores in advance a schedule of communication by a plurality of sound emission and collection devices, and generates the speaker status data in time series based on the schedule or time information. It is characterized by that.

  In this configuration, the communication control server acquires the speaker detection information only while communication is performed according to the schedule. Thereby, speaker detection information can be obtained only when necessary, and the resources of the network can be used effectively together with the simplification of the system described above.

  The network communication system according to the present invention further includes a plurality of video processing means corresponding to the plurality of sound emission and collection devices, respectively, for acquiring and displaying a video. The communication control server of the network communication system according to the present invention generates speaker situation data based on the acquired speaker position information and provides it to a plurality of video processing means. The plurality of video processing means of the present invention is characterized by displaying the acquired speaker status data.

  In this configuration, speaker status data based on the speaker position information group obtained over time is generated and displayed by the video processing means. Thereby, it is possible to visually acquire the speaker situation including the switching of the speaker who is speaking.

  The plurality of video processing means of the network communication system according to the present invention is characterized in that switching control of video to be displayed is performed based on speaker status data.

  In this configuration, each video processing means can perform display switching control such as highlighting the video of the speaker who is currently speaking based on the speaker status data.

  According to the present invention, each communication is performed with a simple system configuration and a simple process without using a large system that manages and uses all of the control data, audio data, and the like collectively in the communication control server. The speaker information in the sound emission and collection device can be managed and used collectively.

It is a block diagram which shows the main structures of the audio | voice communication system of this embodiment. It is a block diagram which shows the main structures of 11 A of sound emission / collection apparatuses. The figure explaining the audio | voice data communication network 201 and the control data communication network 202 in the aspect which mesh-connected the sound emission / collection apparatus 11A-11D, and the sound emission / collection apparatus 11E- consisting of the same structure as the sound emission / collection apparatus 11A-11D It is a figure explaining the audio | voice data communication network 201 and the control data communication network 202 in the aspect which cascade-connected 11J. 2 is a block diagram showing a main configuration of a control server 10. FIG. 2 is a diagram showing the configuration of a video data communication network 203 and a speaker status data communication network 204. FIG. 6 is a flowchart showing a flow of establishing a voice data communication network 201. 5 is a flowchart showing a processing flow of speaker detection after the establishment of the voice data communication network 201.

A network communication system according to an embodiment of the present invention will be described with reference to the drawings. In the following description, as a main configuration of a network communication system, an audio communication system that communicates audio between a plurality of sound emission and collection devices, particularly an audio communication system with a video that includes an image processing device together with a sound emission and collection device is taken as an example. explain. In the following description, an audio communication system including a video processing device will be described. However, the processing in the video processing device is an example of processing based on speaker detection information, and the video processing device may not be provided.
FIG. 1 is a block diagram showing the main configuration of the voice communication system of the present embodiment.
FIG. 2 is a block diagram showing the main configuration of the sound emission and collection device 11A.

3A is a diagram for explaining the voice data communication network 201 and the control data communication network 202 in a mode in which the sound emitting and collecting devices 11A to 11D are mesh-connected, and FIG. 3B is a diagram illustrating the sound emitting and collecting devices 11A to 11A. It is a figure explaining the audio | voice data communication network 201 and the control data communication network 202 in the aspect which cascade-connected the sound emission and collection apparatuses 11E-11J which have the same structure as 11D.
FIG. 4 is a block diagram showing the main configuration of the control server 10.
FIG. 5 is a diagram showing the configuration of the video data communication network 203 and the speaker status data communication network 204.

  The audio communication system has a configuration in which sound emitting and collecting devices 11A to 11D and video processing devices 12A to 12D in conference rooms 101A to 101D at different positions are connected via a network 200 via routers 13A to 13D, respectively. Further, the control server 10 is connected to the network 200 via the router 13E. The network 200 includes one or a plurality of physical layers. On the physical layer, an audio data communication network 201 that transmits audio data, a control data communication network 202 that transmits control data, and a video that transmits video data. A data communication network 203 is formed.

Each of the conference rooms 101A to 101D is provided with sound emission and collection devices 11A to 11D, video processing devices 12A to 12D, and routers 13A to 13D. In each of these conference rooms, the connection configuration of each device is the same, and a description will be given using the conference room 101A as a representative.
In the conference room 101A, a sound emission and collection device 11A, a video processing device 12A, and a router 13A are installed, and the sound emission and collection device 11A and the video processing device 12A are connected to the network 200 via the router 13A.

  The video processing device 12A includes an imaging unit that captures the periphery of the sound emission and collection device 11A, a video data generation unit that generates video data from the captured video, and a video acquired via the video data communication network 203 of the network 200. A video display unit for displaying data and an input / output unit for inputting / outputting data to / from the router 13A are provided. As shown in FIG. 5, the video processing device 12 </ b> A is connected to the control server 10 by a speaker status data communication network 204 that transmits and receives speaker status data.

  The router 13A forms a secure network 200 such as IPSec-VPN with other routers 13B to 13E. More specifically, as shown in FIG. 3A, the router 13A forms a voice data communication network 201 that connects between the sound emission and collection device 11A and the other sound emission and collection devices 11B to 11D. A control data communication network 202 that connects between the sound emission and collection device 11A and the control server 10 is formed. The routers 13B to 13D also form a control data communication network 202 that connects the corresponding sound emitting and collecting apparatuses 11B to 11D and the control server 10 in the same manner. That is, the control data communication network 202 is formed between the sound emission and collection devices 11 </ b> A to 11 </ b> D and the control server 10. Further, as shown in FIG. 5, the router 13A forms a video data communication network 203 that connects the video processing device 12A and the other video processing devices 12B to 12D.

  As shown in FIG. 2, the sound emission and collection device 11A includes a control unit 111, a network I / F 112, a sound emission mixer 113, a speaker detection unit 114, a transmission sound mixer 115, a speaker SP, and a microphone MIC.

  The control unit 111 performs overall control of the sound emission and collection device 11A and controls the network I / F 112 to transmit the sound collection data collected by the own device to the voice data communication network 201, and the speaker detection unit 114. Is transmitted to the control data communication network 202.

When the control unit 111 receives a speaker detection request from the control server 10, the control unit 111 measures the time with a timer (not shown) and periodically performs speaker detection control on the speaker detection unit 114. In addition, after receiving the speaker detection request, the control unit 111 stops the speaker detection control for the speaker detection unit 114 unless a new speaker detection request is received for a predetermined time or longer.
Further, the control unit 111 controls processing of the transmission sound mixer 115 according to the selected connection mode. Specifically, the process control is set depending on whether the mesh connection mode is as shown in FIG. 3A or the cascade connection mode is as shown in FIG.

  The network I / F 112 is connected to the external network 200 via the router 13 </ b> A, and the sound collection data of the sound emission and collection devices 11 </ b> B to 11 </ b> D input via the voice data communication network 201 is transmitted to the voice data communication network 201. The data format is converted into a predetermined audio data format, and is given to the sound output mixer 113 and the transmission sound mixer 115. The network I / F 112 converts the collected sound data collected by the microphone MIC and mixed as necessary by the transmission mixer 115 into the communication data format of the voice data communication network 201, and the voice data communication network 201 is To the sound emission and collection devices 11B to 11D. Further, the network I / F 112 transmits the speaker detection data generated by the speaker detection unit 114 to the control server 10 via the control data communication network 202.

The sound emission mixer 113 mixes the sound data collected by the other sound emission and collection devices 11B to 11D and outputs the mixed sound data to the speaker SP.
The speaker SP emits sound according to the input audio data.
The microphone MIC collects the sound around the own device and provides the collected sound data to the speaker detection unit 114 and the transmission sound mixer 115.
The speaker detection unit 114 has a speaker detection threshold set and stored in advance. If the level of the input sound pickup data is equal to or higher than the speaker detection threshold, a speaker who is present around the device is speaking. Is determined, and speaker detection data indicating “spoken” is generated. On the other hand, if the level of the input sound pickup data is less than the speaker detection threshold, the speaker detection unit 114 determines that there is no speaker and generates speaker detection data indicating no speech.

  In accordance with control from the control unit 111, the transmission sound mixer 115 outputs sound collection data as it is in the mesh connection mode. In the cascade connection mode as shown in FIG. 3B, the transmission sound mixer 115 mixes and outputs the sound data collected by the other sound emission and collection devices to the sound collection data of its own device. At this time, the transmission sound mixer 115 includes data designating a transmission destination, and generates audio data for transmission for each transmission destination. Specifically, the transmission sound mixer 115 mixes the sound data collected by the sound emission and collection device excluding the transmission destination and the sound collection data of the own device, and outputs the result in association with the transmission destination information. . This process is performed for each transmission destination.

  With such a configuration, the sound emitting and collecting apparatus 11A (11B to 11D) directly transmits and receives audio data between the sound emitting and collecting apparatuses using the audio data communication network 201. The sound emission and collection device 11 </ b> A (11 </ b> B to 11 </ b> D) acquires speaker detection data in response to a speaker detection request from the control server 10 and transmits it to the control server 10.

As illustrated in FIG. 4, the control server 10 includes a communication control unit 150, a schedule control unit 151, a speaker information management unit 152, a network I / F 153, and a display unit 154.
The communication control unit 150 controls communication using the control data communication network 202 with the sound emission and collection devices 11A to 11D performed via the network I / F 153. Specifically, the communication control unit 150 does not control transmission / reception of voice data, and uses the control data communication network 202 to establish or cancel the voice data communication network 201 between the sound emission and collection devices 11A to 11D. Take control. Moreover, the communication control part 150 performs control which performs the request | requirement of speaker detection data with respect to each sound emission and collection apparatus 11A-11D, and reception control of speaker detection data. Further, the communication control unit 150 controls communication of speaker status data performed via the speaker status data communication network 204.

  The schedule control unit 151 has a timer function, stores a schedule of a conference that is input in advance, and notifies the communication control unit 150 to perform establishment control of the voice data communication network 201 when the conference start timing is detected. . When the conference end timing is detected, the communication control unit 150 is notified to cancel the voice data communication network 201. Further, the schedule control unit 151 gives the stored schedule, whether the conference is currently being executed, and time information to the speaker information management unit 152.

  The speaker information management unit 152 acquires the speaker detection data of each of the sound emission and collection devices 11A to 11D via the communication control unit 150, and links with the schedule and time information from the schedule control unit 151, so that time series Speaker status data including speaker information is generated. At this time, the speaker information and the sound emission and collection device are associated with each other. The generated speaker status data is given to the display unit 154 and stored, and if requested by the external video processing device 12A or the like via the communication control unit 150, the communication control unit 150, the network I / F 153, the talk It is transmitted to the request source via the person situation data communication network 204.

  The display unit 154 is composed of a liquid crystal panel or the like, and displays a graphical display that allows the change of the speaker to be visually acquired based on the schedule, time information, and speaker status data given from the speaker information management unit 152. Do. Thereby, the person in the position where the control server 10 is installed can easily and visually confirm the situation of the speaker.

  Further, as described above, since the speaker status data can be output to the outside of the control server 10, the speaker status data can be easily used in each of the conference rooms 101A to 101D. For example, when the speaker status data is used by the video processing device 12A, when the “speech” information is acquired, the speaker from the video processing device corresponding to the sound emitting and collecting device associated with the “speech” information. The image can be displayed more highlighted than images from other video processing devices. Thereby, it is possible to provide a conference that is more easily understood by the speaker.

Next, the processing flow for establishing the voice data communication network 201 and detecting the speaker in the network communication system of the present embodiment will be described more specifically with reference to FIGS.
FIG. 6 is a flowchart showing the establishment processing flow of the voice data communication network 201.
FIG. 7 is a flowchart showing a processing flow of speaker detection after the establishment of the voice data communication network 201.

  In the following description, a case where connection is made from the sound emitting and collecting device 11A to the sound emitting and collecting device 11B is shown, but the same processing may be performed for other sound emitting and collecting devices. Further, transmission / reception of control data using the control data communication network 202 as described below is specifically realized by so-called SIP.

  First, the control server 10 reads a previously set and stored schedule (S101), and obtains the current time (S102). When the acquisition time is the conference start time of the read schedule (S103: Y), the control server 10 sends another sound emission / collection device 11A of the sound emission / collection devices participating in the conference to the other conference participants. A transmission request to the sound emission and collection device 11B is transmitted via the control data communication network 202 (S104).

  The sound emission and collection device 11A is in a standby state by a power input of a user who uses this or an automatic power input by a timer (S201), and receives a call request when it is detected from the control server 10 (S202). . The sound emission and collection device 11A transmits a call to the sound emission and collection device 11B to the control server 10 via the control data communication network 202 in accordance with the outgoing call request (S203), and the control server 10 relays the call. Then, it transmits to the sound emitting and collecting apparatus 11B via the control data communication network 202 (S105).

  Similarly to the sound emission and collection device 11A, the sound emission and collection device 11B is in a standby state in advance (S301), and receives a call according to the call from the sound emission and collection device 11A via the control server 10 (S302). At this time, an incoming request is transmitted from the control server 10 via the control data communication network 202 (S106), and the sound emitting and collecting apparatus 11B receives the incoming request after the incoming call (S303).

  When the sound emitting and collecting device 11B confirms that it is not busy, it transmits the communication start possibility information in the voice data communication network 201 to the control server 10 via the control data communication network 202 (S304). Relays this communication start possible information and transmits it to the sound emission and collection device 11A (S107).

  When receiving the communication start enable information (S204), the sound emission and collection device 11A performs communication start control in the voice data communication network 201 via the control server 10 using the control data communication network 202. 11B (S205 → S108 → S305). The sound emitting and collecting apparatus 11B uses the control data communication network 202 to control communication start from the sound emitting and collecting apparatus 11A, and responds to the sound emitting and collecting apparatus 11A via the control server 10 (S305 → S108 → S205). ).

  When the confirmation of such communication start control is completed, a transmission path by the voice data communication network 201 is established between the sound emitting and collecting apparatus 11A and the sound emitting and collecting apparatus 11B (S206, S306).

  When the voice data communication network 201 is established, the sound emitting and collecting apparatus 11A and the sound emitting and collecting apparatus 11B perform transmission and reception of voice data without going through the control server 10 (S207, S307).

  As described above, in the configuration of the present embodiment, until the establishment of the voice data communication network 201 that transmits and receives voice data between the sound emission and collection devices, the control server 10 performs the lead and establishes the voice data communication network 201. If it does so, audio | voice data can be directly transmitted / received between each sound emission and collection apparatus. Thereby, the control server 10 does not need to use a large-scale and complicated one having a mixing function.

  Next, when the voice data communication network 201 is established, the control server 10 starts timing (S111). When detecting the preset speaker detection request transmission timing (S112: Y), the control server 10 sends the speaker detection request to each sound emitting and collecting apparatus 11A, 11B to which connection has been established via the control data communication network 202. (S113). On the other hand, if it is not the speaker detection request transmission timing (S112: N), the control server 10 continues timing and timing detection (S111).

Hereinafter, since the sound emission and collection device 11A and the sound emission and collection device 11B perform the same processing, only the sound emission and collection device 11A will be described.
When receiving the speaker detection request (S211), the sound emission and collection device 11A transmits confirmation data indicating that the detection request has been received to the control server 10 via the control data communication network 202 (S212). The control server 10 obtains this confirmation data, thereby determining that the sound emission and collection device 11A starts speaker detection (S114), and enters a standby state for reception of speaker detection data.

  The sound emission and collection device 11A starts measuring time (S213), and at the detection timing (S214: Y), executes speaker detection and generation of speaker detection data as described above, and sends the speaker to the control server 10. The detection data is transmitted (S215). At this time, the sound emission and collection device 11 </ b> A transmits the speaker detection data to the control server 10 using the control data communication network 202.

  The sound emission and collection device 11A continuously performs speaker detection and generation and transmission of speaker detection data until the detection period ends (S216: N → S213). Here, the end of the detection period is determined as follows, for example. (1) The sound emission and collection device 11A receives a speaker detection request from the control server 10 and performs speaker detection, and ends if it does not receive a new speaker detection request at a preset time. To decide. (2) Although not shown, the sound emission and collection device 11 </ b> A determines the end when the speaker detection request end data indicating the end of the speaker detection request is received from the control server 10. When the end of the detection period is detected, the sound emission and collection device 11A stops the acquisition and transmission of the speaker detection data to the control server 10 (S216: Y → S217).

  The control server 10 receives and stores speaker detection data from the sound emission and collection device 11A (S115), and generates and updates speaker status data (S116). That is, when acquiring the speaker detection data for the first time, the control server 10 newly generates speaker status data, and when acquiring the speaker detection data sequentially thereafter, , Update to add new speaker detection data. Such generation / update of the speaker status data continues to be executed until the conference end timing set in the schedule or the end timing input by the user (S117 → S111). When the end time is detected, the control server 10 stores the speaker status data and performs end processing (S117: N → S118).

  By performing such processing, the control server 10 can acquire speaker detection data, that is, speaker information, of each of the sound emission and collection devices 11A to 11D without performing communication control of voice data. Accordingly, various application processes based on the speaker detection data as described above can be realized without using a large-scale and expensive server. Further, at this time, since the voice data communication network 201 is not used for the transmission / reception of the speaker detection data and the speaker status data, the transmission / reception of the voice data transmitted through the voice data communication network 201 may be affected. It is not affected by transmission and reception. That is, the speaker detection data and the speaker status data can be transmitted and received independently. And since the transmission / reception of such speaker detection data and speaker status data has a relatively small amount of data, even if a transmission path with a small capacity is used, there is no time lag between the voice data and the voice data. The speaker status data can be used in each of the conference rooms 101A to 101D.

101A to 101D-conference room, 10-control server, 11A to 11D-sound emitting and collecting device, 12A to 12D-video processing device, 13A to 13E-router, 200-network, 201-voice data communication network, 202-control data Communication network, 203-video data communication network, 204-speaker status data communication network,
111-Control unit of sound emission and collection device, 112-Network interface of sound emission and collection device, 113-Mixer for sound emission, 114-Speaker detection unit, 115-Mixer for transmission sound,
150-Communication control unit of control server 10, 151-Schedule control unit, 152-Speaker information management unit, 153-Network I / F of control server 10, 154-Display unit

Claims (6)

A plurality of sound emitting and collecting devices connected to a voice data communication network for communicating voice data and each emitting and collecting sound;
A communication control server for controlling connection between a plurality of sound emission and collection devices;
A network communication system comprising:
Each of the plurality of sound emission and collection devices includes a speaker detection means,
The communication control server performs speaker detection information acquisition control for each of the plurality of sound emission and collection devices,
When receiving the speaker detection information acquisition control, the plurality of sound emission and collection devices transmit the speaker detection information acquired by the speaker detection means to the communication control server,
The network communication system, wherein the voice data is directly communicated between a plurality of sound emission and collection devices without going through the communication control server. 2. The network according to claim 1, wherein the speaker detection unit generates the speaker detection information when the level of sound collection data input to the sound emission and collection device is equal to or higher than a predetermined threshold. Communications system. The communication control server includes a speaker information management unit that generates speaker status data in which the acquired speaker detection information is associated with the sound emission and collection device that has transmitted the speaker detection information. The network communication system according to claim 1 or 2, characterized in that The communication control server stores in advance a schedule of communication by the plurality of sound emission and collection devices, and generates the speaker situation data in time series based on the schedule or time information. 4. The network communication system according to 3. A plurality of video processing means corresponding to the plurality of sound emission and collection devices, respectively, for acquiring and displaying a video,
The network communication system according to claim 3 or 4, wherein the plurality of video processing means display the acquired speaker status data. The network communication system according to claim 5, wherein the plurality of video processing means perform switching control of video to be displayed based on the speaker status data.

Images