JP2010021924A - Communication apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology capable of notifying a speaking person about a listening situation of a listening person without disturbing a speech when performing voice communication between a plurality of communication terminals. <P>SOLUTION: When receiving data from another terminal 10, a terminal 10 extracts notification data from the received data. A plurality of notification data items are supplied to a conduction determination part 131 of a notification sound synthesis section 13. When all the supplied notification data are value indicative of "OK", the conduction determination part 131 changes over a switch section so as to output an effective sound component but in the other case, the switch section is changed over so that the effective sound component may not be output. A synthetic sound generation part 133 of the notification sound synthesis section 13 outputs the effective sound component based on voice signals output from a microphone MC. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a communication device.

In recent years, a remote conference system that performs a conference using a plurality of communication terminals connected via a communication network has become widespread. In such a teleconference system, since the speaker and the listener are not directly facing each other, it is difficult for the speaker to feel the listener's reaction, and he / she feels uneasy whether his / her voice reaches the other party. There is a case. Therefore, Patent Document 1 proposes a technique for displaying in real time on the transmission side what kind of state the transmitted voice / image data has reached on the reception side in order to eliminate the anxiety of the speaker. . Patent Document 2 discloses an input voice transmitted from another base (own base) connected via a network in an audio conference system that transmits and receives audio data between a plurality of bases connected via a network. A technique has been proposed in which an echo audio signal is sent back to another site, so that it can be confirmed at the other site that is sent back, whether it is being played back with the clear voice quality expected at the other site.
JP 2005-269498 A JP 2007-274176 A

However, the technique described in Patent Document 1 described above requires a display monitor and cannot be applied to an audio conference apparatus without a display monitor. Further, in the technique described in Patent Document 2, it is necessary to confirm by listening to the echo sound of its own voice. However, since confirmation and conversation are separate, confirmation cannot be performed simultaneously while talking.
The present invention has been made in view of the above-described background, and when voice communication is performed between a plurality of communication terminals, the listener's listening status can be notified to the speaker without disturbing the utterance. The purpose is to provide technology.

  In order to solve the above problems, the present invention provides a receiving means for receiving data transmitted from another terminal connected via a communication network, and audio data representing the sound collected by the sound collecting means. The sound effect component generation for generating the sound effect component signal representing the sound effect component of the sound using the transmission means for transmitting to the other terminal and the sound data representing the sound collected by the sound collecting means Analyzing the data received by the receiving means, determining whether the data satisfies a predetermined condition according to the analysis result, and the determination result by the determining means is affirmative In some cases, there is provided a communication apparatus comprising sound emission control means for outputting a sound effect component signal generated by the sound effect component generation means to the sound emission means.

  In a preferred aspect of the present invention, the receiving means receives data from a plurality of terminals, and the determining means analyzes the data received by the receiving means for each terminal, and the data is determined according to the analysis result. Whether or not a predetermined condition is satisfied is determined for each terminal, and the sound effect component generation unit is responsive to at least one of the number and the ratio of the terminals for which the determination result by the determination unit is positive The sound effect component signals to be generated may be different.

  In a further preferred aspect of the present invention, the receiving means receives data from a plurality of terminals, and the determining means analyzes the data received by the receiving means for each terminal, and the data according to the analysis result. For each of the terminals, the sound emission control means determines in advance at least one of the number and the ratio of terminals whose determination result by the determination means is affirmative. If the specified condition is satisfied, the sound effect component signal may be output to the sound emitting means.

  Further, in a further preferred aspect of the present invention, there is provided position data storage means for storing position data indicating positions of the plurality of terminals for each of the terminals, and the sound emission control means is emitted from the sound emission means. The sound may be controlled so that sound is localized in a direction according to the position data stored in the position data storage means and the determination result for each terminal by the determination means.

  Further, in a further preferred aspect of the present invention, the sound emission control means further comprises sound emission mode data storage means for storing sound emission mode data indicating a sound emission mode by the sound emission means for each terminal, The sound may be emitted in the sound emission mode indicated by the sound emission mode data corresponding to the terminal having a positive determination result by the determination unit.

  Further, in a further preferred aspect of the present invention, the determination means indicates whether or not the audio data transmitted by the transmission means is normally reproduced on the other terminal from the data received by the reception means. Notification data may be extracted and determined according to the extracted notification data.

  Further, in a further preferred aspect of the present invention, the determination means detects the sound pressure of the audio data received by the reception means, and determines whether or not the detected sound pressure satisfies a predetermined condition. Also good.

  Further, in a further preferred aspect of the present invention, the receiving means receives data including voice data from the other terminal, and the determining means receives voice data included in the data received by the receiving means. Analysis may be performed and determination may be made according to the analysis result.

  In addition, it includes collation data storage means for storing collation data representing voice or voice characteristics, and the determination means collates voice data received by the reception means with collation data stored in the collation data storage means. Then, it may be determined whether or not the voice data satisfying a predetermined matching result is included.

  ADVANTAGE OF THE INVENTION According to this invention, when performing voice communication between several communication terminals, a listener's listening condition can be alert | reported with respect to a speaker, without disturbing speech.

<Configuration>
FIG. 1 is a block diagram showing a configuration of a remote conference system 1 according to an embodiment of the present invention. The remote conference system 1 is configured by connecting a plurality of terminals 10a, 10b, 10c... Installed in various places to a communication network 20 such as the Internet. In the following description, for convenience of description, when it is not necessary to distinguish the terminals 10a, 10b, 10c..., These will be referred to as “terminal 10”. A remote conference is realized by a voice conference performed by a participant of the remote conference using the terminal 10.

  FIG. 2 is a block diagram illustrating an example of the configuration of the terminal 10. In the figure, the microphone MC is a sound collecting means for collecting a voice uttered by a person participating in the conference (hereinafter referred to as “participant”) and outputting a voice signal (analog signal) representing the collected voice. is there. The audio signal output from the microphone MC is output to the CODEC 16. The CODEC 16 converts an audio signal output from the microphone MC into digital data. The operation unit 18 outputs a signal corresponding to an operation by a conference participant. The operation unit 18 is provided with a dedicated button B <b> 1 for inputting that the conference participants are listening to other participants properly. In the following description, for convenience of description, a signal output from the operation unit 18 when the button B1 is pressed will be referred to as “operation signal S1”. The operation signal S1 output from the operation unit 18 is output to the packet combining unit 17. The packet combining unit 17 detects the operation signal S1 output from the operation unit 18, and generates notification data according to the operation signal S1 detected at the timing when the operation signal S1 is detected. The packet synthesizing unit 17 packetizes the audio data output from the CODEC 16 and the notification data corresponding to the operation signal S1 output from the operation unit 18, and transmits the packetized data to other terminals 10. This notification data is data generated when the participant actively presses the dedicated button B1, and thus can be used as data indicating whether the conference participant is listening to the speaker. .

  The terminal 10 is provided with a data receiving unit 22 for each of the other terminals 10 that perform voice communication. The terminal 10 individually receives data received from each of the other terminals 10. The packet separator 11 receives data transmitted from another terminal 10 connected via the communication network 20. The data transmitted from the other terminal 10 includes the voice data representing the sound collected by the microphone MC of the other terminal 10 and the listening status of the other participants (the participant listens to the speaker properly). Notification data indicating whether or not) is included. The packet separation unit 11 separates the voice data and the notification data from the received data, outputs the voice data to the CODEC 12, and outputs the notification data to the notification sound synthesis unit 13. The CODEC 12 decodes the received audio data and outputs it to the audio mixer 14. The audio mixer 14 mixes audio signals output from the CODECs 12, 12,... Corresponding to the plurality of other terminals 10, and outputs them to the adder 15.

The notification sound synthesizing unit 13 determines whether or not the listener has properly heard the speaker's story according to the notification data of the other terminal 10 supplied from the packet separation unit 11, and the determination result is affirmative. In some cases, a sound effect component signal representing a sound effect component of sound is generated using the sound signal output from the microphone MC.
FIG. 3 is a block diagram illustrating an example of the configuration of the notification sound synthesis unit 13. In the figure, a continuity determination unit 131 determines notification data transmitted from another terminal 10 for each terminal 10 and determines whether the notification data for each terminal 10 satisfies a predetermined condition. In this embodiment, the continuity determination unit 131 satisfies a predetermined condition (hereinafter referred to as “conduction OK”) when the values of the notification data received from other terminals 10 are all values indicating “OK”. Is determined. On the other hand, when at least one of the notification data received from the other terminals 10 is “NG”, the continuity determination unit 131 does not satisfy a predetermined condition (hereinafter referred to as “conduction NG”). Is determined. The switch unit 132 is switched under the control of the continuity determination unit 131. When the determination result is “conduction OK”, the continuity determination unit 131 switches the switch unit 132 so that the sound effect component is output. On the other hand, when the determination result is “conduction NG”, the continuity determination unit 131 switches the switch unit 132 so that the sound effect component is not output.

The synthesized sound generating unit 133 generates a sound effect component signal representing a sound effect component of the sound indicated by the sound signal using the sound signal output from the microphone MC, and the generated sound effect component signal is switched to the switch unit 132. Output to. Here, the synthesized sound generation unit 133 uses the audio signal output from the microphone MC to generate an audio signal (sound effect) representing a reverb sound effect when the sound represented by the audio signal is emitted in a predetermined acoustic space. Component signal).
FIG. 4 is a diagram illustrating an example of the contents of the process performed by the synthesized sound generation unit 133. FIG. 4A is a diagram illustrating an example of an audio signal output from the microphone MC, and FIG. 4B is a diagram illustrating an example of a sound effect component signal generated by the synthesized sound generation unit 133. In the figure, the horizontal axis indicates time, and the vertical axis indicates amplitude. As illustrated, the synthesized sound generation unit 133 does not generate an audio signal with a reverb effect applied to an audio signal output from the microphone MC, but the audio represented by the audio signal output from the microphone MC. An audio signal representing only a reverb effect sound that can be generated when sound is emitted in a predetermined space is generated.

  The sound effect component signal output from the notification sound synthesizer 13 is output to the adder 15. In the adder 15, the audio signal output from the audio mixer 14 and the sound effect component signal output from the notification sound synthesis unit 13 are added and output to the speaker SP. Thereby, in addition to the voice represented by the voice data received from the other terminal 10, the sound effect component is emitted from the speaker SP.

<Operation>
Next, the operation of this embodiment will be described. First, the terminals 10 perform voice communication with each other. The terminal 10 transmits the audio data representing the sound collected by the microphone MC to the other terminal 10, receives the audio data received from the other terminal 10 via the communication network 20, and receives the received audio data Is emitted as a sound from the speaker SP. Thereby, a remote conference is realized.

  The user of the terminal 10 operates the operation unit 18 and listens to the speaker's story while pressing the dedicated button B1 every predetermined time length. Here, when the button B1 is pressed, the operation unit 18 outputs an operation signal S1 according to the operated content. The sound collected by the microphone MC is encoded by the CODEC 16 and output to the packet synthesizer 17 as sound data. The packet synthesizing unit 17 packetizes the audio data output from the CODEC 16 and the notification data corresponding to the operation signal S1 output from the operation unit 18, and transmits the packetized data to another terminal 10.

  On the other hand, the packet separation unit 11 separates the voice data and the notification data from the received data, outputs the voice data to the CODEC 12, and outputs the notification data to the notification sound synthesis unit 13. The CODEC 12 decodes the supplied audio data and outputs it to the audio mixer 14. The audio mixer 14 mixes the audio data output from the plurality of CODECs 12 and outputs it to the adder 15.

  The plurality of notification data separated by the packet separation unit 11 is supplied to the continuity determination unit 131 of the notification sound synthesis unit 13. The continuity determination unit 131 switches the switch unit 132 so that the sound effect component signal is output when the values of the supplied notification data are all “OK”, while in other cases, the sound effect sound The switch unit 132 is switched so that the component signal is not output.

  The synthesized sound generating unit 133 of the notification sound synthesizing unit 13 generates a sound effect component signal based on the audio signal output from the microphone MC and outputs it to the adder 15. That is, the notification sound synthesizer 13 outputs the sound effect component signal when the values of the notification data received from the other terminals 10 are all “OK”. On the other hand, when the notification data received from the other terminal 10 includes “NG”, the sound effect component signal is not output when the switch unit 132 is switched.

  The sound effect component signal output from the synthesized sound generation unit 133 is mixed with the audio signal output from the audio mixer 14 in the adder 15 and output to the speaker SP. Thereby, in addition to the sound represented by the sound data received from the other terminal 10, the sound effect component represented by the sound effect component signal is emitted from the speaker SP.

Thus, according to the present embodiment, the terminal 10 generates the sound effect component signal from the sound signal output from the microphone MC, and reproduces the sound effect component when “conduction is OK”. In this way, the listener's listening status can be notified to the speaker without disturbing the utterance.
Moreover, according to this embodiment, even when there are many connection points, since the whole is evaluated comprehensively and it is set as one sound effect, it becomes easy for a speaker to grasp | ascertain the whole.
Further, in the present embodiment, by outputting only the sound effect, not the entire sound to which the sound effect is given, the listener's listening status can be notified more naturally to the speaker.

<Modification>
As mentioned above, although embodiment of this invention was described, this invention is not limited to embodiment mentioned above, It can implement with another various form. An example is shown below. In addition, you may combine each following aspect suitably.
(1) In the above-described embodiment, the case where a remote conference is performed using the terminal according to the present invention has been described. However, the present invention is not limited to this, and for example, when a lecture or lecture is performed via a communication network. The present invention can be applied.

(2) In the above-described embodiment, the synthesized sound generation unit 133 generates a sound effect component signal representing a reverb sound effect using the sound signal output from the microphone MC. For example, it may be an audio signal representing an effect sound such as an echo or surround effect, or any audio signal representing an effect sound (acoustic component) of a spatial effect.
In addition, the sound effect component signal generated by the synthesized sound generation unit 133 is not limited to the sound signal representing the acoustic component, and for example, the sound signal representing the hammer sound whose difference from the sound signal becomes a predetermined degree is used as the sound effect component signal. May be generated as

(3) In the above-described embodiment, the sound effect component is emitted only when the notification data received from the other terminals 10 is all “OK”. However, the present invention is not limited to this. For example, when the majority of the notification data received from the other terminal 10 is “OK”, the sound effect component may be emitted, or the notification received from the other terminal 10 may be used. The sound effect component may be emitted when “OK” in the data is equal to or greater than a predetermined threshold. In short, it is only necessary to output the sound effect component signal when at least one of the number and the ratio of the terminals that are OK among the other terminals 10 satisfies a predetermined condition.

  In the above-described embodiment, the terminal 10 may generate different sound effect component signals according to the number and ratio of the terminals 10 whose notification data is “OK”. In this case, for example, the notification sound synthesizing unit 13 increases the sound effect (the sound effect becomes gorgeous) as the number of pieces of notification data indicating “OK” increases (that is, as the number of people who are listening) increases. A sound effect component signal may be generated. In this case, since the spatial effect on the voice of the speaker becomes gorgeous as the number of people listening to the speaker's story increases, the speaker can speak comfortably as the number of listeners increases. Further, for example, the terminal 10 may increase the sound pressure of the sound effect as the number of the terminals 10 whose notification data is “OK” increases, and for example, the terminal whose notification data is “OK”. You may make it emit the chord according to the number of ten. Further, for example, the higher the proportion of the terminals 10 whose notification data is “OK”, the higher the sound effect may be emitted.

In the above-described embodiment, when the number of people who are listening is small (the number of notification data indicating “NG” is large), the terminal 10 emits an audience noise or a whistle sound instead of the sound effect component. You may do it.
In the above-described embodiment, the sound effect is emitted when “conduction is OK”. However, the present invention is not limited to this. For example, the sound effect is emitted when it is determined as “conduction NG”. It may be configured, and can be appropriately changed according to the design of the terminal 10 or the like. The point is that the listener can be informed of the listening situation (whether the speaker is listening, whether voice communication is normally established, etc.).

(4) In the above-described embodiment, the notification data indicating that the dedicated button B1 is pressed by the user is used. However, the notification data is not limited to this, and the notification sound synthesizing unit 13 is connected to the microphone MC of the terminal 10. The notification data may be generated by analyzing the voice data representing the voice collected by. In short, the terminal 10 may use, as the notification data, data indicating whether or not the audio data transmitted from the terminal 10 is normally played back by another terminal. In short, any information may be used as long as the information indicates that the other party is listening properly (or communication is normally connected).

(5) In the above-described embodiment, whether or not continuity is OK is determined by determining the value of notification data received from another terminal 10, but the determination method of continuity is not limited to this. The voice data received from the terminal 10 may be analyzed and determined according to the analysis result. More specifically, for example, the sound pressure of audio data received from another terminal 10 is detected, and when the detected sound pressure satisfies a predetermined condition, it is determined that the connection is OK. Also good.

As another example, for example, the determination may be made based on whether or not a nod sound is detected from audio data received from another terminal 10. An example of a specific configuration in this case will be described with reference to FIG. In the terminal 10A shown in FIG. 5, the packet separation unit 11, the CODEC 12, the notification sound synthesis unit 13, the adder 15, the CODEC 16, and the packet synthesis unit 17 are the same as the respective units of the terminal 10 in the above-described embodiment. Detailed description is omitted. In FIG. 5, the memory 24 stores nodding voice and collation data representing the characteristics of the nodding voice. The voice analysis unit 23 collates the voice data received from the other terminal 10 with the collation data stored in the memory 21, and the data (hereinafter referred to as “nodding sound”) in which the collation result is determined from the received voice data. Data)). The voice analysis unit 23 outputs continuity information indicating “OK” to the notification sound synthesis unit 13 when the nodding sound data is detected at a predetermined frequency, while “NG” otherwise. Is output to the notification sound synthesizing unit 13. The operation of the notification sound synthesizing unit 13 is the same as that in the above-described embodiment, and the detailed description thereof is omitted here.
That is, in this aspect, whether or not the other party is listening is determined by nodding voice data. As a mode of detection of the nodding voice data, it may be detected by collating with the matching data, or the voice data received from the other terminal 10 is analyzed and the voice that cannot be recognized by the nodding voice. May be detected.

(6) In the above-described embodiment, the nodding operation may be detected by image analysis. In this case, the terminal 10 is provided with photographing means for photographing the user, and the terminal 10 performs image analysis on the video data output from the photographing means, detects the nodding action according to the analysis result, and the nodding action is performed. What is necessary is just to determine whether it was detected.

(7) In the above-described embodiment, the terminal 10 may pan the sound effect to be emitted according to the position information of the other terminal 10. In this case, position data indicating the position of the other terminal 10 is stored for each of the other terminals 10, and the terminal 10 performs sound image localization in a direction corresponding to the determination result for each terminal 10 and the position information of each terminal. You may make it control so that it may do.

(8) In the above-described embodiment, the terminal 10 may detect a speaker's gesture and emit a sound effect according to the detection result. In this case, since the meaning of the gesture differs depending on the country, the database may be different for each country. Specifically, the terminal 10 is provided with a photographing means for photographing the user, and the terminal 10 performs image analysis on the video data output from the photographing means and collates the analysis result registered in a predetermined database. The speaker's gesture is detected by comparing with the data, and the sound effect corresponding to the detected gesture is identified by referring to the database storing the correspondence relationship between the gesture and the sound effect, and the identified sound effect is emitted. You may make it do.

(9) Each unit of the terminal 10 or the terminal 10A in the above-described embodiment may be configured as hardware, and a computer in which a control unit such as a CPU (Central Processing Unit) is stored in storage means such as hardware You may make it implement | achieve as software by running a program. In this case, the program executed by the control unit is a computer-readable recording medium such as a magnetic recording medium (magnetic tape, magnetic disk, etc.), an optical recording medium (optical disk, etc.), a magneto-optical recording medium, and a semiconductor memory. Can be provided in a recorded state. It is also possible to download to each device via a network such as the Internet.

It is a block diagram which shows an example of a structure of a remote conference system. It is a block diagram which shows an example of a structure of a terminal. It is a block diagram which shows an example of a structure of a notification sound synthetic | combination part. It is a figure for demonstrating the content of the process which a synthetic sound production | generation part performs. It is a block diagram which shows an example of a structure of a terminal.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Remote conference system, 10 ... Terminal, 11 ... Packet separation part, 12 ... CODEC, 13 ... Notification sound synthesis part, 14 ... Audio mixer, 15 ... Adder, 16 ... CODEC, 17 ... Packet synthesis part, 18 ... Operation , 20 ... communication network, 24 ... memory, 23 ... voice analysis unit, 22 ... data reception unit, 131 ... continuity determination unit, 132 ... switch unit, 133 ... synthesized sound generation unit.

Claims (9)

Receiving means for receiving data transmitted from other terminals connected via a communication network;
Transmitting means for transmitting voice data representing the sound collected by the sound collecting means to the other terminal;
Using sound data representing the sound collected by the sound collecting means, a sound effect component generating means for generating a sound effect component signal representing a sound effect component of the sound; and
Analyzing the data received by the receiving means, and determining means for determining whether the data satisfies a predetermined condition according to an analysis result;
And a sound emission control means for outputting the sound effect component signal generated by the sound effect component generation means to the sound emission means when the determination result by the determination means is affirmative. apparatus. The receiving means receives data from a plurality of terminals,
The determination unit analyzes the data received by the reception unit for each terminal, determines whether the data satisfies a predetermined condition according to the analysis result for each terminal,
The sound effect component generation means varies the sound effect component signal to be generated according to at least one of the number and the ratio of terminals whose determination result by the determination means is affirmative. The communication apparatus as described in. The receiving means receives data from a plurality of terminals,
The determination unit analyzes the data received by the reception unit for each terminal, determines whether the data satisfies a predetermined condition according to the analysis result for each terminal,
The sound emission control means sends the sound effect component signal to the sound emission means when at least one of the number and the ratio of terminals whose determination result by the determination means is positive satisfies a predetermined condition. The communication device according to claim 1, wherein the communication device outputs the information. Comprising position data storage means for storing position data indicating positions of the plurality of terminals for each of the terminals;
The sound emission control means localizes the sound emitted from the sound emission means in a direction according to position data stored in the position data storage means and a determination result for each terminal by the determination means. It controls as follows. The communication apparatus of Claim 2 or 3 characterized by the above-mentioned. A sound emission mode data storage unit for storing sound emission mode data indicating a mode of sound emission by the sound emission unit for each terminal;
5. The sound emission control unit according to claim 2, wherein the sound emission control unit emits sound in a sound emission mode indicated by a sound emission mode data corresponding to a terminal for which a determination result by the determination unit is positive. The communication apparatus as described in. The determination unit extracts notification data indicating whether or not the voice data transmitted by the transmission unit is normally reproduced in the other terminal from the data received by the reception unit, and the extracted notification data The communication apparatus according to claim 1, wherein the determination is made according to The said determination means detects the sound pressure of the audio | voice data received by the said reception means, and determines whether the detected sound pressure satisfy | fills predetermined conditions. The communication apparatus of any one of Claims. The receiving means receives data including audio data from the other terminal,
The communication apparatus according to any one of claims 1 to 5, wherein the determination unit analyzes voice data included in data received by the reception unit, and determines according to an analysis result. Comprising collation data storage means for storing collation data representing voice or voice characteristics;
The determination unit compares the voice data received by the reception unit with the verification data stored in the verification data storage unit, and determines whether the verification result includes voice data that satisfies a predetermined condition. The communication device according to claim 8, wherein:

Images