JP4245575B2 - COMMUNICATION DEVICE, COMMUNICATION METHOD, AND COMMUNICATION PROGRAM - Google Patents

Description

  The present invention relates to a method for visualizing the state of sound propagation in a partner's space in a remote communication device.

  The volume of sound is important in communication including voice, and a method of visualizing the volume so that the user can easily grasp the volume may be used. For example, in a video / audio communication apparatus, there is a technique using a peak meter as a technique for visualizing the volume input to the apparatus on its own side.

However, there was no method for visualizing the volume reproduced from the remote voice presentation unit. Therefore, in the conventional video / audio communication device, since the volume reproduced from the remote voice presentation unit is unknown, establishment of communication may be hindered (see Non-Patent Document 1).
Fish, R.A. S. Kraut, R .; E. , and Chalfonete, B .; L. The Video Window in Information Communications, Proceedings of the ACM 1990 Conference on Computer Supported Work, CSCW90. 1-11 (1990).

  With the conventional method, you can't know how loud your voice is played on the other party's device, and how much is transmitted to the other party. It is difficult to talk with the other party.

  The objective of this invention is providing the communication apparatus which can grasp | ascertain how much the audio | voice to the other party's sound volume is.

  In order to achieve the above object, the communication device of the present invention is a communication device that performs communication by transmitting and receiving at least audio to and from each other, and presents a volume level measurement when reproducing the audio received from the counterpart device A sound collection unit and a sound volume measured by the presentation sound collection unit when a sound propagation model in a space where the sound is presented to the user is obtained in advance and the sound from the counterpart device is reproduced. A propagation state information generating unit that calculates a propagation state of the voice from the counterpart device in the space from the level and the propagation model, and transmits information of the propagation state to the counterpart device; ing.

  According to the present invention, the communication device on the receiving side obtains a sound propagation model in advance, and at the time of communication, the propagation state of the sound from the communication device on the transmitting side is obtained using the propagation model, and this is transmitted. The user of the transmitting communication device can grasp how the voice he / she is presenting in the space around the user of the receiving communication device. .

  In addition to audio, video in the space is mutually transmitted and received, and the propagation state information generation unit adds the information on the propagation state as an effect to the video in the space and transmits it to the counterpart device. It is good to do.

  According to this, since the communication device on the receiving side adds the sound propagation state from the communication device 20 on the transmitting side to the video as an effect and sends it to the communication device on the transmitting side, the user of the communication device on the transmitting side Can visually grasp how the voice uttered by the user is presented in the space around the user of the communication device on the receiving side.

  And a sound field measuring unit that measures sound volume levels at a plurality of locations in the space when a predetermined test sound is generated, and the propagation state information generating unit is measured by the sound field measuring unit. The propagation model may be calculated using a plurality of volume levels.

  According to this, a propagation model can be calculated from the sound volume levels at a plurality of locations in the sound field measuring unit, and an actual sound propagation state can be estimated.

  The propagation model is an nth-order approximation function indicating a relationship between a volume level measured by the presentation sound collection unit and a volume level at an arbitrary point in the space, and the sound field measurement unit It is good also as measuring the volume level of the place of the number according to the order of the next approximation function.

  According to this, even when a complicated propagation state is shown in the space, the actual propagation state can be grasped with high accuracy by using a high-order propagation model as necessary.

  In addition, for the voice received from the counterpart device, it further includes a waveform match determination unit that determines whether the waveform at the counterpart device matches the waveform at the presentation voice collection unit, and the propagation state An information generation part is good also as transmitting the information of the said propagation state to the said other party apparatus about the period determined to be the coincidence in the said waveform coincidence determination part.

  According to this, the communication device on the receiving side determines the match between the waveform on the transmitting side and the waveform on the receiving side, and transmits the propagation state information when they match, so that the communication device on the receiving side is surrounded by Even when a large amount of noise is generated, the influence of the noise can be removed, and the user of the transmitting communication device can be made aware of the propagation state of the voice transmitted from the transmitting communication device.

  According to the present invention, the communication device on the receiving side obtains a sound propagation model in advance, and at the time of communication, the propagation state of the sound from the communication device on the transmitting side is obtained using the propagation model, and this is transmitted. The user of the transmitting communication device can grasp how the voice he / she is presenting in the space around the user of the receiving communication device. .

  Embodiments for carrying out the present invention will be described in detail with reference to the drawings. In this embodiment, a communication system that transmits and receives video and audio bidirectionally is illustrated. The communication system according to the present embodiment adds volume information on the receiver side as an effect to the video on the receiver side presented to the speaker side when sound is transmitted from the speaker to the receiver. Is presented to the sender. The effect is a video effect added to the video. For example, the effect is displayed by superimposing an effect image on the video (superimposition).

  FIG. 1 is a block diagram showing a schematic configuration of a communication system according to the present embodiment. Referring to FIG. 1, in the communication system of the present embodiment, a communication device 10 and a communication device 20 are connected to each other via a communication network 30. Here, a configuration having two communication devices is illustrated, but the present invention is not limited to this configuration, and it is sufficient if there are a plurality of communication devices.

  FIG. 2 is a diagram illustrating a configuration of each device and an arrangement of each unit of the communication system according to the present embodiment. FIG. 3 is a block diagram showing the configuration of the communication apparatus according to the present embodiment. The communication device 10 is used for the user a109, and the communication device 20 is used for the user b110. Here, communication between the user a109 and the user b110 is assumed.

  Referring to FIGS. 1 and 2, the communication devices 10 and 20 have the same configuration in order to enable bidirectional communication.

  The communication device 10 includes a sound collection unit 101, a voice presentation unit 102, an imaging unit 103, a video presentation unit 104, a presentation voice sound collection unit 105, a sound field measurement unit 106, and an effect generation unit 108. The sound field measurement unit 106 includes a plurality of microphone input units 107 arranged in front of the audio presentation unit 102 and the video presentation unit 104.

  Similarly, the communication device 20 includes a sound collection unit 201, a sound presentation unit 202, an imaging unit 203, a video presentation unit 204, a presentation sound collection unit 205, a sound field measurement unit 206, and an effect generation unit 208. . The sound field measurement unit 206 includes a plurality of microphone input units 207. The sound collecting unit 201, the sound presenting unit 202, the image capturing unit 203, the video presenting unit 204, the presented sound collecting unit 205, the sound field measuring unit 206, and the effect generating unit 208 are the sound collecting unit 101 and sound in the communication device 10. This is the same as the presentation unit 102, the imaging unit 103, the video presentation unit 104, the presentation sound collection unit 105, the sound field measurement unit 106, and the effect generation unit 108. Here, the communication device 10 will be described.

  The sound collecting unit 101 collects the utterance of the user a 109 and sends the sound to the communication device 20 on the other side.

  The imaging unit 103 captures an image of the user a 109 and sends the image to the effect generation unit 108.

  The voice presentation unit 102 plays back the voice received from the communication device 20 on the other side and presents it to the user a109.

  The video presentation unit 104 plays back the video from the communication device 20 on the other side and presents it to the user a109.

  The presentation sound collection unit 105 is installed in front of the voice presentation unit 102, measures the volume level of the reproduced sound by the voice presentation unit 102, and notifies the effect generation unit 108 of the volume level.

  The sound field measurement unit 106 includes a plurality of (10 in the present embodiment) microphone input units 107 arranged in a plane within the angle of view of the imaging unit 103, and each of the reproduced sounds from the audio presentation unit 102. The volume level at the microphone input unit 107 is measured, and the volume level is notified to the effect generation unit 108.

  The effect generation unit 108 calculates a spatial propagation quadratic approximation function that is a propagation model for calculating the propagation state of the sound emitted from the audio presentation unit 102 before the apparatus is used for communication. . The spatial propagation quadratic approximation function is calculated based on the sound volume level obtained by the sound collection unit 105 and the sound level obtained by the sound field measurement unit 106 by generating a predetermined test sound from the sound presentation unit 102. The

  Further, the effect generation unit 108 determines the volume level of the sound from the communication device 20 on the other side, which is reproduced by the voice presentation unit 102 and measured by the voice presentation sound collection unit 105 when the apparatus is used for communication. Then, the sound volume level at an arbitrary point in the space is calculated by substituting it into a spatial propagation quadratic approximation function obtained in advance. Then, the effect generation unit 108 superimposes the obtained volume level information in the space as an effect on the image of the space imaged by the imaging unit 103 and sends it to the communication device on the other side.

  Note that the installation position and shooting direction of the imaging unit 103 are fixed with respect to the sound field measuring unit 106, and the correspondence between the image by the imaging unit 103 and the plane coordinates of the sound field measuring unit 106 is obtained in advance. It is good. Alternatively, the correspondence between the captured image and the plane coordinates of the sound field measuring unit 106 may be obtained from the image captured by the image capturing unit 103 and the arrangement of the microphone input unit 107 of the sound field measuring unit 106. Further, the correspondence relationship between the captured image and the plane coordinates of the sound field measuring unit 106 may be obtained from the installation position and shooting direction of the imaging unit 103.

  Next, processing for calculating a spatial propagation quadratic approximate function will be described.

  FIG. 4 is a flowchart showing a spatial propagation quadratic approximate function calculation process of the communication apparatus according to the present embodiment. Here, a plurality of test sounds having different volumes are used, and a number indicating each test sound is m.

  The spatial propagation quadratic approximation function is calculated before the communication device is used for communication. Referring to FIG. 4, first, the communication apparatus according to the present embodiment presents a predetermined test sound from the voice presentation unit 102 (step A101). In this state, the communication apparatus collects the voice by the presentation voice sound collection unit 105 and measures the volume level (step A102), and uses the ten microphone input units 107 of the sound field measurement unit 106 to set the microphone. The sound volume level (V) at the position (x, y) where is arranged is measured (step A103).

  The test sound used here is preferably a voice having either an average fundamental frequency of 130 Hz when a man speaks or an average fundamental frequency of 245 Hz when a woman speaks. In addition, this test sound is one of 20, 30, 40, 50, 60, 70, and 80 dB in which the volume level (Wm) measured by the presentation sound collecting unit 105 is a main volume range when a person speaks. It is preferable that

  The test sound (Wm) measured by the presentation sound collection unit 105 and the measured values of the coordinate positions (x, y) and volume levels (V) of the ten microphone input units 107 are substituted into the equation (1). Each coefficient (a, b, c) is obtained (step A104). As the volume (V), an average value of volumes obtained for each frequency (130 Hz, 245 Hz) by a plurality of test sounds may be used.

次に、通信装置は、得られた各係数に基づき、提示音声集音部１０５での音量レベルがＷであるときの空間内の任意点（ｘ，ｙ）での音の伝播状態を近似する二次関数を式（２）として求める（ステップＡ１０５）。音の伝播状態は、音がどの領域に有効に到達するかを示すものである。例えば、平常時のノイズがある環境において、有意な音声が人間の聴覚により認識可能な程度で到達するか否かを基準とすることとしてもよい。

Next, the communication apparatus approximates the sound propagation state at an arbitrary point (x, y) in the space when the volume level in the presentation sound collection unit 105 is W based on the obtained coefficients. A quadratic function is obtained as equation (2) (step A105). The sound propagation state indicates which region the sound effectively reaches. For example, in an environment where there is a normal noise, it may be based on whether or not a significant voice reaches a level that can be recognized by human hearing.

なお、ここで通信装置１０（ユーザａ１０９側）における空間伝播二次近似関数の算出には通信装置２０（ユーザｂ１１０）における集音部２０１での入力の音量レベルでなく、通信装置１０の提示音声集音部１０５における音量レベルを用いている。これは、通信装置の使用に際し、通信装置２０における集音部２０１のゲインなどのセッティングや、通信装置１０における音声提示部１０２のボリュームなどのセッティングがユーザにより自由に変更される可能性があり、セッティングが変更された場合にも空間伝播二次近似関数が影響を受けないようにしておくためである。

Here, the calculation of the spatial propagation quadratic approximation function in the communication device 10 (user a 109 side) is not the input sound volume level in the sound collection unit 201 in the communication device 20 (user b110), but the voice presented by the communication device 10. The volume level in the sound collection unit 105 is used. This is because the user may freely change settings such as the gain of the sound collection unit 201 in the communication device 20 and the volume of the voice presentation unit 102 in the communication device 10 when the communication device is used. This is to prevent the spatial propagation quadratic approximation function from being affected even when the setting is changed.

  Moreover, it is good also as measuring the stationary noise level (Vavg) of the environment which installs an apparatus with each microphone input part 107 of the sound field measurement part 106 before using an apparatus. The value of the steady noise level can be used to determine the presence / absence of the utterance of the user b110 from the volume level of the presentation sound collection unit 102.

  When the volume level (V) measured by the microphone input unit 107 is larger than the stationary noise level (Vavg), it is considered that the sound measured by the microphone input unit 107 is a reproduced sound from the voice presentation unit 102. For example, the detection of the playback sound may be a condition for superimposing the effect 401 on the video.

  Further, in the present embodiment, when the apparatus is used for communication, the spatial second-order function and the presentation voice collecting unit that are obtained in advance, not from the volume level measured in real time by the sound field measuring unit 106. The volume at an arbitrary point (x, y) is obtained from the real-time volume obtained at 105. Therefore, the sound field measurement unit 106 is only required when calculating the spatial quadratic approximation function, and is not always necessary thereafter. Since the microphone input unit 107 of the sound field measurement unit 106 may collect sound other than the speech of the user b110, for example, when the process for obtaining the spatial quadratic approximation function is completed, the sound field measurement unit 106 is removed. It is good to do.

  Next, effect superimposition processing by the effect generation unit 108 will be described.

  FIG. 5 is a flowchart showing the effect superimposing process of the communication device according to the present embodiment. Referring to FIG. 5, during communication between the communication device 10 and the communication device 20, first, when the voice of the user b 110 is input to the sound collection unit 201 of the communication device 20 (step B <b> 101), the voice presentation unit 102 of the communication device 10. Plays the voice and presents it to the user a109 (step B102). The presentation sound collection unit 105 measures the volume level of the voice reproduced by the voice presentation unit 102 (step B103), and the effect generation unit 108 uses the spatial propagation quadratic approximation function of Equation (2) to set the volume at any point. The level (V) is calculated (step B104).

  On the other hand, in parallel with this, an image is acquired by the imaging unit 103 of the communication device 10 (step C101). Then, the effect generation unit 108 obtains the correspondence between the acquired image and the plane coordinates of the sound field measurement unit 106, and matches the coordinate systems (step C102). Further, the effect generation unit 108 obtains a region satisfying V = Vavg (steady noise level) and W = W + 5 (assuming that the initial value of W is 0) in the expression (2), and identifies the boundary of the region. One wave of the effect 401 is created (step C103). The effect 401 refers to an image effect that makes it possible to distinguish a region of a predetermined coordinate from other regions. Here, as an example, the effect 401 is identified by a wave indicated by a curve.

  Next, it is determined whether or not the volume level (V) at an arbitrary point is greater than the steady noise level (Vavg). If not, the transparency is set to 0 and the effect 401 is superimposed on the video (step D102). The effect 401 is superimposed on the video as 100 (step D103). Here, transparency 0 indicates that it is transparent, and transparency 100 indicates that it is opaque. Therefore, if the sound level is higher than the steady noise level, the effect 401 of an opaque curve is presented as the sound propagates there. On the other hand, if the sound level is not higher than the steady noise level, the curve of the effect 401 is made transparent so that the sound does not propagate so far. The video on which the effect 401 is superimposed is sent to the communication device 20, is played back by the video presentation unit 203 of the communication device 20, and is presented to the user b110 (step D104).

  FIG. 6 is a diagram illustrating an example of correspondence between the coordinate axes of the plane of the sound field measurement unit 106 in the communication device 10 and the coordinate axes of the video presented to the video presentation unit 204 of the communication device 20 in the present embodiment. FIG. 7 is a diagram illustrating an example of an image presented by the image presentation unit 204 of the communication device 20 that is associated with the plane coordinate axis of the sound field measurement unit 106 in the communication device 10.

  6 and 7, the installation position and the imaging direction of the imaging unit 103 are fixed, and each microphone input unit 107 of the sound field measuring unit 106 is installed in a planar shape. From the image captured by the image capturing unit 103, the positions of the three microphone input units 107 whose coordinates are known among the microphone input units 107 installed in a plane are obtained by image recognition, and the coordinate axes and sound of the captured image are obtained. The association with the coordinates on the plane of the field measurement unit 106 is obtained. These processes may be performed in advance before using the apparatus for communication. Thereby, the coordinates of the video presented to the user b110 by the video presentation unit 204 of the communication device 20 and the coordinates of the sound field measurement unit 106 of the communication device 10 can be matched. The effect 401 can be satisfactorily superimposed.

  FIG. 8 is a diagram illustrating an example when the volume of the video on which the effect 401 presented to the user in the video presentation unit is superimposed is low. FIG. 9 is a diagram illustrating an example when the volume of the video on which the effect presented to the user is superimposed by the video presenting unit is high.

  When obtaining the spatial propagation quadratic approximate function, the voice presentation unit 102 generates a plurality of test sounds. Here, the test sound is assumed to have five levels of sound volume at which the sound volume level collected by the presentation sound sound collecting unit 105 is from 0 dB to 5 dB intervals (Wm).

Then, for each test sound, the effect generation unit 108 obtains an area where the volume level (V) is estimated to exceed the steady noise level (Vavg) from the volume obtained by each microphone input unit 107, and corresponds to that area. The video area presented by the video presentation unit 204 is obtained.
By performing this for five levels of volume, five regions (region a501, region b502, region c503, region d504, and region e505 are obtained.

The region obtained for each volume level can be regarded as the sound propagation range when the sound of that volume level is reproduced. Then, by drawing the curve of the effect 401 at the boundary of each region, the effect 401 has a ripple shape, and the video presentation unit 204 of the communication device 20 generates a ripple from the voice presentation unit 102 of the communication device 10. looks like.

  The effect 401 including these curves is preliminarily superimposed on the video imaged by the imaging unit 103 so that it can always be presented to the video presentation unit 204 together with the video image. In the initial state, the transparency of the effect 401 is set to 0, and the effect 401 is drawn transparently.

  When the user b110 speaks in the communication device 20, the video presentation unit 204 of the communication device 20 causes the ripples of the effect 401 within the sound propagation range to be transparent according to the volume level in the presentation sound collection unit 102 of the communication device 10. Is set to 100.

  If the volume level at the voice presentation unit 102 on the user a 109 side is small, the effect 401 does not spread so much and the number of waves forming the effect 401 decreases (see FIG. 8). If the volume level at the voice presentation unit 102 on the user a 109 side is high, the effect 401 spreads greatly and the number of waves forming the effect 401 increases (see FIG. 9).

  As described above, according to the present embodiment, the communication device 10 on the reception side obtains a spatial propagation approximate quadratic function of sound in advance, and the communication device 10 on the reception side receives the spatial propagation approximation quadratic function during communication. Is used to determine the propagation state of the voice from the communication device 20 on the transmission side, and the result is superimposed on the video of the communication device 10 on the reception side and sent to the communication device 20 on the transmission side as an effect. The user b110 can visually grasp how the voice he / she uttered is presented in the space around the user a109 of the communication device 10.

  In the present embodiment, the actual sound propagation state in the space is calculated by calculating the spatial propagation quadratic approximation function obtained from the volume level measured by the microphone input units 107 installed in a plurality of places in the space. Can be estimated. However, the present invention is not limited to a configuration that approximates a spatial propagation function as a quadratic function. By increasing the number of microphone input units 107 of the sound field measuring unit 106, it is possible to calculate a high-order function when calculating the spatial propagation function of sound. High-order functions that can represent complex sound propagation enable more accurate sound propagation estimation. Even when the propagation of sound in the space shows a complicated propagation state that cannot be simply estimated from the volume reproduced by the speech presentation unit 102, it is actually possible to use a high-order propagation model as necessary. Can be grasped with high accuracy.

  In the present embodiment, an example in which the sound propagation state on the video is superimposed as the effect 401 composed of a plurality of curves has been shown, but the present invention is not limited to this. Any means may be used as long as the propagation state of the voice on the receiver side is fed back to the transmitter side. For example, the sound propagation state may be mapped on the video as another effect such as a color change and fed back. Also, the means for feeding back the sound propagation state is not limited to that by video.

  Moreover, although this embodiment illustrated the communication system which transmits / receives a bidirectional | two-way image | video and audio | voice, this invention is not limited to this. Video transmission / reception is not necessarily required as long as audio transmission / reception is possible. In the case of voice communication, it is only necessary to feed back the voice propagation state by means other than video. For example, a simple display device that can display only the voice propagation state may be used.

  Moreover, in this embodiment, although the presentation audio | voice sound collection part 105 measured the volume level of the audio | voice reproduced | regenerated by the audio | voice presentation part 102, this invention is not limited to this. The presented voice collecting unit 105 measures the level of the input signal to the speaker (not shown) of the voice presenting unit 102 as the volume level, and uses the input signal level in the calculation of the propagation model and the actual propagation state. It is good as well.

  Next, another embodiment of the present invention will be described.

  FIG. 10 is a block diagram illustrating a configuration of a communication device according to another embodiment.

  Referring to FIG. 10, the communication device 60 according to the present embodiment includes a sound collection unit 101, a sound presentation unit 102, an imaging unit 103, a video presentation unit 104, a presentation sound collection unit 105, a sound field measurement unit 106, and an effect generation unit. 108 and a waveform match determination unit 601.

  FIG. 10 shows only the communication device used by the user a109, but the communication device used by the user b110 communicating with the user a109 has the same configuration. Although not shown, the communication device 70 used by the user b110 includes a sound collection unit 201, a voice presentation unit 202, an imaging unit 203, a video presentation unit 204, and a presentation voice sound collection unit 205 corresponding to each unit of the communication device 60. Assume that a sound field measurement unit 206, an effect generation unit 208, and a waveform match determination unit 701 are included.

  The sound collecting unit 101, the sound presenting unit 102, the imaging unit 103, the video presenting unit 104, the presented sound collecting unit 105, and the sound field measuring unit 106 in FIG. 10 are the same as those shown in FIG.

  The waveform coincidence determination unit 601 determines whether or not the transmission side waveform and the reception side waveform match.

  When the user b110 speaks, the sound is collected by the sound collection unit 201 of the communication device 70. A waveform obtained by chronologically acquiring the volume level at the sound collecting unit 201 at this time is defined as a transmission side waveform 1101. Note that the transmission side waveform 1101 is a waveform measured by the communication device 70 used by the user b110, and this waveform may be notified to the communication device 60 used by the user a109. Alternatively, the transmission-side waveform 1101 may be obtained from voice data reproduced by the voice presentation unit 102 on the communication device 10 side.

  On the other hand, a waveform obtained by chronologically obtaining the volume level in the presentation sound collection unit 105 in the communication device 60 is defined as a reception-side waveform 1102.

  The transmission side waveform 1101 is acquired when the user b110 speaks and the volume level is changed by the sound collection unit 201, and is acquired when the sound collection unit 201 measures a one-second silence period. To do. The time between them is the input time (T).

  The reception-side waveform 1102 starts to be acquired when voice is reproduced by the voice presentation unit 102 of the communication device 60 on the user a 109 side, and the input time (measured by the sound collection unit 201 of the communication device 70 on the user b 110 side). When T) elapses, acquisition ends.

  The sound collection by the presentation sound collection unit 105 has a delay with respect to the reproduction by the sound presentation unit 102, and a necessary sound waveform can be missed by completing the waveform acquisition at the input time (T). Although it is conceivable, since the time from the start to the end of the sound collection unit 201 on the user b110 side includes a one-second silent period, the voice waveform at the time when the user b110 speaks is not missed.

  The waveform matching determination unit 601 performs matching processing of both waveforms within a certain error range (here, 3 dB) in order to determine the match between the cut-out transmitting-side waveform 1101 and the receiving-side waveform 1102. In this matching process, if the two waveforms are within a certain error range, it is determined that they match. If it is determined that they do not match, the sound input to the presentation sound collection unit 105 is a sound other than the reproduction sound from the sound presentation unit 102, and the sound input to the sound collection unit 201 on the user b 110 side It is considered not. On the other hand, if it is determined that the two waveforms match, it is considered that the sound is input to the microphone input unit 107 of the sound field measurement unit 106.

  This determination result is notified from the waveform match determination unit 601 to the effect generation unit 108. The effect generation unit 108 sets the transparency of the effect 401 to 0 when notified that the waveforms do not match.

  FIG. 11 is a flowchart showing a waveform match determination process by the waveform match determination unit. Referring to FIG. 11, when the user b110 inputs a sound to the sound collection unit 201 of the communication device 70 (step E101), sound collection is started by the sound collection unit 201 of the communication device 70 (step E102). The waveform collected here becomes the transmission side waveform 1101.

  The sound is transmitted from the communication device 70 to the communication device 60. Then, the voice is reproduced by the voice presentation unit 102 of the communication device 60 and presented to the user a 109 (step E103). The voice reproduced by the voice presentation unit 102 is collected by the presentation voice collection unit 105 (step E104). Therefore, the waveform match determination unit 601 starts the match determination between the transmission side waveform and the reception side waveform (step E105).

  When the sound collecting unit 201 of the communication device 70 detects a one-second silence state, the input time (T) is calculated at that time (step E106). When time T elapses from the start of the match determination, the match determination by the waveform match determination unit 601 of the communication device 60 ends (step E107).

  Next, the waveform matching determination unit 601 of the communication device 60 determines whether or not the transmission side waveform and the reception side waveform match as a result of the matching process (step E108). If the transmitting side waveform and the receiving side waveform match, the process is repeated as it is. If the transmitting side waveform and the receiving side waveform are not similar, the effect generating unit 108 sets the transparency of the effect 401 to 0 (step E109).

  FIG. 12 is a diagram showing the relationship between the transmitting side waveform and the receiving side waveform. Referring to FIG. 12, a transmission side waveform 1101 and a reception side waveform 1102 are shown. Although there is a delay of t (ms) between the transmission side waveform and the reception side waveform, the portion (a) of the transmission side waveform is similar to the portion (b) of the reception side waveform. The waveform match determination unit 601 determines that this part matches, and the effect generation unit 108 presents the effect 401 in this part. On the other hand, the waveform appearing in the portion (c) or the portion (d) of the reception side waveform is not in the transmission side waveform 1101. The waveform match determination unit 601 determines that there is no match in this part, and the effect generation unit 108 sets the transparency of the effect 401 in this part to 0 and makes the effect 401 transparent.

  As described above, according to the present embodiment, the communication device 10 on the reception side determines whether the transmission side waveform and the reception side waveform match, and when they match, the opaque effect 401 is superimposed. Even when a large noise is generated around the communication device 10 on the receiving side other than the voice presentation unit 102, the influence of the noise is removed, and the propagation state of the voice transmitted from the communication device 10 on the transmitting side is determined. The user b110 of the communication device 10 can grasp this.

  Note that the communication device in each of the embodiments described above can be realized by a software program and a computer that executes the software program. The software program can be recorded on a recording medium, or can be provided through a network.

It is a block diagram which shows schematic structure of the communication system by this embodiment. It is a figure which shows the structure of each apparatus of the communication system by this embodiment, and arrangement | positioning of each part. It is a block diagram which shows the structure of the communication apparatus by this embodiment. It is a flowchart which shows the space propagation quadratic approximation function calculation process of the communication apparatus by this embodiment. It is a flowchart which shows the effect superimposition process of the communication apparatus by this embodiment. It is a figure which shows an example of a response | compatibility with the coordinate axis of the plane of the sound field measurement part in the communication apparatus of a receiving side, and the coordinate axis of the image | video presented to the video presentation part of the communication apparatus of a transmission side. It is a figure which shows an example of the image | video shown in the image | video presentation part of the communication apparatus of a transmission side matched with the coordinate axis of the plane of the sound field measurement part in the communication apparatus of a reception side. It is a figure which shows an example when the sound volume of the image | video with which the effect shown to a user is superimposed by the image | video presentation part is low. It is a figure which shows an example when the sound volume of the image | video on which the effect shown to a user is superimposed by the image | video presentation part is large. It is a block diagram which shows the structure of the communication apparatus by other embodiment. It is a flowchart which shows the waveform matching determination process by a waveform matching determination part. It is a figure which shows the relationship between a transmission side waveform and a receiving side waveform.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10, 60 Communication apparatus 101 Sound collection part 102 Audio | voice presentation part 103 Image pick-up part 104 Image | video presentation part 105 Presented sound collection part 106 Sound field measurement part 107 Microphone input part 108 Effect generation part 109 User a
110 User b
20, 70 Communication device 201 Sound collection unit 202 Audio presentation unit 203 Imaging unit 204 Video presentation unit 205 Presented sound collection unit 206 Sound field measurement unit 207 Microphone input unit 208 Effect generation unit 208
30 communication network 401 effect 501 area a
502 Region b
503 region c
504 area d
505 area e
61 Waveform coincidence determination unit 1101 Transmission side waveform 1102 Reception side waveform A101 to A105, B101 to B104, C101 to C103, D101 to D1041, E101 to E109 Step

Claims (10)

A communication device that performs communication by transmitting and receiving at least voice to each other,
A presentation sound collector that measures the volume level when the sound received from the other device is played,
A voice propagation model in a space where the voice is presented to the user is obtained in advance, and when the voice from the counterpart device is reproduced, the volume level measured by the presentation voice collecting unit and the propagation model A propagation state information generating unit that calculates a propagation state of the voice from the counterpart device in the space and transmits information on the propagation state to the counterpart device. In addition to audio, video in the space is sent and received mutually,
The communication apparatus according to claim 1, wherein the propagation state information generation unit adds the propagation state information to the video in the space as an effect and transmits the effect to the counterpart apparatus. A sound field measuring unit that measures sound volume levels at a plurality of locations in the space when a predetermined test sound is generated;
The communication apparatus according to claim 1, wherein the propagation state information generation unit calculates the propagation model using a plurality of volume levels measured by the sound field measurement unit. The propagation model is an nth-order approximation function indicating a relationship between a volume level measured by the presentation sound collection unit and a volume level at an arbitrary point in the space;
The communication apparatus according to claim 3, wherein the sound field measurement unit measures a volume level at a number corresponding to an order of the n-order approximation function. For the voice received from the counterpart device, further includes a waveform match determination unit that determines whether the waveform at the counterpart device matches the waveform at the presentation voice collection unit,
5. The propagation state information generation unit according to claim 1, wherein the propagation state information is transmitted to the counterpart device for a period determined to be coincident by the waveform coincidence determination unit. Communication device. A communication method for performing communication by transmitting and receiving at least voice to and from each other in a communication device,
Obtaining a voice propagation model in advance in a space where the voice is presented to the user;
Measuring the volume level when the audio received from the other device is played back;
From the volume level measured when the sound from the counterpart device is reproduced and the propagation model, a propagation state of the sound from the counterpart device in the space is calculated, and the propagation state Transmitting information to the counterpart device. In addition to audio, video in the space is sent and received mutually,
The communication method according to claim 6, wherein the propagation state information is added to an image in the space as an effect and transmitted to the counterpart device. Measure the volume level of multiple places in the space when a predetermined test sound is uttered,
The communication method according to claim 6 or 7, wherein the propagation model is calculated using the sound volume levels measured at a plurality of locations in the space.   For the voice received from the counterpart device, it is determined whether or not the waveform at the counterpart device matches the waveform measured by the own device, and the propagation state information is determined for the period determined to match. The communication method according to claim 6, wherein the communication method is transmitted to the counterpart device.   The communication program for making a computer perform the operation | movement of each step in the communication method of any one of Claims 6-9.

Images