JP7137033B1 - Communication terminal and communication method - Google Patents

Abstract

The object of the present invention is to reduce the voice of another person superimposed on the voice of a speaker due to proximity between terminals in communication for transmitting and receiving voice. A first communication unit that communicates with a destination terminal of a call destination using a first network, a sound collection unit that collects ambient sound and converts it into first sound data, and a second network. wirelessly communicates with a second user terminal of a second user who is making a call within a predetermined distance from the communication terminal, and receives second audio data based on audio collected by the second user terminal A communication terminal, comprising: a communication unit; and a voice processing unit that reduces first mixed data based on a second user's voice mixed in first voice data. [Selection drawing] Fig. 2

Description

The present invention relates to a communication terminal and communication method.

BACKGROUND ART Techniques for reducing howling and echoes that occur between adjacent terminals in communication that transmits and receives voice over a network are known (see Patent Document 1, for example).

JP 2013-251630 A

However, when the terminals are close to each other, the voice collected by one terminal may include the voice of the speaker of the one terminal and the voice of another person who is talking on another terminal. In this case, it may be difficult for a listener at the destination of the call to hear the voice. Techniques for reducing howling and echo have found it difficult to reduce such voice mixed in with the speaker's voice.

Therefore, the present invention has been made in view of these points. aim.

In a first aspect of the present invention, a communication terminal operated by a first user includes a first communication unit that communicates with a called terminal of a called party using a first network, and collects surrounding sounds. and a second user terminal of a second user who is talking within a predetermined distance from the communication terminal using a second network different from the first network. A second communication unit that wirelessly communicates and receives second audio data based on sound collected by the second user terminal, the first audio data converted by the sound collection unit, and the second communication unit a voice processing unit that reduces first mixed data based on the second user's voice mixed in the first voice data based on the received second voice data, wherein the first communication unit reduces the A communication terminal is provided in which a voice processing unit transmits communication data obtained by reducing the first mixed data from the first voice data to the destination terminal of the destination.

The audio processing unit controls the ratio between the volume of the first user's voice indicated by the first audio data and the volume of the second user's voice indicated by the first audio data, and the volume of the first user's voice indicated by the second audio data. and a separation generating unit that generates two separated voice data by performing independent component analysis on the first voice data and the second voice data based on the fact that the ratio between the volume of the voice of the second user and the volume of the voice of the second user is different , a correlation coefficient in a time series between one of the two separated audio data and the first audio data, and the other of the two separated audio data and the first audio data. a selection unit that compares the correlation coefficients in the time series of the First separated audio data reduced from the first audio data, and second separated audio obtained by reducing second mixed data based on the first user's voice mixed in the second audio data from the second audio data data.

It further comprises a timing adjustment unit that adjusts the timing of the time axis of the first audio data and the timing of the time axis of the second audio data so as to be close to each other, and the audio processing unit adjusts the timing adjusted by the timing adjustment unit. The first mixed data may be reduced from the first audio data using one audio data.

The timing adjustment unit may determine the timing adjustment amount by comparing time stamp information added to the first audio data and time stamp information added to the second audio data.

When the first communication unit starts communication or during communication of the first communication unit, the second communication unit receives a notification signal transmitted from the second user terminal, the A wireless connection may be made with the second user terminal.

The notification signal is a BLE advertising signal, the second communication unit establishes a BLE connection with the second user terminal based on the reception of the advertising signal, and the first communication unit terminates the communication. , the BLE connection with the second user terminal may be released.

The second communication unit may cancel the pairing setting with the second user terminal in response to the communication being terminated by the first communication unit.

When the reception strength of the notification signal received by the second communication unit exceeds a threshold, a display control unit that displays on the display unit that the second user terminal exists, and based on the operation by the first user, an acquisition unit configured to acquire information indicating whether or not to connect to the second user terminal, wherein the second communication unit acquires information indicating that the acquisition unit connects to the second user terminal; wirelessly connected to the second user terminal.

Further comprising a storage unit for storing setting information indicating that the second communication unit automatically establishes a wireless connection with another terminal including the second user terminal, wherein the second communication unit automatically establishes the wireless connection. When the storage unit stores the setting information indicating the setting information, wireless connection with the second user terminal may be established in response to the reception strength of the received annunciation signal exceeding a threshold.

The second communication unit, when wirelessly communicating with the second user terminal, establishes a wireless connection with the second user terminal in response to a reception strength of the received annunciation signal becoming equal to or less than a threshold. may be released.

The second communication unit may start transmitting a notification signal when the first communication unit starts communication.

The second communication unit transmits a notification signal when an application for communication by the first communication unit is executed in the communication terminal, or when the operation of the application returns from the background. may start.

The second communication unit may end transmission of the notification signal in response to the end of communication by the first communication unit.

A second aspect of the present invention is a communication method in which a communication terminal operated by a first user communicates with a called terminal of a called party using a first network, wherein surrounding sounds are collected and converting into voice data; wirelessly communicating with a second user terminal of a second user who is talking within a predetermined distance from the communication terminal using a second network different from the first network; a step of receiving second audio data based on audio collected by a second user terminal; and mixing the first audio data with the first audio data based on the first audio data and the received second audio data. reducing first mixed data based on the voice of a second user; and transmitting communication data obtained by reducing the first mixed data from the first voice data to the called party terminal of the called party, Provide a method of communication.

Advantageous Effects of Invention According to the present invention, it is possible to reduce the voice of another person superimposed on the voice of the speaker due to the proximity of the terminals in the communication of sending and receiving voice.

1 shows an example of a communication system S according to this embodiment. 1 shows a configuration example of a communication terminal 100 according to this embodiment. A first example of the operation flow of the communication system S according to the present embodiment is shown. A second example of the operation flow of the communication system S according to this embodiment is shown. A third example of the operation flow of the communication system S according to this embodiment is shown. A fourth example of the operation flow of the communication system S according to this embodiment is shown.

<Example of communication system S>
FIG. 1 shows an example of a communication system S according to this embodiment. The communication system S is a system for voice communication using a first network (N1 in FIG. 1) by a terminal operated by a user. The example of FIG. 1 is an example in which the first user 11 uses the communication terminal 100 to communicate with the called terminal of the called party, and the second user 12 uses the second user terminal 20 to communicate with the called terminal of the called party. ((1) in FIG. 1). Here, the third user terminal 30 is the called party terminal of the first user 11 , and the third user 13 is the operator of the third user terminal 30 . Similarly, the destination terminal of the second user 12 is assumed to be the fourth user terminal 40 and the operator of the fourth user terminal 40 is assumed to be the fourth user 14 .

Here, when the communication terminal 100 and the second user terminal 20 are close to each other, the communication terminal 100 may collect the voice of the second user 12 . In this case, the third user 13 hears the voice of the first user 11 mixed with the voice of the second user as noise, making it difficult to hear the voice of the first user 11 . In the example shown in FIG. 1, when the first user 11 says, "Thank you very much," and the second user says, "It's nice weather today, isn't it?" Nice to meet you." is superimposed on the voice saying "It's nice weather today, isn't it?"

In this state, the second user terminal 20 also collects the voice of the first user 11 . In this case, the fourth user 14 hears the voice of the second user mixed with the voice of the first user 11 as noise, which makes it difficult to hear the voice of the second user. In the example shown in FIG. 1, the fourth user 14 hears the voice "It's nice weather today, isn't it?" superimposed on the voice "Yoroshiku onegaishimasu".

Therefore, in the communication system S, the communication terminal 100 communicates with the second user terminal 20 using the second network (N2 in FIG. 1) ((2) in FIG. 1), and the second user terminal 20 collects sound Acquire an audio signal. The communication terminal 100 operates to reduce the signal component based on the voice of the second user 12 who is the other person from the voice signal of the first user 11 who is the speaker of the communication terminal 100 . Similarly, the second user terminal 20 communicates with the communication terminal 100 using the second network ((2) in FIG. 1), acquires the audio signal collected by the communication terminal 100, and obtains the audio signal of the second user 12 may operate to reduce the signal component based on the first user's 11 voice. The communication terminal 100 of such a communication system S will be described below. The second user terminal 20 has the same configuration as the communication terminal 100 and preferably operates to reduce the signal component based on the voice of the first user 11 from the voice signal of the second user 12 .

<Configuration example of communication terminal 100>
FIG. 2 shows a configuration example of the communication terminal 100 according to this embodiment. Communication terminal 100 is a terminal operated by first user 11 . The communication terminal 100 may be any terminal that has a microphone, a speaker, earphones, headphones, or the like and has a communication function for making a call. The communication terminal 100 is, for example, a mobile terminal such as a smart phone, an information processing device such as a PC, a headset, VR goggles, or the like. The communication terminal 100 includes a first communication unit 110, a sound collection unit 120, an audio output unit 130, a display unit 140, a second communication unit 150, a timing adjustment unit 160, a storage unit 170, and a control unit 180. and

The first communication unit 110 communicates with a call destination terminal of a call destination using the first network, and exchanges voice signals. The first network is, for example, a telephone network, the Internet, a local network, or the like.

The sound collector 120 collects surrounding sounds and converts them into first sound data. The sound collector 120 is a microphone or the like. The audio output unit 130 converts audio data into audio and outputs the audio. The audio output unit 130 is, for example, a speaker, earphones, headphones, or the like.

The display unit 140 is a display device that displays information input by the user, operation status of the communication terminal 100, operation and operation result of the communication terminal 100, and the like. The display unit 140 may operate as an input device 141 through which a user's instruction or the like is input. FIG. 2 shows an example in which a display unit 140 and an input device 141 are integrated, such as a touch panel. Alternatively, display 140 and input device 141 may be separate devices.

The second communication unit 150 wirelessly communicates with the second user terminal 20 using a second network different from the first network. The second user terminal 20 is a terminal operated by the second user 12 making a call within a predetermined distance from the communication terminal 100 . The second network is, for example, a network using Bluetooth (registered trademark), Wi-Fi (registered trademark), or the like. The second communication unit 150 receives second audio data based on audio collected by the second user terminal 20 .

Here, the predetermined distance range is a range within which the sound of the second user 12 is collected by the sound collector 120 . In other words, when the communication terminal 100 and the second user terminal 20 come close to each other within a predetermined distance range, the first voice data converted by the sound collecting unit 120 contains the first mixed data based on the voice of the second user 12. will be mixed. Also, in this case, the second mixed data based on the voice of the first user 11 is mixed in the second voice data received by the second communication unit 150 .

The timing adjustment unit 160 adjusts the timing of the time axis of the first audio data and the timing of the time axis of the second audio data so as to be close to each other. The timing adjustment section 160 is provided, for example, between the second communication section 150 and the control section 180 . Also, the timing adjustment section 160 may be further provided between the sound collection section 120 and the control section 180 . The timing adjustment unit 160 has, for example, a delay element, a delay circuit, a flip-flop, a FIFO, etc., and adjusts and synchronizes the timing of the first audio data and/or the timing of the second audio data.

It is desirable that the first audio data and the second audio data are provided with time stamps that serve as timing references on the time axis. In this case, the timing adjustment unit 160 determines the timing adjustment amount by comparing the time stamp information added to the first audio data and the time stamp information added to the second audio data.

Alternatively, the amount of adjustment to be adjusted by the timing adjuster 160 is obtained by measuring in advance the timing at which the first audio data is transmitted to the controller 180 and the timing at which the second audio data is transmitted to the controller 180, and determines the transmission timing. It may be determined in advance from the difference. The timing adjustment unit 160 supplies the first audio data and the second audio data whose timings have been adjusted to the control unit 180 .

The storage unit 170 stores information such as setting values set in the communication terminal 100 . The storage unit 170 may also store intermediate data generated (or used) by the communication terminal 100 in the process of operation, calculation results, thresholds, reference values, parameters, and the like. Further, the storage unit 170 may supply the stored data to the request source in response to a request from each unit within the communication terminal 100 .

The storage unit 170 is a storage medium including ROM (Read Only Memory), RAM (Random Access Memory), and the like. The storage unit 170 may also include a large-capacity storage device such as a HDD (Hard Disk Drive) and/or an SSD (Solid State Drive). For example, when a processor such as a CPU (Central Processing Unit) functions as part of the communication terminal 100, the storage unit 170 may store information such as an OS (Operating System) that causes the CPU to function, and programs. Further, the storage unit 170 may store various information including a database that is referred to during program execution.

Control unit 180 controls each unit in communication terminal 100 . For example, the control unit 180 processes voice data and causes the first communication unit 110 to transmit the processed voice data to the third user terminal 30 . Also, the control unit 180 causes the audio data received by the first communication unit 110 from the third user terminal 30 to be output from the audio output unit 130 as audio.

The control unit 180 is a processor such as a CPU, for example. The control unit 180 has an acquisition unit 181 , an audio processing unit 183 and a display control unit 182 . In other words, the CPU functions as a control unit 180 having an acquisition unit 181 , a display control unit 182 and an audio processing unit 183 by executing programs stored in the storage unit 170 .

Acquisition unit 181 acquires information based on the operation of first user 11 . The acquisition unit 181 acquires, for example, information input to the input device 141 by the user. The control unit 180 controls each unit according to the information acquired by the acquisition unit 181 . The display control unit 182 controls the display unit 140 to display characters, images, and the like on the display unit. Also, the acquisition unit 181 acquires information received by the first communication unit 110 and the second communication unit 150 . Furthermore, the acquisition unit 181 may acquire the first audio data converted by the sound collection unit 120 .

The voice processing unit 183 reduces the first mixed data based on the voice of the second user 12 mixed in the first voice data based on the first voice data and the second voice data. The audio processing unit 183 reduces the first mixed data from the first audio data using the first audio data whose timing has been adjusted by the timing adjustment unit 160 . The audio processor 183 has a separation generator 184 and a selector 185 .

The separation generator 184 performs independent component analysis on the first audio data and the second audio data to generate two separated audio data. The two separated audio data include first separated audio data and second separated audio data. The first separated audio data is audio data obtained by reducing the first mixed data from the first audio data, and the second separated audio data is audio data obtained by reducing the second mixed data from the second audio data.

The independent component analysis performed by the separation generating unit 184 is based on the ratio between the volume of the voice of the first user 11 indicated by the first voice data and the volume of the voice of the second user 12 and the volume of the voice of the first user 11 indicated by the second voice data. is a method of generating the first separated audio data and the second separated audio data based on the fact that the volume ratio of the voice of the second user 12 differs from the volume of the voice of the second user 12 .

For example, in the example of FIG. 1, the first voice data is the voice of the first user 11 "Yoroshiku onegaishimasu" mixed with the voice of the second user 12 "It's nice weather today." data shown. Since the first user 11 is closer to the sound collecting unit 120 than the second user 12, the volume of "Yoroshiku onegaishimasu" is louder than the volume of "It's nice weather today."

On the other hand, the second voice data is data representing a voice in which the voice of the first user 11 "Yoroshiku onegaishimasu" is mixed with the voice of the second user 12 "It's nice weather today". be. Since the second user 12 is closer to the second user terminal 20 than the first user 11 is, the volume of "It's nice weather today, isn't it?"

In independent component analysis, using such first speech data and second speech data, the first separated speech data obtained by reducing the noise of "It's nice weather today" from the first speech data, and the second speech Second separated voice data with reduced noise of "Yoroshiku onegaishimasu" is generated from the data. Since such independent component analysis is a known technique, detailed description is omitted here.

The separation generation unit 184 outputs two pieces of audio data generated by such independent component analysis. Note that when using the independent component analysis, it is not possible to determine which is the first separated sound data and which is the second separated sound data at the time when the two separated sound data are generated. Since the communication terminal 100 transmits voice data based on the voice of the first user 11 to the third user terminal 30 of the third user 13, the data to be transmitted is the first separated voice data.

Here, in both the first separated audio data and the first audio data, the volume of the audio of the first user 11 is higher than the volume of the audio of the second user 12, and the audio data based on the audio of the first user 11 dominates. data. On the other hand, in the second separated voice data and the second voice data, the voice data based on the voice of the second user 12 is dominant data. Therefore, the correlation coefficient between the first separated audio data and the first audio data tends to be larger than the correlation coefficient between the second separated audio data and the first audio data.

Therefore, the selection unit 185 selects the correlation coefficient in the time series between one of the two separated audio data generated by the separation generation unit 184 and the first audio data, and the correlation coefficient of the other of the two separated audio data A correlation coefficient in time series between the separated audio data and the first audio data is calculated and compared. Then, the selection unit 185 determines that the audio data having a larger correlation coefficient with the first audio data is the first separated audio data, and selects it as the communication data to be transmitted to the third user terminal 30 .

Alternatively, or in addition to this, the selection unit 185 selects the correlation coefficient in the time series between one of the two separated audio data and the second audio data, and the correlation coefficient of the two separated audio data A correlation coefficient in a time series between the other separated audio data and the second audio data may be calculated and compared. For example, when the first user 11 does not speak and the second user 12 speaks, the two different correlation coefficients calculated using the two separated speech data and the first speech data are almost the same. may result in a value that does not exist.

However, in this case, the correlation coefficient between the first separated audio data and the second audio data tends to be smaller than the correlation coefficient between the second separated audio data and the second audio data. Therefore, the selection unit 185 determines that the audio data having a smaller correlation coefficient with the second audio data is the first separated audio data, and selects it as communication data to be transmitted to the third user terminal 30 .

For example, when the difference between the correlation coefficients of the two pieces of separated audio data and the first audio data is less than a predetermined value, the selection unit 185 compares the correlation coefficients of the two pieces of separated audio data and the second audio data. Results may be used. This allows the selection unit 185 to more accurately select the first separated audio data from the two separated audio data.

Although the method of using independent component analysis as the speech separation processing has been described so far, another method may be used to perform the speech separation processing. An example of another method will be described below. The audio processing unit 183 inverts the phase of the first audio data or the second audio data, multiplies the first audio data and/or the second audio data by a predetermined coefficient, and converts the first audio data and the second audio data. You may calculate the sum of The absolute value of the voice data based on the voice of the second user 12 included in the first voice data tends to be smaller than the absolute value of the voice data based on the voice of the first user 11 . Also, the absolute value of the voice data based on the voice of the second user 12 included in the second voice data tends to be larger than the absolute value of the voice data based on the voice of the first user 11 .

For example, by multiplying the second voice data by a predetermined coefficient less than 1, the absolute value of the voice data based on the voice of the second user 12 included in the first voice data and the absolute value of the voice data included in the second voice data The absolute value of the voice data based on the voice of the second user 12 who is present can be approximately the same. In this case, the absolute value of the voice data based on the voice of the first user 11 included in the second voice data is the absolute value of the voice data based on the voice of the first user 11 included in the first voice data. By comparison, it becomes almost negligible.

Therefore, the phase of the second audio data multiplied by a coefficient of less than 1 is inverted and added to the first audio data. Thereby, the voice data based on the voice of the second user 12 included in the first voice data can be reduced. Accordingly, the audio processing unit 183 may use the audio data obtained by inverting the phase of the second audio data and adding it to the first audio data as the communication data to be transmitted to the third user terminal 30 .

As described above, the first communication unit 110 of the communication terminal 100 converts the communication data (first separated audio data) obtained by reducing the first mixed data from the first audio data by the audio processing unit 183 to the destination terminal of the call destination (first 3 to the user terminal 30). An operation flow of the communication system S including such a communication terminal 100 will be described below.

<First example of operation flow of communication system S>
FIG. 3 shows a first example of the operation flow of the communication system S according to this embodiment. The operation flow of the first example shows an example in which the first user 11 and the third user 13 perform voice communication, and the second user 12 and the fourth user 14 perform voice communication, as shown in FIG. The operational flow of the first example shows an example in which the first user 11 operates the communication terminal 100 to start a call with the third user 13 . It is also assumed that the second user terminal 20 is within a predetermined distance from the communication terminal 100, and that the call of the second user 12 has already started.

Note that the first network for voice communication of each user (for example, communication by the first communication unit 110) is a communication network using the Internet, and the first network for communication between the communication terminal 100 and the second user terminal 20 is used. 2 network is a communication network using BLE (Bluetooth Low Energy).

First, an instruction for starting a call from the first user 11 to the third user 13 is input to the input device 141 of the communication terminal 100 (S61). For example, the first user 11 causes the display unit 140 to display a phonebook list, and taps the position where the contact information of the third user 13 is displayed on the display unit 140 . Thereby, the acquiring unit 181 acquires an instruction to start a call to the third user terminal 30 .

Next, the control unit 180 causes the second communication unit 150 to transmit a notification signal (S62). The notification signal is a beacon signal, for example, a BLE advertisement signal. Thus, the second communication unit 150 starts transmitting the notification signal when the first communication unit 110 starts communication. The second communication unit 150 periodically transmits the notification signal until the call ends.

Since the second user terminal 20 has already started communication for the call, it receives the notification signal transmitted from the second communication unit 150 . Also, the second user terminal 20 periodically transmits a notification signal, similarly to the second communication unit 150 of the communication terminal 100 (S63). Thereby, the second communication unit 150 of the communication terminal 100 receives the notification signal transmitted from the second user terminal 20 .

Next, the second communication unit 150 transmits a request signal requesting wireless connection by BLE to the second user terminal 20 in response to receiving the notification signal of the second user terminal 20 (S64). Thereby, the second user terminal 20 receives the request signal transmitted from the second communication unit 150 of the communication terminal 100 . Similarly, the second user terminal 20 transmits a request signal requesting wireless connection to the communication terminal 100 (S65). Thereby, the second communication unit 150 of the communication terminal 100 receives the request signal transmitted from the second user terminal 20 .

Next, the display control unit 182 causes the display unit 140 to display that the second user terminal 20 exists when the reception strength of the notification signal received by the second communication unit 150 exceeds the threshold. For example, when the second user terminal 20 is located within a predetermined distance from the communication terminal 100, the threshold is approximately the same as the reception strength of the notification signal received by the communication terminal 100 from the second user terminal 20. value.

Here, the display control unit 182 causes the display unit 140 to further display a message asking whether to connect to the second user terminal 20 (S66). Answer information indicating whether to connect to the second user terminal 20 based on the operation from the first user 11 is input to the input device 141, and the obtaining unit 181 receives the answer information input by the first user 11. to get

The control unit 180 establishes communication connection between the second communication unit 150 and the second user terminal 20 in response to the acquisition unit 181 acquiring response information indicating connection with the second user terminal 20 (S66: Yes ). Thereby, the second communication unit 150 returns a response signal to the request signal of the second user terminal 20 so as to wirelessly connect with the second user terminal 20 (S67).

Similarly, the second user terminal 20 may ask the second user 12 whether or not to connect to the communication terminal 100 . The operation flow of FIG. 3 shows an example in which the second user 12 inputs response information to the second user terminal 20 indicating connection with the communication terminal 100 . Thereby, the second user terminal 20 returns a response signal to the request signal of the communication terminal 100 so as to wirelessly connect with the communication terminal 100 (S68).

This establishes a BLE connection between the communication terminal 100 and the second user terminal 20 (S69). That is, pairing with the second user terminal 20 is set (the second user terminal 20 is registered as a communication partner and encrypted communication is enabled), and a wireless connection is established by BLE. Note that if the first user 11 inputs answer information indicating that the connection is not to be made (S66: No), the communication system S omits the operations from S67 to S69 and proceeds to the next step.

Next, the control unit 180 connects the first communication unit 110 and the third user terminal 30 using the first network. The first communication unit 110, for example, transmits a request signal requesting connection to the third user terminal 30 (S70). The operation flow of FIG. 3 shows an example in which the third user responds to a request for connection with the communication terminal 100 . This establishes a voice communication connection between the communication terminal 100 and the third user terminal 30 (S71).

Next, the sound collector 120 collects the voice of the first user 11 and converts it into first voice data (S72). Also, the second communication unit 150 receives the second audio data based on the sound collected by the second user terminal 20 (S73). Thereby, the acquiring unit 181 acquires the first audio data and the second audio data.

Next, the voice processing unit 183 reduces the first mixed data based on the voice of the second user 12 mixed in the first voice data based on the first voice data and the received second voice data (S74 ). The separation generator 184 performs independent component analysis on the first audio data and the second audio data to generate two separated audio data. Then, the selection unit 185 calculates the correlation coefficients between the two separated audio data and the first audio data, and selects the audio data with the larger correlation coefficient as the communication data to be transmitted to the third user terminal 30. do.

Next, the control unit 180 transmits the communication data selected by the selection unit 185 to the third user terminal 30 (S75). If the connection with the second user terminal is rejected by the user in S66 (S66: No), the processes of S73 and S74 are omitted, and the first voice data converted in S72 is transferred to the third user terminal. 30. Also, the first communication unit 110 receives the voice data of the third user 13 transmitted from the third user terminal 30 (S76). The audio output unit 130 converts the received audio data into audio and outputs the audio. Communication terminal 100 repeats the operations from S72 to S76.

According to the communication system S according to the present embodiment described above, the first user 11 hears the voice of the second user 12 mixed in the voice of the first user 11 due to the close proximity of the communication terminal 100 and the second user terminal 20. It is possible to make a call with the third user while reducing the This makes it easier for the third user 13 to hear the voice of the first user 11 .

The second user terminal 20 operated by the second user 12 may also have the same configuration as the communication terminal 100 and operate in the same manner. For example, in S<b>73 , the second communication unit 150 transmits the first voice data to the second user terminal 20 . In this case, since the second mixed data based on the voice of the first user 11 can be reduced in the second user terminal 20 as well, the voice of the second user 12 can be easily heard by the fourth user 14 .

Also, the third user terminal 30 operated by the third user 13 may have the same configuration as the communication terminal 100 and operate in the same manner. As a result, even if another terminal comes close to the third user terminal 30, it is possible to reduce the difficulty that the first user 11 can hear the third user 13's voice.

In the operation flow of the communication system S according to the present embodiment described above, an example in which a call between the second user 12 and the fourth user 14 has already started has been described, but the present invention is not limited to this. For example, the second user 12 may initiate a call while the first user 11 is on the phone. In this case, the second communication unit 150 receives the notification signal transmitted from the second user terminal 20 during the communication of the first communication unit 110 instead of when the first communication unit 110 starts communication. wireless connection with the second user terminal 20 based on. The operation of the communication system S in such a case will now be described.

<Second Example of Operation Flow of Communication System S>
FIG. 4 shows a second example of the operation flow of the communication system S according to this embodiment. In the operation flow of the communication system S of the second example, the same reference numerals are given to the operations that are substantially the same as those of the operation flow of the communication system S of the first example shown in FIG. 3, and overlapping explanations are omitted.

In FIG. 4, it is assumed that a call between the first user 11 and the third user 13 has already started (S60). Since the call by the second user 12 has not started, the communication terminal 100 transmits the first voice data collected and output by the sound collector 120 from the first communication section 110 to the third user terminal 30 . Also, the communication terminal 100 receives the audio signal transmitted from the third user terminal 30 and outputs it as audio from the audio output unit 130 .

The second communication unit 150 of the communication terminal 100 periodically transmits the notification signal even during such a call (S62). When starting a call, the second user terminal 20 starts transmitting a notification signal (S63). Next, the second communication unit 150 transmits a request signal requesting wireless connection to the second user terminal 20 in response to receiving the notification signal of the second user terminal 20 (S64). Similarly, the second user terminal 20 transmits a request signal requesting wireless connection to the communication terminal 100 (S65).

When the reception strength of the notification signal received by the second communication unit 150 exceeds the threshold, the display control unit 182 inquires about the presence of the second user terminal 20 and whether or not to connect to the second user terminal 20. The message is displayed on the display unit 140 (S66). In response to the acquisition unit 181 acquiring response information indicating connection with the second user terminal 20 (S66: Yes), the control unit 180 returns a response signal from the second communication unit 150 (S67).

Similarly, the second user terminal 20 may ask the second user 12 whether or not to connect to the communication terminal 100 . In the operation flow of FIG. 4, the second user 12 shows an example of inputting reply information to the second user terminal 20 indicating connection with the communication terminal 100 . Thereby, the second user terminal 20 returns a response signal to the request signal of the communication terminal 100 so as to wirelessly connect with the communication terminal 100 (S68). This establishes a BLE connection between the communication terminal 100 and the second user terminal 20 (S69).

Next, the communication terminal 100 reduces the first mixed data based on the voice of the second user 12 mixed in the first voice data to be transmitted to the third user terminal 30 during the call. The operation of the communication terminal 100 is the same as that of S72 to S76 in FIG. 3, so the explanation is omitted here.

As described above, in the communication system S according to the present embodiment, during a call by the first user 11, the second user 12 starts a call using the second user terminal 20 that is close to the communication terminal 100, Even if the voice of the second user 12 is mixed with the voice of the user 11, the call between the first user 11 and the third user 13 can be continued while reducing the mixed voice. Thereby, the third user 13 can reduce difficulty in hearing the voice of the first user 11 .

In the operation flow of the communication system S according to the present embodiment, an example in which the second user terminal 20 is located within a predetermined distance range of the communication terminal 100 has been described, but the present invention is not limited to this. . For example, multiple terminals may be located within a predetermined distance range of communication terminal 100 . The operation of the communication system S in such a case will now be described.

<Third example of operation flow of communication system S>
FIG. 5 shows a third example of the operation flow of the communication system S according to this embodiment. In the operation flow of the communication system S of the third example, the same reference numerals are given to the operations that are substantially the same as those of the operation flow of the communication system S of the first example shown in FIG. 3, and overlapping explanations are omitted. In the operation flow of the third example, an example in which the second user terminal 20 and the fifth user terminal are located within a predetermined distance range of the communication terminal 100 is shown. It is also assumed that the second user terminal 20 and the fifth user terminal have already started talking.

First, the input device 141 of the communication terminal 100 receives an instruction from the first user 11 to start a call with the third user terminal 30 of the third user 13 (S61). In response to the acquisition unit 181 acquiring such an instruction, the control unit 180 causes the second communication unit 150 to transmit a notification signal (S62). Thereby, the second user terminal 20 and the fifth user terminal receive the notification signal. Also, the second user terminal 20 and the fifth user terminal respectively transmit notification signals (S63, S63'). Thereby, the second communication unit 150 of the communication terminal 100 receives the notification signals transmitted from the second user terminal 20 and the fifth user terminal.

Next, the second communication unit 150 transmits a request signal requesting wireless connection to the second user terminal 20 in response to receiving the notification signal of the second user terminal 20 (S64). In addition, the second communication unit 150 transmits a request signal requesting wireless connection to the fifth user terminal in response to receiving the notification signal of the fifth user terminal (S64'). The second user terminal 20 transmits a request signal requesting wireless connection to the communication terminal 100 (S65). Similarly, the fifth user terminal transmits a request signal requesting wireless connection to the communication terminal 100 (S65').

Next, when the reception strength of the notification signal received by the second communication unit 150 from the second user terminal 20 exceeds the threshold, the display control unit 182 determines that the second user terminal 20 exists and 20 is displayed on the display unit 140 (S66). Similarly, when the reception strength of the notification signal received by the second communication unit 150 from the fifth user terminal exceeds the threshold, the display control unit 182 determines that the fifth user terminal exists and that the fifth user terminal is connected. The display unit 140 displays a message inquiring whether or not to

In response to the acquisition unit 181 acquiring response information indicating that the acquisition unit 181 connects to the second user terminal 20 (S66: Yes), the control unit 180 transmits a response signal from the second communication unit 150 to the second user terminal 20. Reply (S67). Similarly, the control unit 180 returns a response signal from the second communication unit 150 to the fifth user terminal in response to the acquisition unit 181 acquiring response information indicating connection with the fifth user terminal (S67 '). Note that the example of FIG. 5 simply shows an example in which the control unit 180 acquires response information indicating connection with the second user terminal 20 and the fifth user terminal. As described above, it is desirable that the control unit 180 separately performs the operation of whether to connect to the second user terminal 20 and the operation of whether to connect to the fifth user terminal.

The operation flow of FIG. 5 shows an example in which the second user 12 and the fifth user input response information indicating connection with the communication terminal 100 to the terminals they operate. Thereby, the second user terminal 20 returns a response signal to the request signal of the communication terminal 100 so as to wirelessly connect with the communication terminal 100 (S68). Also, the fifth user terminal returns a response signal to the request signal of the communication terminal 100 so as to wirelessly connect with the communication terminal 100 (S68'). This establishes a BLE connection between the communication terminal 100 and the second user terminal 20 and a BLE connection between the communication terminal 100 and the fifth user terminal (S69).

Next, the control unit 180 connects the first communication unit 110 and the third user terminal 30 using the first network (S70, S71). Next, the communication terminal 100 reduces the first mixed data based on the voice of the second user 12 and the mixed data based on the voice of the fifth user mixed in the first voice data (S72, S73, S73′, S74 ). It should be noted that if the connection with another user terminal is rejected in S66 (S66: No), the processes of S73, S73' and S74 are omitted.

Note that the separation generating unit 184 performs independent component analysis on the first audio data, the second audio data, and the audio data received from the fifth user terminal to generate three separate audio data. Then, the selection unit 185 calculates the correlation coefficients between the three separated audio data and the first audio data, and selects the audio data with the larger correlation coefficient as the communication data to be transmitted to the third user terminal 30. do. Then, the first communication unit 110 transmits the communication data selected by the selection unit 185 to the third user terminal 30 (S75), and receives the voice data of the third user 13 transmitted from the third user terminal 30 ( S76).

As described above, in the communication system S according to the present embodiment, even if a plurality of terminals are close to the communication terminal 100 and the voices of a plurality of users are mixed with the voice of the first user 11, the mixed voice is reduced. It is possible to have the first user 11 and the third user 13 talk to each other. Thereby, the third user 13 can reduce difficulty in hearing the voice of the first user 11 .

Further, in the operational flow of FIGS. 3 to 5, when a request signal for requesting connection is received after receiving the notification signal, a message asking whether or not to connect to the user terminal that has sent the request signal is displayed. (S66) Although the explanation has been made assuming that this is the case, the display timing of the message is not limited to this. For example, when the reception strength of the received annunciation signal exceeds the threshold, the above message is displayed, and when response information indicating connection based on the user's operation is acquired, the user terminal that transmitted the annunciation signal Alternatively, a request signal requesting connection may be transmitted.

3 to 5, both the communication terminal 100 and the second user terminal 20 (or the fifth user terminal) transmit a notification signal, a request signal for requesting connection, and a response signal to the request signal. Although an example in which a BLE connection is established by doing so has been described, a BLE connection may be established by only one of them transmitting each of the above signals. For example, when the communication terminal 100 transmits a notification signal, the second user terminal 20 (or the fifth user terminal) receiving the notification signal transmits a request signal, and the communication terminal 100 transmits a response signal, the BLE connection is established. may be established.

<Fourth example of operation flow of communication system S>
FIG. 6 shows a fourth example of the operation flow of the communication system S according to this embodiment. In the operation flow of the fourth example, the first user 11 and the third user 13 make a call according to the operation flow of the first example shown in FIG. 3, the operation flow of the second example shown in FIG. Here is an example of the case where

First, an instruction for ending the call from the first user 11 to the third user 13 is input to the input device 141 of the communication terminal 100 (S81). In response to the acquisition unit 181 acquiring such an instruction, the control unit 180 terminates communication between the first communication unit 110 and the third user terminal 30 (S82).

Then, the control unit 180 cancels the BLE connection between the second communication unit 150 and the second user terminal 20 in response to the first communication unit 110 ending communication. Here, it is desirable that the second communication unit 150 cancels the pairing setting with the second user terminal 20 in response to the first communication unit 110 ending the communication.

For example, the second communication unit 150 transmits a request signal requesting disconnection of the BLE connection to the second user terminal 20 (S83). Then, when the second communication unit 150 receives a response signal to the request signal from the second user terminal 20 (S84), the BLE connection is released (S85). The second communication unit 150 also cancels the pairing setting with the second user terminal 20 when canceling the BLE connection. When communication terminal 100 cancels the pairing and turns on the BLE function for a purpose other than a call in communication terminal 100 (for example, when turning on BLE to connect to earphones), the second Communication with the two-user terminal 20 can be prevented from being automatically connected.

The second communication unit 150 ends transmission of the notification signal when the first communication unit 110 ends communication (S86). As a result, the call of communication terminal 100 is terminated. In the operation flow of the fourth example, an example in which the call of the communication terminal 100 is terminated and the connection between the second communication unit 150 and the second user terminal 20 is terminated has been described. The disconnection operation is not limited to this. For example, when the second user terminal 20 moves out of a predetermined distance range from the communication terminal 100 , the communication terminal 100 may disconnect the second communication unit 150 and the second user terminal 20 .

For example, when the second communication unit 150 wirelessly communicates with the second user terminal 20, in response to the reception strength of the notification signal received from the second user terminal 20 falling below the threshold, The wireless connection with the terminal 20 is released. In this case as well, it is desirable that the second communication unit 150 cancel the pairing setting with the second user terminal 20 .

In the communication system S according to the present embodiment, an example has been described in which the first user 11 and the third user 13 perform voice communication, and the second user 12 and the fourth user 14 perform voice communication. is not limited to For example, it can also be applied when the first user 11 and the second user 12 talk with the same person (for example, the third user 13), such as a telephone conference, a TV conference, a web conference, or a multiplayer game. be.

In this case as well, the communication terminal 100 transmits to the third user terminal 30 communication data in which the first mixed data mixed in the first voice data of the first user 11 is reduced. The second user terminal 20 also transmits to the third user terminal 30 communication data in which the second mixed data mixed in the second voice data of the second user 12 is reduced. Thereby, the third user terminal 30 can hear the voice of the first user 11 and the voice of the second user 12 more clearly.

The communication terminal 100 according to the present embodiment described above wirelessly connects the second user terminal 20 to the first user 11 when detecting that the second user terminal 20 is within a predetermined distance range from the communication terminal 100. Although an example of inquiring whether or not has been described, the present invention is not limited to this. For example, when the communication terminal 100 detects that the second user terminal 20 is within a predetermined distance from the communication terminal 100, the communication terminal 100 wirelessly connects to the second user terminal 20 without inquiring the first user 11. may be set to

Setting information indicating such settings is preferably stored in communication terminal 100 in advance. For example, the storage unit 170 stores setting information indicating whether or not the second communication unit 150 automatically wirelessly connects with other terminals including the second user terminal 20 . Then, when the storage unit 170 stores setting information indicating automatic wireless connection, the second communication unit 150, in response to the reception strength of the received annunciation signal exceeding the threshold, A wireless connection is established with the terminal 20 . For example, when the second user terminal 20 is known to be a safe terminal, such as when a web conference is held in a company, the communication terminal 100 can quickly connect to the second user terminal 20 by setting automatic wireless connection. A wireless connection with the terminal 20 is possible.

Although the communication terminal 100 according to the present embodiment described above transmits a notification signal from the second communication unit 150 when starting a call, the communication terminal 100 is not limited to this. For example, in the communication terminal 100, the second communication unit 150 outputs the notification signal when an application for the first communication unit 110 to communicate is executed, or when the operation of the application returns from the background. You can start sending. The application is, for example, a call application or a web conference application. Accordingly, communication terminal 100 can quickly detect whether or not another terminal exists within a predetermined distance range.

The present invention makes it possible to contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, "Industry and technological innovation."

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist thereof. be. For example, all or part of the device can be functionally or physically distributed and integrated in arbitrary units. In addition, new embodiments resulting from arbitrary combinations of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment caused by the combination has the effect of the original embodiment.

11 First user 12 Second user 13 Third user 14 Fourth user 20 Second user terminal 30 Third user terminal 40 Fourth user terminal 100 Communication terminal 110 First communication unit 120 Sound collection unit 130 Audio output unit 140 Display unit 141 Input device 150 Second communication unit 160 Timing adjustment unit 170 Storage unit 180 Control unit 181 Acquisition unit 182 Display control unit 183 Audio processing unit 184 Separation generation unit 185 Selection unit

Claims (14)

A communication terminal operated by a first user,
a first communication unit that communicates with a call destination terminal of a call destination using a first network;
a sound collecting unit that collects ambient sound and converts it into first sound data;
Wireless communication is performed with a second user terminal of a second user who is talking within a predetermined distance from the communication terminal using a second network different from the first network, and sound is collected by the second user terminal. a second communication unit configured to receive second audio data based on the audio generated;
Based on the first audio data converted by the sound collecting unit and the second audio data received by the second communication unit, a first sound based on the second user's voice mixed in the first audio data an audio processor that reduces mixed data; and
The first communication unit transmits communication data obtained by reducing the first mixed data from the first audio data by the audio processing unit to the destination terminal of the destination.
communication terminal. The audio processing unit is
A ratio of the volume of the voice of the first user indicated by the first voice data and the volume of the voice of the second user, and the volume of the voice of the first user indicated by the second voice data and the volume of the voice of the second user a separation generating unit configured to generate two separated audio data by performing independent component analysis on the first audio data and the second audio data based on differences in the volume ratio of the audio;
A correlation coefficient in a time series between one of the two separated audio data and the first audio data, and a correlation coefficient between the other separated audio data of the two separated audio data and the first audio data. a selection unit that compares correlation coefficients in a time series and selects voice data with a larger correlation coefficient as the communication data,
The two separated audio data are first separated audio data obtained by reducing the first mixed data from the first audio data, and second mixed data based on the first user's voice mixed in the second audio data. from the second audio data, and
A communication terminal according to claim 1. further comprising a timing adjusting unit that adjusts the timing of the time axis of the first audio data and the timing of the time axis of the second audio data so as to be close to each other;
The audio processing unit reduces the first mixed data from the first audio data using the first audio data adjusted by the timing adjustment unit.
The communication terminal according to claim 1 or 2. 4. The timing adjustment unit determines a timing adjustment amount by comparing time stamp information added to the first audio data and time stamp information added to the second audio data. The communication terminal described in . When the first communication unit starts communication or during communication of the first communication unit, the second communication unit receives a notification signal transmitted from the second user terminal, the 5. A communication terminal according to any one of claims 1 to 4, in wireless connection with a second user terminal. The notification signal is a BLE advertising signal,
The second communication unit is
making a BLE connection with the second user terminal based on receiving the advertising signal;
releasing the BLE connection with the second user terminal in response to the first communication unit having finished communication;
The communication terminal according to claim 5. The second communication unit cancels the pairing setting with the second user terminal in response to the communication being terminated by the first communication unit.
A communication terminal according to claim 6. a display control unit that causes a display unit to display that the second user terminal exists when the reception strength of the notification signal received by the second communication unit exceeds a threshold;
an acquisition unit that acquires information as to whether or not to connect to the second user terminal based on the operation by the first user,
8. The second communication unit wirelessly connects to the second user terminal in response to the acquisition unit acquiring information indicating connection to the second user terminal. The communication terminal described in the item. Further comprising a storage unit for storing setting information indicating that the second communication unit automatically establishes a wireless connection with another terminal including the second user terminal,
When the storage unit stores the setting information indicating automatic wireless connection, the second communication unit performs the second 9. A communication terminal according to claim 8, for wireless connection with a user terminal. The second communication unit, when wirelessly communicating with the second user terminal, establishes a wireless connection with the second user terminal in response to a reception strength of the received annunciation signal becoming equal to or less than a threshold. 10. A communication terminal according to any one of claims 5 to 9, which is released. The communication terminal according to any one of claims 1 to 10, wherein said second communication unit starts transmission of a notification signal when said first communication unit starts communication. The second communication unit transmits a notification signal when an application for communication by the first communication unit is executed in the communication terminal, or when the operation of the application returns from the background. 12. A communication terminal according to any one of claims 1 to 11, which initiates. The second communication unit ends transmission of the notification signal in response to the first communication unit ending communication.
A communication terminal according to claim 11 or 12. A communication method in which a communication terminal operated by a first user communicates with a called terminal of a called party using a first network,
a step of collecting ambient sound and converting it into first sound data;
Wireless communication is performed with a second user terminal of a second user who is talking within a predetermined distance from the communication terminal using a second network different from the first network, and sound is collected by the second user terminal. receiving second audio data based on the audio generated;
reducing first mixed data based on the second user's voice mixed in the first audio data, based on the first audio data and the received second audio data;
and transmitting communication data obtained by reducing the first mixed data from the first audio data to the called terminal of the called party.

Images