JP2018110301A - Telephone calling device and echo suppression program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently and correctly determine whether or not a microphone input sound in a telephone calling device is a double talk state.SOLUTION: A telephone calling device comprises: a received sound processing part; a component extraction part; a transfer characteristic learning part; and a determination part. The received sound processing part removes and outputs components of a predetermined frequency band in a received voice received from an external device to a receiver. The component extraction part extracts the components of the predetermined frequency band in an input voice input from a microphone. The transfer characteristic learning part learns the transfer characteristics of an echo path going from the receiver to the microphone on the basis of the received voice and the input voice. The determination part causes the transfer characteristic learning part to learn the transfer characteristics when the scale of the components of the predetermined frequency band extracted from the input voice is equal to or less than a threshold.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a communication device and an echo suppression program.

  Some communication devices such as mobile terminals suppress an echo component included in a microphone input sound input from a microphone of the own device, and transmit an input sound in which the echo component is suppressed to another communication device. Here, the echo component is a component of the received sound output from the receiver of the own device in the microphone input sound.

  In this type of communication device, the transfer characteristic of the echo path is learned based on the received sound and the microphone input sound during a call, and the echo component contained in the microphone input sound is suppressed based on the transfer characteristic. However, when learning the transfer characteristics, if the microphone input sound is in a double-talk state that includes speech components other than the received sound, it is difficult to correctly suppress the echo component because an incorrect transfer coefficient is learned. It becomes. For this reason, in a call device that suppresses echo components, only when the microphone input sound is not in a double talk state, the transmission characteristics are learned based on the received sound and the microphone input sound to suppress the echo component.

  As one of methods for determining whether or not the microphone input sound is in a double talk state, there is a method of determining by capturing the movement of the user's mouth with a camera (see, for example, Patent Document 1).

  As another method for determining whether or not a microphone input sound is in a double talk state, there is a method for determining based on a volume ratio between an audio output signal and an audio input signal (see, for example, Patent Document 2). reference).

  Further, there is a method for obtaining a control parameter for suppressing the echo component by outputting a signal obtained by removing a specific frequency component from the received signal to the echo path and using a component having the same frequency as the specific frequency detected from the transmission signal. (For example, see Patent Document 3). In this method, the noise power is obtained based on the power of the specific frequency component detected from the transmission signal, the combined power including the noise and the echo component is obtained based on the power of the frequency component including the echo component, and then the noise power and The control parameter is obtained using the total power.

JP 2000-295338 A JP 2007-053512 A JP 2011-130170 A

  When capturing the movement of the user's mouth with the camera, it is impossible to correctly determine whether or not the user is in a double talk state if the user's mouth is not within the imaging range of the camera. In addition, when detecting whether or not a double talk state is detected based on the volume ratio between the audio output signal and the audio input signal, if the level of the user's voice or ambient noise included in the audio input signal is small, the double talk It is not possible to correctly determine whether or not this is the state. Furthermore, when the control parameter is obtained using the noise power obtained based on the power of the specific frequency component detected from the transmission signal and the combined power obtained based on the power of the frequency component including the echo component, the processing load Will increase.

  In one aspect, an object of the present invention is to efficiently and correctly determine whether or not a microphone input sound in a call device is in a double talk state.

  The communication device according to one aspect includes a received sound processing unit, a component extraction unit, a transfer characteristic learning unit, and a determination unit. The received sound processing unit removes a component of a predetermined frequency band from the received voice received from the external device and outputs the result to the receiver. The component extraction unit extracts a component in a predetermined frequency band from the input sound input from the microphone. The transfer characteristic learning unit learns the transfer characteristic of the echo path from the receiver to the microphone based on the received voice and the input voice. The determination unit causes the transfer characteristic learning unit to learn transfer characteristics when the magnitude of a component in a predetermined frequency band extracted from the input speech is equal to or less than a threshold value.

  According to the above-described aspect, it is possible to efficiently and correctly determine whether or not the microphone input sound in the call device is in a double talk state.

It is a figure which shows the functional structure of the telephone apparatus which concerns on one Embodiment. It is a graph explaining mask information. It is a flowchart explaining the process which the telephone apparatus which concerns on one Embodiment performs. It is a flowchart explaining the content of the process which frequency-analyzes the frame of a received sound. It is a flowchart explaining the content of the process which determines the mask frequency removed from an incoming call sound. It is a flowchart (the 1) explaining the content of the process which calculates a 1st mask frequency. It is a flowchart (the 2) explaining the content of the process which calculates a 1st mask frequency. It is a graph explaining the calculation method of the 1st mask frequency. It is a flowchart (the 1) explaining the content of the process which calculates a 2nd mask frequency. It is a flowchart (the 2) explaining the content of the process which calculates a 2nd mask frequency. It is a graph explaining the calculation method of the 2nd mask frequency. It is a flowchart explaining the content of the process which removes the component of a mask frequency from an incoming call sound, and outputs it. It is a flowchart explaining the content of the process which determines whether it is a double talk. It is a figure which shows the hardware constitutions of a mobile terminal. It is a figure which shows the hardware constitutions of a computer.

FIG. 1 is a diagram illustrating a functional configuration of a call device according to an embodiment.
As shown in FIG. 1, the communication device 1 according to the present embodiment includes a transmission / reception unit 110, a received sound analysis unit 120, a mask frequency determination unit 130, a received sound processing unit 140, a component extraction unit 150, a double A talk determination unit 160 and an input sound processing unit 170 are provided. In addition, the communication device 1 includes a mask information storage unit 191, an analysis result holding unit 192, and a transfer characteristic storage unit 193. Further, the communication device 1 includes a receiver 2 and a microphone 3.

  The transmission / reception unit 110 receives an audio signal (received sound) from the other call device 4 and receives an audio signal in which an echo component contained in an audio signal (microphone input sound) acquired by collecting the sound from the microphone 3 is suppressed. It transmits to the other communication device 4. The echo component is a component of the sound output from the receiver 2 included in the microphone input sound. In this specification, the receiver 2 is given as an example in which the received sound (sound signal) received from the other call device 4 is output as a sound wave to the outside of the call device 1, but the call device 1 is replaced with the receiver 2. A device having a function equivalent to that of the receiver 2 such as a speaker may be provided.

  The received sound analysis unit 120 performs frequency analysis on the received sound received from the other communication device 4 and calculates a frequency peak.

  The mask frequency determination unit 130 calculates a frequency (mask frequency) to be removed from the received sound based on the result of frequency analysis for the received sound. Since the mask frequency is masked by other frequency components in the received sound, it is a frequency of a component that does not affect the perception even if it is removed from the received sound. The mask frequency determination unit 130 calculates a mask frequency based on the mask information stored in the mask information storage unit 191 and the frequency analysis result held in the analysis result holding unit 192.

  The received sound processing unit 140 generates a processed received sound from which the mask frequency component included in the received sound has been removed. The reception sound processing unit 140 outputs the generated processed reception sound to the receiver 2 and outputs information indicating the mask frequency to the component extraction unit 150.

  The component extraction unit 150 extracts a mask frequency component included in the microphone input sound acquired from the microphone 3.

  The double talk determination unit 160 determines whether or not the double talk state is based on the frequency component extracted from the microphone input sound. The double talk state is a state in which the voice of the user who speaks toward the microphone 3 of the call device 1 and the noise generated around the call device 1 are included in the microphone input sound.

  The input sound processing unit 170 generates a processed input sound that suppresses the echo component included in the microphone input sound based on the determination result of the double talk determination unit 160 and the transmission characteristic of the echo path from the receiver 2 to the microphone 3. To do. When not in the double talk state, the input sound processing unit 170 learns the transfer characteristic of the echo path, and then removes the echo component included in the microphone input sound based on the learned transfer characteristic. At this time, the input sound processing unit 170 stores the learned transfer characteristic in the transfer characteristic storage unit 193. On the other hand, in the case of the double talk state, the input sound processing unit 170 uses the microphone input sound based on the transfer characteristic stored in the transfer characteristic storage unit 193 (that is, the transfer characteristic learned when not in the double talk state). The echo component contained in is removed. The input sound processing unit 170 includes a learning determination unit 171, a transfer characteristic learning unit 172, and an echo component suppression unit 173. The learning determination unit 171 determines whether to learn transfer characteristics based on the determination result of the double talk determination unit 160. The transfer characteristic learning unit 172 learns the transfer characteristic of the echo path based on the received sound and the microphone input sound. The echo component suppression unit 173 generates a processed input sound in which the echo component included in the microphone input sound is suppressed based on the transfer characteristic. The processed input sound in which the echo component is suppressed is transmitted to the other communication device 4 via the transmission / reception unit 110.

  As described above, the mask frequency determination unit 130 in the communication device 1 according to the present embodiment refers to the mask information and calculates a frequency (mask frequency) masked by other frequency components in the voice included in the received sound. To do. Here, there are two types of other frequency components that mask the mask frequency component, that is, the peak frequency component in the same frame and the past frame component that peaks in the temporal change of the same frequency. For this reason, in the present embodiment, the first mask information and the second mask information are used as the mask information. The first mask information is information indicating the relationship between the distance (frequency difference) from the peak frequency in the frame to be processed and the maximum volume (power) masked for each frequency. The peak frequency is a frequency having a maximum value in the frequency spectrum. The second mask information is information indicating the relationship between the distance (time difference) from the peak time in the time change of the component of the same frequency in a plurality of frames for a predetermined time in the past and the maximum sound volume to be masked. The peak time is a time at which the maximum value is obtained in the time change of the same frequency component in the frequency spectrum for a predetermined time.

FIG. 2 is a graph for explaining the mask information.
FIG. 2A shows a graph for explaining the first mask information among the mask information referred to by the mask frequency determination unit 130. In the graph of FIG. 2A, the horizontal axis is the difference from the peak frequency, and the vertical axis is the power.

  As described above, the first mask information is information indicating the relationship between the distance from the peak frequency (frequency difference from the peak frequency) and the maximum volume (power) to be masked. This first mask information is given by, for example, the function G (f) shown in FIG. The function G (f) is a function that becomes power PW2 (that is, G (0) = PW2) when the difference f from the peak frequency is 0, and decreases as the absolute value of the difference f from the peak frequency increases. . The function G (f) can be set by known statistical processing.

  The value calculated by the function G (f) is the maximum sound volume (power) to be masked as described above. Therefore, when the power of a certain frequency in the power spectrum for the processing target frame is smaller than the maximum volume calculated by the function G (f), the certain frequency component is masked by the peak frequency component. That is, it is difficult for a person to perceive a frequency component whose power is smaller than the maximum volume calculated by the function G (f) in the power spectrum for the processing target frame, and even if it is removed from the processing target frame. The impact on is small.

  In the graph of FIG. 2A, the power P2 of the frequency whose difference f from the peak frequency is F1 is larger than the maximum volume G (F1) calculated by the function G (f). Thus, when the power of a certain frequency in the power spectrum for the processing target frame is equal to or higher than the maximum volume calculated by the function G (f), a person can perceive the frequency component.

  On the other hand, in the graph of FIG. 2A, the power P3 of the frequency whose difference f from the peak frequency is −F1 is smaller than the maximum volume G (−F1) calculated by the function G (f). . Thus, when the power of a certain frequency in the power spectrum for the processing target frame is smaller than the maximum volume calculated by the function G (f), it becomes difficult for a person to perceive the frequency component. In particular, the greater the difference (mask amount) between the power in the power spectrum and the maximum volume calculated by the function G (f), the more difficult it is to perceive the frequency component masked by the peak frequency component.

  In the present embodiment, of the frequencies masked by the peak frequency component, the frequency having the maximum mask amount is set as the first mask frequency. For this reason, it becomes possible to minimize the influence on perception by removing the component of the first mask frequency from the received sound.

  When calculating the first mask frequency based on the first mask information, the mask frequency determination unit 130, for example, based on the peak frequency in the frequency spectrum for the frame to be processed and the component P1 of the peak frequency. , Translate function G (f).

  On the other hand, FIG. 2B shows a graph for explaining the second mask information of the mask information referred to by the mask frequency determining unit 130. In the graph of FIG. 2B, the horizontal axis represents the time difference from the continuous peak frame, and the vertical axis represents the power.

  As described above, the second mask information includes the distance (time difference from the continuous peak) from the peak (continuous peak) in the time change of the components of the same frequency in a plurality of frames, and the maximum volume (power) to be masked. ). This second mask information is given by, for example, the function H (t) shown in FIG. The function H (t) is a function that becomes power PW2 (that is, H (0) = PW2) when the time difference t from the continuous peak is 0, and decreases as the time difference t from the continuous peak increases. The function H (t) can be set by known statistical processing.

  The value calculated by the function H (t) is the maximum sound volume (power) to be masked as described above. Therefore, when the power of a certain frequency in the power spectrum for the processing target frame is smaller than the maximum volume calculated by the function H (t), the component of the certain frequency is masked by the same frequency component of the past frame. That is, it is difficult for a person to perceive a frequency component whose power is smaller than the maximum volume calculated by the function H (t) in the power spectrum of the processing target frame, and it is perceived even if it is removed from the processing target frame. The impact on is small.

  In the graph of FIG. 2B, the power P3 of the same frequency component in the frame whose time difference t from the frame including the successive peak is T2 is the maximum volume H (T2) calculated by the function H (t). Bigger than. In this way, when the power of a certain frequency in the processing target frame is larger than the maximum volume calculated by the function H (t), a person can perceive the frequency component.

  On the other hand, in the graph of FIG. 2B, the power P2 of the same frequency component in the frame whose time difference t from the frame including the successive peak is T1 is the maximum volume calculated by the function H (t). It is smaller than H (T1). As described above, when the power of a certain frequency in the power spectrum for the processing target frame is smaller than the maximum volume calculated by the function H (t), it becomes difficult for a person to perceive the frequency component. In particular, the greater the difference (mask amount) between the power in the power spectrum and the maximum volume calculated by the function H (t), the more difficult it is to perceive the frequency component masked by the same frequency component of the past frame. .

  In the present embodiment, the frequency at which the mask amount becomes the maximum value among the frequencies masked by the same frequency component (continuous peak) of the past frame in the processing target frame is set as the second mask frequency. For this reason, it is possible to minimize the influence on perception by removing the second mask frequency component from the received sound.

  When calculating the second mask frequency based on the second mask information, the mask frequency determination unit 130, for example, the time difference from the frame including the continuous peak to the currently processed frame, Based on the peak power P1, the function H (t) is translated.

  After the call connection with the other call device 4 is established, the call device 1 outputs the received sound received from the other call device 4 from the receiver 2 and the microphone input sound input from the microphone 3. The process which suppresses the echo component contained in and transmits to the other communication apparatus 4 is performed. The call device 1 according to the present embodiment performs the process shown in FIG. 3 after establishing a call connection with another call device 4.

FIG. 3 is a flowchart for describing processing performed by the telephone device according to the embodiment.
The communication device 1 first selects a frame to be processed from the received sound (step S1), and performs frequency analysis on the frame to be processed (step S2). The received sound analysis unit 120 performs the processes of steps S1 and S2. The received sound analysis unit 120 converts the speech waveform in the frame to be processed into a frequency spectrum according to a known analysis method, and calculates a peak frequency value and the number of peaks. In addition, the received sound analysis unit 120 causes the analysis result holding unit 192 to hold the frequency analysis result for the processing target frame. At this time, the received sound analysis unit 120 causes the analysis result holding unit 192 to hold the frequency analysis results for the past predetermined number of frames including the frame that is currently processed. In the following description, the frequency spectrum and the frequency band in the power spectrum calculated based on the frequency spectrum are also simply referred to as “frequency”.

  Next, the call device 1 calculates the mask frequency of the frame based on the result of frequency analysis for the frame to be processed in the received sound (step S3). The process of step S3 is performed by the mask frequency determination unit 130. The mask frequency determination unit 130 in the communication device 1 of the present embodiment is based on the mask information stored in the mask information storage unit 191 and the frequency analysis result for a predetermined number of frames held in the analysis result holding unit 192. Thus, the first mask frequency and the second mask frequency are calculated. The first mask frequency is a frequency having the largest mask amount among the frequencies masked by the frequency component that becomes a peak in the frequency spectrum of the processing target frame. The second mask frequency is a frequency having the largest mask amount among the frequencies masked by the frequency component of the frame that peaks due to the time change of the same frequency component in the frequency spectrum for the predetermined number of frames. The frequency masked by the peak frequency component has a spectrum value (power) higher than the maximum volume (function G (f) or function H (t)) set based on the peak frequency or the distance to the frame. It is a small frequency. The mask amount is a value obtained by subtracting power from the maximum volume to be masked.

  Next, the communication device 1 removes the mask frequency component from the frequency spectrum of the frame to be processed and outputs it (step S4). The received sound processing unit 140 performs the process of step S4. The received sound processing unit 140 sets the spectrum values of the first mask frequency and the second mask frequency in the frequency spectrum to 0, and then inversely converts the frequency spectrum into a speech waveform to generate a processed received sound. The received sound processing unit 140 outputs the processed received sound to the receiver 2 and outputs information indicating the mask frequency to the component extraction unit 150. Thereby, the user of the communication device 1 listens to the received sound 5 from which the first mask frequency and the second mask frequency are removed. However, as described above, the mask frequency component is a component that is masked by other frequency components and is difficult for the user of the communication device 1 to perceive. For this reason, the influence on the perception of the user of the communication device 1 by outputting the reception sound 5 from which the mask frequency component is removed is very small.

  Next, the communication device 1 extracts a mask frequency component from the processing target frame in the microphone input sound (step S5). The component extraction unit 150 performs the process in step S5. The component extraction unit 150 converts a frame to be processed in the microphone input sound input from the microphone 3 from a speech waveform to a frequency spectrum, and extracts a mask frequency component received from the received sound processing unit 140 in the frequency spectrum. . When the received sound processing unit 140 removes the components of the first mask frequency and the second mask frequency from the received sound, the component extraction unit 150 uses the first mask frequency component from the frequency spectrum of the microphone input sound, The component of the second mask frequency is extracted.

  Next, based on the value of the mask frequency component extracted from the microphone input sound, the communication device 1 determines whether or not the voice included in the processing target frame is in a double talk state (step S6). The processing in step S6 is performed by the double talk determination unit 160. The double talk determination unit 160 calculates power based on the mask frequency component extracted from the frequency spectrum of the microphone input sound, and determines that the processing target frame is in a double talk state when the power is greater than the threshold value. To do. When the component of the first mask frequency and the component of the second mask frequency are extracted from the microphone input sound, the double-talk determining unit 160 has, for example, the power of the first mask frequency and the power of the second mask frequency. When it is larger than the threshold value, it is determined that the processing target frame is in a double talk state.

  Next, the call device 1 performs processing (steps S7 to S9) for removing the echo component from the microphone input sound based on the determination result of the double talk determination unit 160. The input sound processing unit 170 performs the processes of steps S7 to S9. First, the input sound processing unit 170 determines whether or not the sound included in the processing target frame is in a double talk state from the determination result of the double talk determination unit 160 (step S7). The process of step S7 is performed by the learning determination unit 171 of the input sound processing unit 170.

  When the frame to be processed is not in a double talk state (step S7; NO), the input sound processing unit 170 performs a process of learning the transfer characteristic of the echo path (step S8), and then performs a microphone based on the transfer characteristic. A process of suppressing the echo component of the input sound (step S9) is performed. The process of step S8 is performed by the transfer characteristic learning unit 172 of the input sound processing unit 170. The transfer characteristic learning unit 172 learns the transfer characteristic of the echo path from the receiver 2 to the microphone 3 according to a known learning method. For example, the transfer characteristic learning unit 172 calculates the transfer characteristic at each frequency based on the frequency spectrum for the processing target frame in the received sound and the frequency spectrum for the processing target frame in the microphone input sound. The processing of step S9 is performed by the echo component suppression unit 173 of the input sound processing unit 170. The echo component suppression unit 173 suppresses echo components included in the microphone input sound by applying transfer characteristics to each frequency component in the frequency spectrum of the microphone input sound. When the process of step S8 is performed, the echo component suppressing unit 173 suppresses the echo component included in the microphone input sound based on the transfer characteristic learned in step S8.

  That is, when the processing target frame is not in the double talk state (step S7; NO), the learning determination unit 171 of the input sound processing unit 170 next causes the transfer characteristic learning unit 172 to learn the transfer characteristic. When the transfer characteristic learning unit 172 learns the transfer characteristic based on the frequency spectrum of the frame that is currently processed, the transfer characteristic learning unit 172 stores the learned transfer characteristic in the transfer characteristic storage unit 193 and passes it to the echo component suppression unit 173. Upon receiving the transfer characteristic from the transfer characteristic learning unit 172, the echo component suppressing unit 173 suppresses the echo component included in the processing target frame in the microphone input sound based on the received transfer characteristic. The echo component suppression unit 173 applies a transfer characteristic to each frequency component in the frequency spectrum of the microphone input sound, and suppresses the echo component included in the microphone input sound.

  On the other hand, when the processing target frame is in a double talk state (step S7; YES), the input sound processing unit 170 skips the process of step S8 and performs the process of step S9. That is, when the processing target frame is in a double talk state, the input sound processing unit 170 does not learn transfer characteristics. Next, the learning determination unit 171 of the input sound processing unit 170 causes the echo component suppression unit 173 to suppress the echo component included in the microphone input sound. At this time, the learning determination unit 171 causes the echo component suppression unit 173 to acquire, from the transfer characteristic storage unit 173, the transfer characteristic learned based on the frame (past frame) before the frame that is the current processing target. The echo component suppression unit 173 suppresses the echo component included in the processing target frame in the microphone input sound based on the transfer characteristic acquired from the transfer characteristic storage unit 193.

  After suppressing the echo component included in the microphone input sound, the communication device 1 transmits the microphone input sound (processed input sound) in which the echo component is suppressed to the other communication device 4 (step S10). The transmission / reception unit 110 performs the processing in step S10. The transmission / reception unit 110 performs predetermined processing on the microphone input sound in which the echo component is suppressed by the input sound processing unit 170, and then transmits the processed signal to the other communication device 4.

  After transmitting the microphone input sound to the other call device 4, the call device 1 determines whether the call with the other call device 4 is continued (step S11). The processing in step S11 is performed by, for example, the transmission / reception unit 110 or the received sound analysis unit 120. When the call is continued (step S11; YES), the call device 1 repeats the processes after step S1. When the call is finished (step S11; NO), the call device 1 finishes the above process.

  The flowchart of FIG. 3 shows the content of performing steps S1 to S10 on the next processing target frame after performing the processing of steps S2 to S10 on the processing target frame extracted in step S1. However, the communication device 1 may perform each process of steps S1 to S10 in a pipeline.

  The process of step S2 among the processes of steps S1 to S11 performed by the call device 1, that is, the process of performing frequency analysis on the frame of the received sound is performed by the received sound analysis unit 120 as described above. The received sound analysis unit 120 performs, for example, the process shown in FIG.

  FIG. 4 is a flowchart for explaining the contents of processing for frequency analysis of the frame of the received sound.

  As shown in FIG. 4, the received sound analysis unit 120 first converts the speech waveform of the frame to be processed in the received sound into a frequency spectrum (step S201). In the process of step S201, the received sound analysis unit 120 converts the speech waveform into a frequency spectrum according to a known conversion method. For example, the received sound analysis unit 120 converts a speech waveform into a frequency spectrum by fast Fourier transform.

  Next, the received sound analysis unit 120 calculates a power spectrum for the processing target frame based on the frequency spectrum (step S202). In the process of step S202, the received sound analysis unit 120 calculates the power of each frequency band in the frequency spectrum according to a known calculation method.

  Next, the received sound analysis unit 120 extracts the peak frequency in the frame currently being processed based on the power spectrum (step S203). In the process of step S203, the received sound analysis unit 120 extracts a frequency band that has a peak (maximum) in the power spectrum of the frame currently being processed according to a known extraction method. For example, the received sound analysis unit 120 determines that the power P (i, t) of the i-th frequency band in the power spectrum for the frame at time t satisfies the following expressions (1-1) and (1-2). The i-th frequency band is a peak frequency.

P (i, t) -P (i-1, t)> TH1 (1-1)
P (i, t) -P (i + 1, t)> TH1 (1-2)

  The determination threshold value TH1 in the expressions (1-1) and (1-2) can be set as appropriate.

  Next, the received sound analysis unit 120 extracts the continuous peak of each frequency band based on the time change of the power spectrum (step S204). In the process of step S204, the received sound analysis unit 120 refers to the power spectrum of a plurality of frames including the frame currently being processed, and peaks (maximum) in the time change of the power P (i, t) for each frequency band. Extract the frame time. For example, the received sound analysis unit 120 includes power P (i, t−1) satisfying the following expressions (2-1) and (2-2) among the powers of the i-th frequency band in the power spectrum of each frame. ) Including a power spectrum frame (time t-1) is a continuous peak.

P (i, t-1) -P (i, t-2)> TH1 (2-1)
P (i, t-1) -P (i, t)> TH1 (2-2)

  Next, the received sound analysis unit 120 causes the analysis result holding unit 192 to hold the frequency spectrum of the frame currently being processed, information indicating the peak frequency, and information indicating the continuous peak (step S205). When the process of step S205 is completed, the received sound analysis unit 120 ends the frequency analysis for the frame currently being processed.

  When the frequency analysis for the frame to be processed in the received sound is completed, the communication device 1 next performs a process of determining a mask frequency to be removed from the frame (step S3). The process of step S3 is performed by the mask frequency determination unit 130. The mask frequency determination unit 130 performs the process shown in FIG. 4 as the process of step S3.

  FIG. 5 is a flowchart for explaining the contents of the process for determining the mask frequency to be removed from the received sound.

  In the process of step S4, the mask frequency determination unit 130 first sets a variable i for designating a frequency band (frequency bin) in the frequency spectrum to 1 (step S301).

  Next, the mask frequency determination unit 130 calculates a first mask frequency Max_FM_freq masked by the peak frequency in the frame based on the peak frequency and the first mask information (step S302). The first mask frequency is a frequency at which the mask amount is maximum among the frequencies masked by the peak frequency component. The mask amount is a value obtained by subtracting the power in the frequency spectrum from the maximum value of the power (sound volume) masked by the peak frequency component. The mask frequency determination unit 130 calculates a mask amount for the i-th frequency band for each peak frequency in the processing target frame based on the first mask information. Then, the frequency band where the calculated mask amount is the maximum value is held as the first mask frequency.

  Next, the mask frequency determination unit 130 calculates the second mask frequency Max_TM_freq based on the succession peak and the second mask information (step S303). The second mask frequency is a frequency at which the mask amount is maximum among the frequencies masked by the same frequency component of the past frame in the processing target frame. The mask frequency determination unit 130 calculates the mask amount in the processing target frame for each successive peak in the time change of the i-th frequency band. Then, the frequency band in which the calculated mask amount is the maximum value is held as the second mask frequency.

  After completing the processing of steps S302 and S303, the mask frequency determination unit 130 next updates the variable i to i + 1 (step S304), and determines whether or not the updated variable i is larger than the closing price N (step S304). Step S305). The closing price N is the total number of frequency bands in the frequency spectrum. That is, in step S305, the mask frequency determination unit 130 determines whether or not the processing in steps S302 and S303 has been performed on all frequency bands in the frequency spectrum. When i ≦ N is satisfied (step S305; NO), the mask frequency determination unit 130 performs the processing from step S302 onward for the i-th frequency band at the present time. If i> N (step S305; YES), the mask frequency determination unit 130 determines the first mask frequency Max_FM_freq and the second mask frequency Max_TM_freq as mask frequencies (step S306). When the process of step S306 is completed, the mask frequency determination unit 130 ends the process of determining the mask frequency for the frame currently being processed.

  FIG. 6A is a flowchart (part 1) for explaining the content of the process of calculating the first mask frequency. FIG. 6B is a flowchart (part 2) illustrating the content of the process of calculating the first mask frequency.

  In the process of calculating the first mask frequency, the mask frequency determination unit 130 first determines whether or not the variable i specifying the frequency band is 1 as shown in FIG. 6A (step S30201). When i = 1 (step S30201; YES), the mask frequency determination unit 130 next sets the maximum mask amount Max_FM and the first mask frequency Max_FM_freq in the processing target frame to 0, respectively. (Step S30202). Thereafter, the mask frequency determination unit 130 sets the variable n to 1 and sets the maximum mask amount Max_FM_T and the provisional mask frequency Max_FM_freq_T in the i-th frequency band to 0 (step S30203). Here, the variable n is a value for identifying the peak frequency in the processing target frame. On the other hand, if i> 1 (step S30201; NO), the mask frequency determination unit 130 skips the process of step S30202 and then performs the process of step S30203.

  After the process of step S30203, the mask frequency determination unit 130 calculates a distance (frequency difference) from the nth peak frequency to the ith frequency (step S30204).

  Next, the mask frequency determination unit 130 calculates the maximum volume FM_TH (n, i) masked by the n-th peak frequency component in the i-th frequency band (step S30205). In step S30205, the mask frequency determination unit 130 calculates the maximum volume FM_TH (n, i) based on the first mask information indicating the relationship between the frequency difference and the maximum volume to be masked. The mask frequency determination unit 130 acquires first mask information from the mask information storage unit 191.

  Next, the mask frequency determination unit 130 determines the mask amount FM (n, i) based on the power P (i, t) of the i-th frequency in the processing target frame and the maximum volume FM_TH (n, i). = FM_TH (n, i) -P (i, t) is calculated (step S30206).

  Next, the mask frequency determination unit 130 determines whether or not the calculated mask amount FM (n, i) is larger than the current maximum mask amount value Max_FM_T in the i-th frequency band (step S30207). When FM (n, i)> Max_FM_T (step S30207; YES), the mask frequency determination unit 130 sets the mask amount maximum value Max_FM_T and the provisional mask frequency Max_FM_freq_T to the mask amount FM (n, i), And the variable i is updated (step S30207). Thereafter, the mask frequency determination unit 130 updates the variable n to n + 1 (step S30209), and determines whether or not the updated variable n is larger than the closing price N_FP (step S30210). The closing price N_FP is the number of peak frequencies in the processing target frame. On the other hand, when FM (n, i) ≦ Max_FM_T (step S30207; NO), the mask frequency determination unit 130 skips the process of step S30208, and performs the process of step S30209 and the determination of step S30210.

  When n ≦ N_FP is satisfied (step S30210; NO), the mask frequency determination unit 130 performs the processing after step S30204.

  If n> N_FP (step S30210; YES), the mask frequency determination unit 130 then sets the maximum mask amount Max_FM_T at the i-th frequency to the mask amount in the processing target frame, as shown in FIG. 6B. It is determined whether it is larger than the maximum value Max_FM (step S30211).

  When Max_FM_T> Max_FM (step S30211; YES), the mask frequency determination unit 130 updates the maximum mask amount Max_FM and the first mask frequency Max_FM_freq in the processing target frame to Max_FM_T and Max_FM_freq_T, respectively. (Step S30212). When the process of step S30212 is performed, the mask frequency determination unit 130 ends the process of calculating the first mask frequency while maintaining the updated maximum value Max_FM of the mask amount and the first mask frequency Max_FM_freq. .

  On the other hand, when Max_FM_T ≦ Max_FM (step S30211; NO), the mask frequency determination unit 130 skips the process of step S30212 and ends the process of calculating the first mask frequency. In this case (when Max_FM_T ≦ Max_FM), the mask frequency determination unit 130 holds the first mask frequency Max_FM and the first mask frequency Max_FM_freq in the state up to the (i−1) th frequency in the first state. The process of calculating the mask frequency is terminated.

FIG. 7 is a graph illustrating a method for calculating the first mask frequency.
In the graph of FIG. 7, the horizontal axis represents the frequency in the power spectrum for the processing target frame, and the vertical axis represents the power.

  A thick curve PS in FIG. 7 is a power spectrum in a frame to be processed, and includes a first peak P1, a second peak P2, and a third peak P3. When calculating the mask amount for the i-th frequency in the power spectrum PS according to the flowcharts of FIGS. 6A and 6B, the mask frequency determination unit 130 performs the following processing.

  The mask frequency determination unit 130 first calculates a frequency difference F1 (= i−IP1) between the frequency IP1 of the first peak P1 and the i-th frequency (step S30204).

  Next, the mask frequency determination unit 130 calculates the maximum volume of the i-th frequency masked by the first peak P1 (step S30205). At this time, the mask frequency determination unit 130 sets the first function G1 (f) based on the frequency IP1 and power PWP1 of the first peak P1 and the first mask information. The first function G1 (f) is a function indicating the relationship between the frequency difference from the frequency IP1 of the first peak P1 and the maximum volume to be masked in the same frame. The mask frequency determination unit 130 calculates the maximum volume FM_TH (1, i) for the i-th frequency band based on the first function G1 (f).

  Next, the mask frequency determination unit 130 calculates a mask amount FM (1, i) = FM_TH (1, i) −P (i, t) P (i, t) (step S30206).

  Next, the mask frequency determination unit 130 sets the maximum value of the mask amount and the provisional mask frequency in the processing target frame as the mask amount FM (1, i) and the variable i, respectively (step S30208).

  Thereafter, the mask frequency determination unit 130 updates the variable n for identifying the peak to 2 (step S30209), and performs the processing after step S30204. That is, the mask frequency determination unit 130 calculates the frequency difference −F2 between the second peak P2 and the i-th frequency band, and then based on the frequency and power of the second peak P2 and the first mask information. Thus, the second function G2 (f) is set. The second function G2 (f) is a function indicating the relationship between the frequency difference from the frequency IP2 of the second peak P2 and the maximum volume to be masked. The mask frequency determination unit 130 calculates the maximum volume FM_TH (2, i) for the i-th frequency band based on the second function G2 (f), and the mask amount FM (2, i) = FM_TH (2, i) −P (i, t) is calculated. In the example shown in FIG. 7, the mask amount FM (2, i) due to the second peak P2 is larger than the mask amount FM (1, i) due to the first peak P1. Therefore, the mask frequency determination unit 130 sets the maximum value of the mask amount and the temporary mask frequency in the processing target frame as the mask amount FM (2, i) and the variable i, respectively (step S30208).

  Thereafter, the mask frequency determination unit 130 updates the variable n for identifying the peak to 3 (step S30209), and performs the processing after step S30204. At this time, the mask frequency determination unit 130 sets the third function G3 (f) based on the frequency IP3 and power of the third peak P3 and the first mask information. The third function G3 (f) is a function indicating the relationship between the frequency difference of the third peak P3 from the frequency IP3 and the maximum volume to be masked. In the example shown in FIG. 7, the maximum volume FM_TH (3, i) for the i-th frequency band calculated based on the third function G3 (f) is the power P (i of the i-th frequency band. , t). Therefore, the mask frequency determination unit 130 sets the maximum mask amount Max_FM_T and the provisional mask frequency Max_FM_freq_T in the i-th frequency band as the mask amount FM (2, i) and the variable i, respectively, and makes the determination in step S30211. I do. Here, when the maximum value Max_FM_T of the mask amount at the i-th frequency is larger than the maximum value Max_FM of the mask amount in the process up to the i−1th frequency, the mask frequency determination unit 130 performs the process of step S30212. . That is, the mask frequency determination unit 130 updates the maximum mask amount Max_FM and the first mask frequency Max_FM_freq in the processing target frame to the maximum mask amount Max_FM_T and the variable i in the i-th frequency band, respectively. . On the other hand, when the maximum value Max_FM_T of the mask amount in the i-th frequency band is equal to or less than the maximum value Max_FM of the mask amount in the processing up to the i−1th frequency band, the mask frequency determination unit 130 performs the process of step S30212. Omitted. That is, the mask frequency determination unit 130 sets the maximum mask amount Max_FM and the first mask frequency Max_FM_freq in the processing target frame to the maximum value Max_FM_T in the processing up to the i−1th frequency band, and 1 to i, respectively. Leave any value up to -1.

  As shown in FIG. 5, the mask frequency determination unit 130 according to the present embodiment performs the processing of step S302 (steps S30201 to S30212) for each frequency band in the frequency spectrum for the processing target frame. As a result, the frequency band in which the mask amount FM (n, i) has the maximum value in the processing target frame becomes the first mask frequency.

  FIG. 8A is a flowchart (part 1) for explaining the contents of the process of calculating the second mask frequency. FIG. 8B is a flowchart (part 2) illustrating the content of the process of calculating the second mask frequency.

  In the process of calculating the second mask frequency, the mask frequency determination unit 130 first determines whether or not the variable i specifying the frequency band is 1 as shown in FIG. 8A (step S30301). When i = 1 (step S30301; YES), the mask frequency determination unit 130 next sets the maximum mask amount Max_TM and the second mask frequency Max_TM_freq in the processing target frame to 0, respectively. (Step S30302). Thereafter, the mask frequency determining unit 130 sets the variable n to 1, and sets the maximum mask amount Max_TM_T and the provisional mask frequency Max_TM_freq_T to 0 in the i-th frequency band, respectively (step S30303). Here, the variable n is a value for identifying the successive peak extracted from the time change of the i-th frequency component in each of the plurality of frames. On the other hand, if i> 1 (step S30301; NO), the mask frequency determination unit 130 skips the process of step S30302 and then performs the process of step S30303.

  After the process of step S30303, the mask frequency determination unit 130 calculates the distance (time difference) from the frame including the nth successive peak to the frame to be processed (step S30304).

  Next, the mask frequency determination unit 130 calculates the maximum volume TM_TH (n, i) masked by the n-th peak component in the i-th frequency band of the processing target frame (step S30305). In step S30305, the mask frequency determination unit 130 calculates the maximum volume TM_TH (n, i) based on the second mask information indicating the relationship between the time difference and the maximum volume to be masked. The mask frequency determination unit 130 acquires second mask information from the mask information storage unit 191.

  Next, the mask frequency determination unit 130 determines the mask amount TM (n, i) based on the power P (i, t) of the i-th frequency band in the processing target frame and the maximum volume TM_TH (n, i). ) = TM_TH (n, i) −P (i, t) is calculated (step S30306).

  Next, the mask frequency determination unit 130 determines whether or not the calculated mask amount TM (n, i) is larger than the maximum value Max_TM_T of the current mask amount in the i-th frequency band (step S30307). When TM (n, i)> Max_TM_T (step S30307; YES), the mask frequency determining unit 130 sets the mask amount maximum value Max_TM_T and the provisional mask frequency Max_TM_freq_T to the mask amount TM (n, i), And the variable i is updated (step S30307). Thereafter, the mask frequency determination unit 130 updates the variable n to n + 1 (step S30309), and determines whether or not the updated variable n is larger than the closing price N_TP (step S30310). The closing price N_TP is the number of successive peaks in the time change of the i-th frequency component. On the other hand, when TM (n, i) ≦ Max_TM_T (step S30307; NO), the mask frequency determination unit 130 skips the process of step S30308, and performs the process of step S30309 and the determination of step S30310.

  When n ≦ N_TP is satisfied (step S30310; NO), the mask frequency determination unit 130 performs the processing after step S30304.

  When n> N_TP is satisfied (step S30310; YES), the mask frequency determination unit 130 then sets the maximum mask amount Max_TM_T in the i-th frequency band as shown in FIG. It is determined whether or not it is larger than the maximum value Max_TM of the mask amount at step S30311.

  When Max_TM_T> Max_TM is satisfied (step S30311; YES), the mask frequency determination unit 130 updates the maximum mask amount Max_TM and the second mask frequency Max_TM_freq in the processing target frame to Max_TM_T and Max_TM_freq_T, respectively. (Step S30312). When the process of step S30312 is performed, the mask frequency determination unit 130 ends the process of calculating the second mask frequency while holding the updated maximum value Max_TM of the mask amount and the second mask frequency Max_TM_freq. .

  On the other hand, when Max_TM_T ≦ Max_TM (step S30311; NO), the mask frequency determination unit 130 skips the process of step S30312 and ends the process of calculating the second mask frequency. In this case (when Max_TM_T ≦ Max_TM), the mask frequency determination unit 130 holds the maximum mask amount Max_TM and the second mask frequency Max_TM_freq in the process up to the (i−1) -th frequency band. The process of calculating the mask frequency of 2 ends.

FIG. 9 is a graph illustrating a method for calculating the second mask frequency.
In the graph of FIG. 9, the horizontal axis represents time (frame) in the received sound, and the vertical axis represents power.

  The thick curve PSi in FIG. 9 represents the time change of the power of the i-th frequency band in a plurality of frames, and the first successive peak P1, the second successive peak P2, and the third successive peak P3. including. When calculating the mask amount at time t in the power spectrum PSi along the flowcharts of FIGS. 8A and 8B, the mask frequency determination unit 130 performs the following processing.

  First, the mask frequency determination unit 130 calculates a time difference T1 (= t−TP1) between a frame including the first succession peak P1 and a frame currently being processed (step S30304).

  Next, the mask frequency determination unit 130 calculates the maximum volume of the i-th frequency band masked by the first succession peak P1 (step S30305). At this time, the mask frequency determination unit 130 sets the first function H1 (T) based on the time TP1 and the power PWPa of the first continuous peak P1 and the second mask information. The first function H1 (T) is a function showing the relationship between the time difference from the time TP1 of the first continuous peak P1 in the time direction and the maximum volume to be masked. After that, the mask frequency determination unit 130 calculates the maximum volume FM_TH (1, i) for the i-th frequency band in the frame at the time t currently being processed based on the first function H1 (T). To do.

  Next, the mask frequency determination unit 130 calculates the mask amount TM (1, i) = TM_TH (1, i) −P (i, t) (step S30306). In the example shown in FIG. 9, the maximum volume TM_TH (1, i) based on the first transition peak P1 is smaller than the power P (i, t) in the frame at time t. For this reason, the mask frequency determination unit 130 omits step S30308, and then performs the processing after step S30304 for the second continuous peak P2. However, the maximum volume TM_TH (2, i) calculated based on the second function H2 (T) for the second transition peak P2 is smaller than the power P (i, t) in the frame at time t. . For this reason, the mask frequency determination unit 130 omits step S30308, and then performs the processing after step S30304 for the third continuous peak P3. Here, the second function H2 (T) is a function indicating the relationship between the time difference from the time TP2 of the second continuous peak P2 in the time direction and the maximum volume to be masked.

  The maximum volume FM_TH (3, i) calculated based on the third function H3 (T) for the third continuous peak P3 is larger than the power P (i, t) in the frame at time t. Therefore, in step S30308, the mask frequency determining unit 130 sets the maximum mask amount and the temporary mask frequency in the processing target frame as the mask amount TM (3, i) and the variable i, respectively. Here, the third function H3 (T) is a function indicating the relationship between the time difference from the time TP3 of the third successive peak P3 in the time direction and the maximum volume to be masked.

  Thereafter, the mask frequency determining unit 130 sets the maximum mask amount Max_TM_T and the provisional mask frequency Max_TM_freq_T in the i-th frequency band of the frame at time t as the mask amount FM (3, i) and the variable i, respectively. The determination in step S30311 is performed. Here, if the maximum value Max_TM_T of the mask amount in the i-th frequency band is larger than the maximum value Max_TM of the mask amount in the process up to the i−1th frequency band, the mask frequency determination unit 130 performs the process of step S30312. I do. That is, the mask frequency determination unit 130 updates the maximum mask amount Max_TM and the second mask frequency Max_TM_freq in the processing target frame to the maximum mask amount Max_TM_T and the variable i in the i-th frequency band, respectively. . On the other hand, when the maximum value Max_TM_T of the mask amount in the i-th frequency band is equal to or less than the maximum value Max_TM of the mask amount in the processing up to the i−1th frequency band, the mask frequency determination unit 130 performs the process of step S30312. Omitted. That is, the mask frequency determination unit 130 sets the maximum mask amount Max_TM and the second mask frequency Max_TM_freq in the frame to be processed to the maximum mask amount Max_TM_T in the processing up to the (i−1) th frequency band, and It remains either 1 to i-1.

  As described above, the mask frequency determination unit 130 according to the present embodiment performs the processing of step S303 (steps S30301 to S30312) for each frequency band in the frequency spectrum for the processing target frame. As a result, the frequency band in which the mask amount TM (n, i) has the maximum value in the processing target frame becomes the second mask frequency.

  After calculating the first mask frequency and the second mask frequency by the above processing, the communication device 1 removes the component of the mask frequency from the received sound and outputs it (step S4). The process of removing the mask frequency component is performed by the received sound processing unit 140. The received sound processing unit 140 performs, for example, the process shown in FIG.

  FIG. 10 is a flowchart for explaining the contents of the process of removing the mask frequency component from the received sound and outputting it.

  In the process of removing the mask frequency component from the received sound and outputting it, the received sound processing unit 140 first changes the value of the first mask frequency Max_FM_freq in the frequency spectrum of the received sound to 0 (step S401).

  Next, the received sound processing unit 140 changes the value of the second mask frequency Max_TM_freq in the frequency spectrum of the received sound to 0 (step S402).

  Next, the received sound processing unit 140 inversely converts the frequency spectrum in which the values of the first mask frequency and the second mask frequency are set to 0 into a speech waveform (step S403). In step S403, the received sound processing unit 140 converts the frequency spectrum of the received sound into a speech waveform by an inverse conversion method corresponding to the conversion method used when converting the speech waveform into a frequency spectrum in step S201.

  After that, the received sound processing unit 140 outputs the voice waveform to the receiver 2 and outputs the information of the first mask frequency and the second mask frequency to the component extraction unit 150 (step S404), and performs the process of step S4. finish.

  As described above, the communication device 1 according to the present embodiment generates a processed reception sound obtained by removing the first mask frequency component and the second mask frequency component from the reception sound, and the processed reception sound is received from the receiver 2. It outputs (emits) to the external space of the communication device 1. Therefore, when the received sound output from the receiver 2 is picked up by the microphone 3, the components corresponding to the first mask frequency and the second mask frequency in the microphone input sound are almost zero. However, when the received sound output from the receiver 2 is picked up by the microphone 3, if the user of the communication device 1 is speaking or if noise is generated around the communication device 1, the microphone input sound is generated. Includes sound other than the received sound. When the microphone input sound is in a double talk state in which the voice of the user of the communication device 1 is included in the microphone input sound or noise is included around the communication device 1, the microphone input sound includes a mask frequency component removed from the received sound. It is often done. For this reason, in the communication device 1 of the present embodiment, it is determined whether or not the microphone input sound is in a double talk state based on the mask frequency component removed from the received sound in the microphone input sound. Therefore, as described above, the communication device 1 extracts the mask frequency component included in the microphone input sound (step S5), and determines whether or not it is in a double talk state (step S6). The component extraction unit 150 performs the process in step S5, and the double talk determination unit 160 performs the process in step S6. The component extraction unit 150 extracts the components of the first mask frequency and the second mask frequency after converting the processing target frame in the microphone input sound from the speech waveform to the frequency spectrum. The double talk determining unit 160 performs the process of FIG. 11 to determine whether or not the processing target frame is in a double talk state.

  FIG. 11 is a flowchart for explaining the contents of the process for determining whether or not it is double talk.

  First, the double talk determination unit 160 calculates the first power PF based on the value of the first mask frequency Max_FM_freq extracted from the frequency spectrum of the microphone input sound (step S601). Next, the double talk determination unit 160 calculates the second power PT based on the value of the second mask frequency Max_TM_freq extracted from the frequency spectrum of the microphone input sound (step S602). In steps S601 and S602, the double talk determination unit 160 calculates the first power PF and the second power PT based on a known calculation method.

  Next, the double talk determination unit 160 determines whether or not the first power PF is larger than the first power threshold TH2 (step S603). The first power threshold TH2 can be selected as appropriate. When PF ≦ TH2 is satisfied (step S603; NO), the double talk determination unit 160 determines that the processing target frame in the microphone input sound is not double talk, and sets a flag DT_flag indicating whether or not it is double talk to 0. (Step S605).

  On the other hand, if PF> TH2 (step S603; YES), the double-talk determining unit 160 next determines whether or not the second power PT is larger than the second power threshold TH3 (step S604). . The second power threshold TH3 can be selected as appropriate. When PT ≦ TH3 is satisfied (step S604; NO), the double talk determining unit 160 determines that the frame to be processed in the microphone input sound is not double talk, and sets a flag DT_flag indicating whether or not it is double talk to 0. (Step S605).

  On the other hand, when PT> TH3 (step S604; YES), the double talk determining unit 160 determines that the frame to be processed in the microphone input sound is double talk, and indicates whether or not it is double talk. The flag DT_flag is set to 1 (step S606).

  When the flag for the processing target frame is set in step S605 or S606, the double talk determination unit 160 ends the determination processing for the frame.

  In this way, when the value of the flag DT_flag indicates whether or not the frame to be processed in the microphone input sound is in a double talk state, the learning determination unit 171 of the input sound processing unit 170 transmits the value based on the value of the flag DT_flag. Determine whether to learn the characteristics. When the flag DT_flag is 0 (that is, when not in a double talk state), the learning determination unit 171 causes the transfer characteristic learning unit 172 to learn transfer characteristics. On the other hand, when the flag DT_flag is 1 (that is, in the case of a double talk state), the learning determination unit 171 omits the process of learning the transfer characteristic, and the transmission learned by the echo component suppression unit 173 through the process for the past frame. The echo component is suppressed based on the characteristics.

  As described above, in the communication device 1 according to the present embodiment, the mask frequency component that does not affect human perception in the voice included in the received sound is removed, and then the receiver 2 is connected to the outside of the communication device 1. Output. For this reason, when a significant voice component is included in the mask frequency in the microphone input sound collected by the microphone 3, the component is a double such as the voice of the user of the call device 1 and noise around the call device 1. It can be regarded as a talk component. In addition, by removing the mask frequency component included in the received sound, it is possible to determine the presence or absence of the double talk component even when the volume of the double talk component is low, and the microphone input sound is in a double talk state. It is possible to accurately determine whether or not. In addition, in the communication device 1 according to the present embodiment, the received reception sound from which the mask frequency component is removed from the reception sound received from the other communication device 4 and output to the receiver 2 is removed. Is output from the receiver 2. For this reason, in the communication device 1 according to the present embodiment, it is possible to reduce the processing amount as compared with the case where the suppression of the echo component is controlled using the pseudo echo signal generated based on the received sound. Therefore, according to the present embodiment, it is possible to efficiently and correctly determine whether or not the microphone input sound in the call device is in a double talk state. As a result, it is possible to prevent learning of erroneous transfer characteristics due to an erroneous determination as to whether or not a double talk state or suppression of echo components due to inappropriate transfer characteristics.

  Note that the flowchart in FIG. 5 is merely an example of a process for determining a mask frequency to be removed from the received sound. The process for determining the mask frequency is not limited to the process according to the flowchart of FIG. For example, the process of step S302 and the process of step S303 in the flowchart of FIG. 5 are out of order, and step S303 may be performed first. Further, the process of step S302 and the process of step S303 may be performed in parallel. The process for determining the mask frequency includes, for example, a process for extracting a third mask frequency having the third largest mask amount when the first mask frequency and the second mask frequency are the same. It may be included. Further, the process for determining the mask frequency may be a process for determining only one of the first mask frequency and the second mask frequency. In calculating the first mask frequency, for example, only the frequency band in which the frequency difference from the peak frequency is within a predetermined range, that is, the frequency band near the peak frequency may be processed.

  The mask information referred to when calculating the mask frequency is not limited to the functions G (f) and H (t) shown in FIG.

  Further, the flowchart of FIG. 11 is merely an example of a process for determining whether or not double talk. The process for determining whether or not it is double talk is not limited to the process according to the flowchart of FIG. For example, after calculating the first power PF in step S601, the double talk determining unit 160 may continue to perform the determination in step S603. By doing so, for example, when it is determined that it is not double talk based on the first power PF, it is possible to omit the process of calculating the second power PT, and in the double talk determination unit 160 The processing amount can be reduced. Further, instead of the determination in steps S603 and S604, the double talk determination unit 160 may determine that the processing target frame is in a double talk state when PF> TH2 or PT> TH3, for example. . Furthermore, the double talk determination unit 160 and the learning determination unit 171 may be combined into one, and it may be determined whether or not the transfer characteristics are to be learned in steps S605 and S606.

  In addition, the determination result of whether or not the microphone input sound is in a double talk state is not limited to determining whether or not to learn the transfer characteristic used for suppressing the echo component, and determining whether or not to perform other processing. It is also possible to use it. For example, the determination result as to whether or not the state is a double talk state can be used for a process of classifying voice data for each speaker in a telephone conference system or the like.

  The call device 1 according to the above-described embodiment can be realized, for example, by causing a mobile terminal having a call function such as a mobile phone terminal or a smartphone to execute a predetermined echo suppression program.

FIG. 12 is a diagram illustrating a hardware configuration of the mobile terminal.
As illustrated in FIG. 12, the mobile terminal 7 includes a processor 701, a memory 702, an input device 703, a display device 704, an RF transceiver 705, an antenna 706, a receiver 2, and a microphone 3.

  The processor 701 is a central processing unit (CPU), a micro processing unit (MPU), or the like. The processor 701 controls the overall operation of the mobile terminal 7 by executing various programs including an operating system. Further, the processor 701 executes various application programs such as an echo suppression program including the processes of steps S1 to S11 in FIG. The processor 701 includes, for example, a baseband processing circuit 701a and an application processing circuit 701b.

  The memory 702 includes a read only memory (ROM) and a random access memory (RAM) not shown. In the ROM of the memory 702, for example, a predetermined basic control program read by the processor 701 when the mobile terminal 7 is started is recorded in advance. On the other hand, the RAM of the memory 702 is used as a working storage area as needed when the processor 701 executes various programs, for example. The memory 702 can be used to store various programs executed by the processor 701 and various data. The memory 702 can be used for storing mask information, transfer characteristics, analysis results for a plurality of frames in the received sound, and the like. The memory 702 may be a non-volatile memory (including a solid state drive (SSD)) such as a flash memory built in the mobile terminal 7, or may be a Secure Digital (SD) standard memory card (flash Memory) or the like.

  The input device 703 is, for example, a keyboard device or a touch panel device. When the operator (user) of the mobile terminal 7 performs a predetermined operation on the input device 703, the input device 703 transmits input information associated with the operation content to the processor 701. The input device 703 can be used to input, for example, a command to start a call and a command to end a call.

  The display device 704 is a display device such as a liquid crystal display device. The display device 704 can be used to display a name, a telephone number, etc. for identifying a call partner.

  The RF transceiver 705 is a device that connects the mobile terminal 7 to a network such as a telephone network, and controls transmission / reception of voice data between the mobile terminal 7 and the other communication device 4 via the network. The RF transmitter / receiver 705 converts the radio wave received by the antenna 706 into an electric signal and passes it to the processor 701, and performs processing for outputting the electric signal received from the processor 701 from the antenna 706 as a radio wave.

  In the mobile terminal 7, for example, when an operator inputs a call start command using the input device 703 or the like, the processor 701 reads out an echo suppression program stored in a non-temporary recording medium such as a RAM of the memory 702. And execute. When the echo suppression program is a program including the processes of steps S1 to S11 in FIG. 3, the mobile terminal 7 performs a process of removing the mask frequency component from the received sound and a process of suppressing the echo component from the microphone input sound. repeat. While executing the echo suppression program, the processor 701 receives the received sound analysis unit 120, the mask frequency determination unit 130, the received sound processing unit 140, the component extraction unit 150, the double talk determination unit 160 in the communication device 1 of FIG. And functions (operates) as the input sound processing unit 170. Further, while the processor 701 is executing the echo suppression program, the RF transceiver 705 and the antenna 706 function as the transceiver 110 in the call device 1 of FIG. Further, while the processor 701 is executing the echo suppression program, the memory 702 functions as a mask information storage unit 191, an analysis result holding unit 192, and a transfer characteristic storage unit 193.

  Note that the mobile terminal 7 includes an arithmetic device different from the processor 701 such as a digital signal processor (DSP), for example, and a part of the processing of steps S1 to S11 in FIG. May be performed.

  Further, the call device 1 according to the present embodiment is not limited to the mobile terminal 7 described above, and can also be realized by causing a computer capable of transmitting and receiving voice data using a network such as the Internet to execute a predetermined echo suppression program. It is.

FIG. 13 is a diagram illustrating a hardware configuration of a computer.
As shown in FIG. 13, the computer 8 includes a processor 801, a main storage device 802, an auxiliary storage device 803, an input device 804, an output device 805, an input / output interface 806, a communication control device 807, and a medium. A driving device 808. These elements 801 to 808 in the computer 8 are connected to each other via a bus 810 so that data can be exchanged between the elements.

  The processor 801 is a CPU, MPU, or the like. The processor 801 controls the overall operation of the computer 8 by executing various programs including an operating system. Further, the processor 801 executes various application programs such as a voice processing program including the processes of steps S1 to S11 in FIG.

  The main storage device 802 includes a read only memory (ROM) and a random access memory (RAM) not shown. In the ROM of the main storage device 802, for example, a predetermined basic control program read by the processor 801 when the computer 8 is started is recorded in advance. On the other hand, the RAM of the main storage device 802 is used as a working storage area as needed when the processor 801 executes various programs. The RAM of the main storage device 802 can be used, for example, for storing mask information, transfer characteristics, analysis results for a plurality of frames in the received sound, and the like.

  The auxiliary storage device 803 is a storage device having a larger capacity than the RAM of the main storage device 802, and includes, for example, a hard disk drive (HDD) and a non-volatile memory (Solid State Drive (SSD)) such as a flash memory. ) Etc. The auxiliary storage device 803 can be used for storing various programs executed by the processor 801 and various data. The auxiliary storage device 803 can be used, for example, for storing an echo suppression program including the processes of steps S1 to S11 in FIG. The auxiliary storage device 803 can be used for storing mask information, transfer characteristics, analysis results of a plurality of frames in received sound, storage of received sound and microphone input sound, and the like.

  The input device 804 is, for example, a keyboard device or a touch panel device. When an operator (user) of the computer 8 performs a predetermined operation on the input device 804, the input device 804 transmits input information associated with the operation content to the processor 801. For example, the input device 804 can be used to input, for example, a command to start or end a call.

  The output device 805 is a display device such as a liquid crystal display device. The output device 805 can be used to display a name, a telephone number, etc. for identifying a call partner.

  The input / output interface 806 connects the computer 8 and other electronic devices. The input / output interface 806 includes, for example, a phone jack, a universal serial bus (USB) standard connector, and the like. The input / output interface 806 can be used, for example, for connection between the computer 8 and the headset 10 including the receiver 2 and the microphone 3.

  The communication control device 807 is a device that connects the computer 8 to a network such as the Internet and controls various communications between the computer 8 and other communication devices via the network. The communication control device 807 can be used for transmission / reception of audio data between the computer 8 and another communication device 4 or the like, for example.

  The medium driving device 808 reads a program and data recorded in the portable storage medium 890 and writes data stored in the auxiliary storage device 803 to the portable storage medium 9. For the medium driving device 808, for example, a memory card reader / writer corresponding to one type or a plurality of types of standards can be used. When a memory card reader / writer is used as the medium driving device 808, the portable storage medium 9 is a standard compatible with the memory card reader / writer, such as a Secure Digital (SD) standard memory card (flash memory). ) Etc. can be used. In addition, as the portable recording medium 9, for example, a flash memory having a USB standard connector can be used. Further, when the computer 8 is equipped with an optical disk drive that can be used as the medium driving device 808, various optical disks that can be recognized by the optical disk drive can be used as the portable recording medium 9. Examples of the optical disc that can be used as the portable recording medium 9 include a Compact Disc (CD), a Digital Versatile Disc (DVD), and a Blu-ray Disc (Blu-ray is a registered trademark). The portable recording medium 9 can be used, for example, for storing an echo suppression program including the processes of steps S1 to S11 in FIG. In addition, the portable recording medium 9 can be used for storing mask information, transfer characteristics, analysis results of a plurality of frames in received sound, storage of received sound and microphone input sound, and the like.

  For example, when the operator inputs an instruction to start a call using the input device 804 or the like, the computer 8 reads the echo suppression program stored in a non-temporary recording medium such as the auxiliary storage device 803 by the processor 801. Run. When the echo suppression program is a program including the processes of steps S1 to S11 in FIG. 3, the computer 8 removes the mask frequency component from the received sound and outputs it, and suppresses the echo component from the microphone input sound. Repeat the output process. While executing the voice processing program, the processor 801 receives the received sound analysis unit 120, the mask frequency determination unit 130, the received sound processing unit 140, the component extraction unit 150, the double talk determination unit 160 in the communication device 1 of FIG. And functions (operates) as the input sound processing unit 170. While the processor 801 is executing the echo suppression program, the communication control device 807 functions as the transmission / reception unit 110 in the call device 1 of FIG. Furthermore, while the processor 801 is executing the echo suppression program, the RAM of the main storage device 802, the auxiliary storage device 803, the portable recording medium 9 and other recording media include a mask information storage unit 191, an analysis result holding unit 192, And functions as a transfer characteristic storage unit 193.

  Note that the computer 8 that operates as the communication device 1 does not need to include all the elements 801 to 808 illustrated in FIG. 13, and some elements may be omitted depending on applications and conditions. For example, the computer 8 may be one in which the medium driving device 808 is omitted.

The following additional notes are disclosed with respect to the embodiment described above.
(Appendix 1)
A received sound processing unit that removes a component of a predetermined frequency band from the received sound received from the external device and outputs the component to the receiver;
A component extractor for extracting a component of the predetermined frequency band in the input sound input from the microphone;
A transfer characteristic learning unit for learning a transfer characteristic of an echo path from the receiver to the microphone based on the received voice and the input voice;
A determination unit that causes the transfer characteristic learning unit to learn the transfer characteristic when the magnitude of the component of the predetermined frequency band extracted from the input speech is equal to or less than a threshold;
A call device comprising:
(Appendix 2)
The call device further includes a frequency determination unit that determines the predetermined frequency band based on a frequency band component that peaks in the frequency spectrum of the received voice.
The telephone call device according to supplementary note 1, wherein:
(Appendix 3)
The frequency determination unit determines the magnitude of the component in the frequency spectrum based on the relationship between the frequency difference from the peak frequency band in one frequency spectrum and the maximum volume masked by the component in the peak frequency band. A frequency band having a frequency smaller than the maximum volume is determined as the predetermined frequency band;
The telephone call device according to Supplementary Note 2, wherein
(Appendix 4)
The frequency determination unit determines, as the predetermined frequency band, a frequency band having a maximum difference from the maximum volume among a plurality of frequency bands having a component size in the frequency spectrum smaller than the maximum volume. To
The telephone call device according to Supplementary Note 3, wherein
(Appendix 5)
The frequency determination unit has a frequency difference from the peak frequency band among a plurality of frequency bands whose component in the frequency spectrum is smaller than the maximum volume, and the maximum A frequency band having a maximum difference from the volume is determined as the predetermined frequency band;
The telephone call device according to Supplementary Note 3, wherein
(Appendix 6)
The frequency determination unit is masked by a time difference from a peak time in a time change of a component size and a component of the peak time from a plurality of frequency bands in the frequency spectrum continuous in the time direction. Based on the relationship with the maximum volume to extract a frequency band whose component in the current frequency spectrum is smaller than the maximum volume, and to determine the predetermined frequency band from the extracted frequency band,
The telephone call device according to Supplementary Note 3, wherein
(Appendix 7)
The frequency determination unit is configured to select a frequency band having a maximum difference from the maximum volume among the plurality of frequency bands having a component size in the current frequency spectrum smaller than the maximum volume as the predetermined frequency. Decide on the obi,
The telephone call device according to supplementary note 6, wherein:
(Appendix 8)
The call device further includes an echo component suppression unit that suppresses an echo component included in the input voice based on the transfer characteristic.
The telephone call device according to supplementary note 1, wherein:
(Appendix 9)
Remove the component of the predetermined frequency band from the received voice received from the external device and output it to the receiver.
Extracting the component of the predetermined frequency band in the input voice input from the microphone;
Learning the transmission characteristics of the echo path from the receiver to the microphone based on the received voice and the input voice when the magnitude of the component of the predetermined frequency band extracted from the input voice is less than or equal to a threshold value And
Suppressing echo components included in the input speech based on the transfer characteristics;
An echo suppression program for causing a computer to execute processing.

1, 4 Communication device 2 Receiver 3 Microphone 5,501 Processed reception sound 6 User's voice 7 Mobile terminal 8 Computer 9 Portable recording medium 10 Headset 110 Transmission / reception unit 120 Reception sound analysis unit 130 Mask frequency determination unit 140 Received sound processing Unit 150 component extraction unit 160 double talk determination unit 170 input sound processing unit 171 learning determination unit 172 transmission characteristic learning unit 173 echo component suppression unit 191 mask information storage unit 192 analysis result holding unit 193 transmission characteristic storage units 701, 801 processor 701a base Band processing circuit 701b Application processing circuit 702 Memory 703, 804 Input device 704 Display device 705 RF transceiver 706 Antenna 802 Main storage device 803 Auxiliary storage device 805 Output device 806 I / O interface 807 Communication control device 80 A medium driving device

Claims (6)

A received sound processing unit that removes a component of a predetermined frequency band from the received sound received from the external device and outputs the component to the receiver;
A component extractor for extracting a component of the predetermined frequency band in the input sound input from the microphone;
A transfer characteristic learning unit for learning a transfer characteristic of an echo path from the receiver to the microphone based on the received voice and the input voice;
A determination unit that causes the transfer characteristic learning unit to learn the transfer characteristic when the magnitude of the component of the predetermined frequency band extracted from the input speech is equal to or less than a threshold;
A call device comprising: The call device further includes a frequency determination unit that determines the predetermined frequency band based on a frequency band component that peaks in the frequency spectrum of the received voice.
The call device according to claim 1. The frequency determination unit determines the magnitude of the component in the frequency spectrum based on the relationship between the frequency difference from the peak frequency band in one frequency spectrum and the maximum volume masked by the component in the peak frequency band. A frequency band having a frequency smaller than the maximum volume is determined as the predetermined frequency band;
The communication device according to claim 2. The frequency determination unit is masked by a time difference from a peak time in a time change of a component size and a component of the peak time from a plurality of frequency bands in the frequency spectrum continuous in the time direction. Based on the relationship with the maximum volume to extract a frequency band whose component in the current frequency spectrum is smaller than the maximum volume, and to determine the predetermined frequency band from the extracted frequency band,
The communication device according to claim 3. The frequency determination unit is configured to select a frequency band having a maximum difference from the maximum volume among the plurality of frequency bands having a component size in the current frequency spectrum smaller than the maximum volume as the predetermined frequency. Decide on the obi,
The call device according to claim 4, wherein: Remove the component of the predetermined frequency band from the received voice received from the external device and output it to the receiver.
Extracting the component of the predetermined frequency band in the input voice input from the microphone;
Learning the transmission characteristics of the echo path from the receiver to the microphone based on the received voice and the input voice when the magnitude of the component of the predetermined frequency band extracted from the input voice is less than or equal to a threshold value And
Suppressing echo components included in the input speech based on the transfer characteristics;
An echo suppression program for causing a computer to execute processing.

Images