Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS
  • H04R1/00—Details of transducers, loudspeakers or microphones
  • H04R1/10—Earpieces; Attachments therefor ; Earphones; Monophonic headphones
  • H04R1/1041—Mechanical or electronic switches, or control elements
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/16—Sound input; Sound output
  • G06F3/165—Management of the audio stream, e.g. setting of volume, audio stream path
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/16—Sound input; Sound output
  • G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
  • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
  • G10L21/0208—Noise filtering
  • G10L21/0216—Noise filtering characterised by the method used for estimating noise
  • G10L21/0232—Processing in the frequency domain
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
  • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
  • G10L21/0316—Speech enhancement, e.g. noise reduction or echo cancellation by changing the amplitude
  • G10L21/0324—Details of processing therefor
  • G10L21/034—Automatic adjustment
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
  • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
  • G10L21/0316—Speech enhancement, e.g. noise reduction or echo cancellation by changing the amplitude
  • G10L21/0364—Speech enhancement, e.g. noise reduction or echo cancellation by changing the amplitude for improving intelligibility
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
  • G10L25/78—Detection of presence or absence of voice signals
  • G10L25/84—Detection of presence or absence of voice signals for discriminating voice from noise
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04M—TELEPHONIC COMMUNICATION
  • H04M3/00—Automatic or semi-automatic exchanges
  • H04M3/08—Indicating faults in circuits or apparatus
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04M—TELEPHONIC COMMUNICATION
  • H04M3/00—Automatic or semi-automatic exchanges
  • H04M3/42—Systems providing special services or facilities to subscribers
  • H04M3/56—Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities
  • H04M3/563—User guidance or feature selection
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04M—TELEPHONIC COMMUNICATION
  • H04M3/00—Automatic or semi-automatic exchanges
  • H04M3/42—Systems providing special services or facilities to subscribers
  • H04M3/56—Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities
  • H04M3/568—Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS
  • H04R3/00—Circuits for transducers
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS
  • H04R3/00—Circuits for transducers
  • H04R3/005—Circuits for transducers for combining the signals of two or more microphones
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
  • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
  • G10L21/0208—Noise filtering
  • G10L21/0216—Noise filtering characterised by the method used for estimating noise
  • G10L2021/02161—Number of inputs available containing the signal or the noise to be suppressed
  • G10L2021/02165—Two microphones, one receiving mainly the noise signal and the other one mainly the speech signal
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
  • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
  • G10L21/0208—Noise filtering
  • G10L21/0216—Noise filtering characterised by the method used for estimating noise
  • G10L2021/02161—Number of inputs available containing the signal or the noise to be suppressed
  • G10L2021/02166—Microphone arrays; Beamforming
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
  • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
  • G10L21/0208—Noise filtering
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
  • G10L25/78—Detection of presence or absence of voice signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS
  • H04R2201/00—Details of transducers, loudspeakers or microphones covered by H04R1/00 but not provided for in any of its subgroups
  • H04R2201/10—Details of earpieces, attachments therefor, earphones or monophonic headphones covered by H04R1/10 but not provided for in any of its subgroups
  • H04R2201/107—Monophonic and stereophonic headphones with microphone for two-way hands free communication
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS
  • H04R2410/00—Microphones
  • H04R2410/05—Noise reduction with a separate noise microphone
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS
  • H04R2420/00—Details of connection covered by H04R, not provided for in its groups
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS
  • H04R2420/00—Details of connection covered by H04R, not provided for in its groups
  • H04R2420/07—Applications of wireless loudspeakers or wireless microphones
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS
  • H04R2430/00—Signal processing covered by H04R, not provided for in its groups
  • H04R2430/01—Aspects of volume control, not necessarily automatic, in sound systems

Definitions

• the present invention relates to the field of audio communication, such as two- way audio communication over a communication link, e.g. on-line two-way communication.
• the invention proposes a method for microphone mute notification to the user based on a voice activity detection algorithm, e.g. using one or more microphone inputs to eliminate or reduce disturbing notifications of the user.
• a headset has many advantages for attending online calls or meetings, but some advantages also have drawbacks. It is desired to be able to mute the headset microphone if e.g., the user has nothing to add to the call for a while. One drawback with this functionality is that once the user wants to speak in the call, the user might have forgot that the microphone is muted.
• This problem is in several cases solved by detecting if the user is speaking into a muted microphone and then making a visual or an audible notification to the user.
• the audible notification can be an advantage as the user in a call will always hear the notification.
• one drawback of this functionality is that sometimes the user might intentionally speak into a muted microphone.
• the headset microphone also picks up surrounding sounds. These surroundings sounds may include a colleague speaking. This can lead to the situation where the user is in a call with the microphone muted, and when a colleague is speaking, the headset microphone picks up the speech and alerts the user that he is speaking into a muted microphone, which is not the case.
• EP 2 881 946 A1 describes a microphone mute/unmute system which detects silence events and voice activities from far-end and near-end audio signals to determine whether an audio event is an interference event or a speaker event.
• the system described may further detect faces or motion from images from a camera to determine a mute or unmute indication.
• US 2015/0195411 A1 describes a system for providing intelligent and automatic mute notification.
• the system provides mechanics for controlling false positive determinations of speaking through mute by utilizing a combination of recorded characteristics and initiated timers.
• the invention provides a method for notifying a user of a mute state of a microphone system during a call with one or more other participants, in case the user speaks while the primary microphone of the microphone system is muted, the method comprising
• VAD Voice Activity Detection
• the microphone system comprises a primary microphone arranged to capture the user's speech and an additional microphone located to capture sound from the user's surroundings, and wherein the method comprises performing a noise cancellation algorithm by processing output signals from the primary microphone and output signals from the additional microphone to suppress surrounding noise.
• the additional microphone is preferably not connected to transmit audio to the call at any time, since its role is to provide information to determine whether to mute the primary microphone or not or to determine whether or not to send mute notifications.
• the method is suitable for a headset arranged for connection to a computer, a tablet or smartphone or the like.
• the method can be used in various devices with a microphone intended for use in a call, and where the microphone has a mute function.
• Some parts of the method may advantageously be implemented on a processor in a headset or other device with a microphone and a loudspeaker, while other parts of the method may be implemented in components involved in facilitating the call, e.g. a computer or at a server providing an online call.
• a headset or other device may simply eliminate unintentional mute state notifications from the online call by muting the primary microphone.
• the invention is based on the insight that a significant number of unintended microphone mute notifications can be eliminated or reduced in an intelligent way by rather simple processing which can be implemented e.g. in a headset.
• VAD Voice Activity Detection
• VAD Voice Activity Detection
• it can be ensured that only speech in the muted microphone will trigger a mute state notification.
• an intelligent detection of the user's activity can be provided, so as to significantly reduce disturbing mute state notifications, e.g. if the user speaks to a colleague in the physical surroundings.
• the VAD algorithm finds that the user is speaking into the microphone, then an interrupt will be sent from the VAD algorithm, which will make the headset play e.g. a voice prompt saying, "Headset muted".
• the mute state notification may be audible as the headset user will be most likely to observe this form of notification.
• a frequency of the mute state notification should be configurable, e.g. configurable by the user, to avoid notifications being played too often if the headset user has a physical conversation.
• a separate VAD algorithm can be used for detection of speech captured by such additional microphones.
• a separate VAD algorithm can be applied to an incoming audio signal in the call, thus allowing detection of other participants speaking, and e.g. only allow mute state notification to the user in case none of the other participants in the call are speaking, thereby indicating that it may be the intention of the user to speak.
• Noise Cancellation with one or more additional microphones assists the VAD algorithm and improving the VAD algorithm in making the decision of the presence of speech.
• the additional microphones can be used for performing beamforming to determine if the user is facing a speech source in the user's surroundings. If the user is facing a surrounding speech source, then the user is most likely to have a physical conversation and a notification will not be sent.
• the additional microphones can also be used for beamforming to detect if the user's head is turned towards a speech source in the user's surroundings. If this is the case the user is most likely to have a physical conversation and a notification will not be sent.
• the additional microphones can also be used for estimating if the headset user is answering a question from a person in their surroundings. The surrounding microphones will detect speech in the user's surroundings and if the primary microphone detects speech from the user in a flow that is estimated to be an answer or contribution to a physical conversation, then a notification will not be sent.
• a noise cancellation algorithm in combination with one or more additional microphones has been found to improve the efficiency of the VAD algorithm(s) and thus helps to distinguish between background noise and speech. Still further, with the use of an additional microphone, the discrimination of the user being in a physical conversation with a person in the surroundings is significantly improved. Thereby, unintentional mute notifications can be significantly reduced or even eliminated.
• the method comprises determining if it is likely that determined speech comes from a speech source in the user's surroundings, and providing the mute state notification to the user only if it is not likely that determined speech comes from a speech source in the user's surroundings.
• the method may comprise processing output signals from a plurality of microphones to so as to allow discrimination between speech from the user and speech from the user's surroundings.
• the method may comprise processing the output signals from the plurality of microphones to provide a beamforming sensitivity pattern so as to allow discrimination between speech from the user and speech from the user's surroundings.
• the method comprises determining if it is likely that the user has a physical conversation, and providing the mute state notification to the user only if it is not likely that the user has a physical conversation.
• the method may comprise performing a first VAD algorithm on output signals from a microphone capturing the user's speech, such as a mouth microphone in a headset, and performing a second Voice Activity Detection algorithm on output signals from at least one additional microphone to determine speech from another source.
• the method may comprise determining a timing between speech from the user and speech from another source so as to determine if it is likely that the user has a physical conversation.
• the method may comprise performing a VAD algorithm on a signal indicative of sound from the at least one other participant in the call, so as to detect speech from the at least one other participant in the call.
• the method may comprise providing a mute state notification to the user only in case it is detected that the user speaks, while at the same time there is no speech detected from the at least one other participant in the call.
• the method may comprise performing a noise cancellation algorithm by processing output signals from a primary microphone, e.g. headset mouth microphone, and output signals from an additional microphone to suppress surrounding noise. This may help increasing performance of the VAD algorithm.
• a primary microphone e.g. headset mouth microphone
• a measure of frequency of mute state notifications can be set by the user.
• an even lower disturbance can be experienced, since the user can lower the frequency of notifications in case they are still found disturbing.
• a VAD algorithm detects the presence of speech in a signal. Preferably, features are extracted from the signal in time or frequency domain and used in a classification rule to determine if speech is present or not. While in a muted state, the microphone, e.g. in a headset, provides a real time signal to the VAD algorithm. Implementations of a VAD algorithm will be known by the skilled person.
• the method comprises the performing, by a first processor, the steps of: performing a noise cancellation on signals from the primary microphone and the additional microphone to suppress surrounding noise, processing an output signal from the microphone system according to a VAD algorithm, determining if speech is present, and determining if an additional condition is fulfilled, are performed by a first processor, such as a processor in a headset comprising the primary microphone, the additional microphone and a loudspeaker.
• the step of providing mute state notification is performed by a second processor, such as a processor in a computer device or computer system facilitating said call.
• the mentioned steps are utilized to determine whether to mute or transmit audio from the primary microphone to the second processor facilitating the call, namely to determine only to transmit audio from the primary microphone to the second processor facilitating the call, if it is determined that speech is present and the additional condition is fulfilled. In this way an unintended mute state notification is avoided, even though a traditional call system is used, since the normal mute state notification will not be triggered due to the muting of the primary microphone unless it is likely that the user intends to speak in the call.
• the method comprises performing a noise cancellation algorithm on the output signals from the primary microphone and from the additional microphone involving a VAD algorithm providing an output indicative of presence of speech, and generating a noise cancelled version of the output signal from the primary microphone based on said output indicative of presence of speech.
• the noise cancellation algorithm may comprise applying said output indicative of presence of speech to a noise estimator which estimates noise in the output signal from the primary microphone in periods without speech present.
• the noise cancellation algorithm may comprise multiplying a gain vector with a frequency domain representation with a set of frequency bins of the primary microphone signal, wherein the gain vector has been generated with low gain values for frequency bins not containing speech, preferably with high gain values for frequency bins containing speech.
• the noise cancellation algorithm may comprise generating the gain vector in response to an input from the noise estimator, thus the gain vector is preferably generated based on the noise estimate from the noise estimator.
• the noise estimate is improved, since it can be based on periods only where there is not speech present. This in turn allows a good suppression of noise in the signal from the primary microphone, and with such good noise suppression, it has been found that the VAD algorithm performed for determining mute state notification is improved.
• An alternative noise cancellation algorithm is based on generating a noise cancelled version of the output signal from the primary microphone by applying an adaptive noise cancellation algorithm involving an adaptive filter.
• the adaptive filter may be implemented by a Least Mean Square or a Normalized Least Mean Square algorithm, such as known by the skilled person.
• the invention provides a device arranged for two-way audio communication, such as in a wireless format, the device comprising a microphone system comprising a primary microphone and an additional microphone, and processor system arranged to perform all steps of the method according to the first aspect, or at least the steps of the method except providing the mute state notification.
• said processor system may be arranged to determine to mute the primary microphone in response to said additional condition, so as to provide an audio output from the primary microphone only in case it is determined to be likely that the user intends to speak in the call.
• the device determines to mute the primary microphone to avoid any audio being transmitted to the processor system facilitating the call, unless it is found likely that the user intends to speak in the call.
• the device may be a headset, such as with the processor system forming an integral part of the headset.
• the device preferably comprises a loudspeaker, so as to allow two-way audio communication.
• the device may e.g. be a standalone device with the microphone system and a loudspeaker in one unit with a wired (e.g. USB) or wireless connection (e.g. Bluetooth) to a computer or a smartphone or the like.
• a wired e.g. USB
• wireless connection e.g. Bluetooth
• the device may comprise a headset system arranged for two-way audio communication, such as in a wireless format, the headset system comprising
• the headset comprising a microphone system comprising a mouth microphone, an additional microphone positioned separate from the mouth microphone, and at least one ear cup with a loudspeaker,
• mute activation function which can be activated by the user to mute sound from the mouth microphone in a mute state during the call
• a processor system arranged to perform the method according to the first aspect, or at least the steps except the step of transmitting the mute state notification, so as to determine if it is appropriate to notify the user of a mute state, when the user speaks while the mouth microphone is in the mute state, or so as to determine whether to mute the mouth microphone when the user speaks while the mouth microphone is in the mute state.
• the processor system of the device is arranged to determine whether it is likely that the user intends to speak, and to transmit audio accordingly from the mouth microphone only in case it is determined to be likely that the user intends to speak, so as to avoid any mute state notification being sent by an entity, such as a processor system, facilitating the call.
• the microphone system may comprise more than one additional microphone positioned separate from the mouth microphone.
• the mouth microphone may be implemented as a plurality of separate microphones so as to allow beamforming for suppressing surrounding sound captured by the mouth microphone.
• one or several additional microphones may be located on one or both earcups of the headset to capture surrounding sounds, e.g. for active noise cancellation of sound reaching the ears of the user.
• an array of additional microphones are arranged for beamforming to allow capturing speech from a limited direction relative to the user only, and/or e.g. determine a direction from which the speech comes, so as to allow determining whether it is likely that the speech is intended for the user as part of a conversation with the user, of if such speech can be considered as speech unintended for the user.
• the processor system is arranged to provide the notification to the user as an audible notification via the loudspeaker, e.g. as a voice message.
• the mute function may be implemented as a user operable knob or push button or contact or other means located on a part of the headset.
• the processor system may be a processor as known in existing device, such as a headset.
• the invention is suited for easy implementation in device having a processor with extra capacity for performing the VAD algorithm etc.
• the necessary processing can be implemented also in a compact headset, however if preferred, the processing system may be implemented on a computer or smartphone or a dedicated device separate from the headset.
• the invention provides a communication system comprising
• a communication device arranged to provide a two-way call via a communication channel and to provide two-way audio to the at least one device according to the first aspect accordingly, e.g. in a digital wireless format such as DECT, or Bluetooth or other similar short range wireless formats.
• a digital wireless format such as DECT, or Bluetooth or other similar short range wireless formats.
• the communication device may comprise a computer or a mobile phone such as a smartphone.
• the communication channel may be such as a mobile network e.g. 2G, 3G, 4G, 5G or the like, the internet, or a dedicated wired or wireless communication channel.
• the connection between the communication device and the communication channel may be a wired or a wireless connection, e.g. the connection may involve a wi-fi connection.
• the communication system may be such as a teleconference system or the like.
• the invention provides use of the method according to the first aspect for performing one or more of: a telephone call, an on-line call, and a teleconference call.
• the invention provides use of the device according to the second aspect for performing one or more of: a telephone call, an on-line call, and a teleconference call.
• the invention provides user of the system according to the third aspect for performing one or more of: a telephone call, an on-line call, and a teleconference call.
• the invention provides a program code arranged to cause the method according to the first aspect to be performed, when the program code is executed on a processor or on two separate processors.
• the program code may be stored in memory on a chip, or on one or more tangible storage media, or available on the internet in a version for downloading.
• the program code may be in a general code format or in a processor dedicated format.
• FIG. 1 illustrates the situation where a headset user in an online call with call participants while being and present in a physical room with another person who speaks to the headset user during the call
• FIG. 2 illustrates steps of a method embodiment
• FIG. 3 illustrates a block diagram with elements of an embodiment
• FIG. 4 illustrates a headset system embodiment
• FIG. 5 illustrates a block diagram of elements of an embodiment with noise cancellation provided on both the primary microphone (mouth microphone) and an additional microphone prior to providing the signals from these microphones to VAD algorithms
• FIG 6 illustrates a headset system embodiment with an additional microphones placed on the earcup and with a processor determining to transmit an audio output from the primary microphone (mouth microphone) only if it is determined that it is likely that the user intends to speak in an ongoing call,
• FIG. 7 illustrates a block diagram of an example of a noise cancellation algorithm example generating a noise cancelled version of the audio signal from the primary microphone based on audio inputs from the primary microphone and an additional microphone, and
• FIG. 8 illustrates a block diagram of another example of a noise cancellation algorithm based on adaptive noise cancellation.
• FIG. 1 shows the basic situation behind the invention, namely a user U present in a physical room RM with another person P, e.g. a colleague.
• the user U is in a call CL with other call participants CL_P, e.g. an online meeting via a computer or the like.
• the user U wears a headset for two-way communication with the call participants CL_P. If the user U has muted the headset microphone for some reason, and noise or speech is captured by the mouth microphone of the headset, a mute state notification is provided to the user U either a visible message on a display or an audible message via the loudspeaker in the headset. However, such notification is unintended and disturbing for the user U e.g. in case the sound captured is speech from the person P in the room RM and/or speech by the user U in a conversation with the person P in the room RM.
• VAD Voice Activity Detection
• FIG. 2 illustrates steps of a method embodiment, i.e. a method for notifying a user of a mute state of a microphone system during a call with one or more other participants, in case the user speaks while the microphone system is muted.
• the method comprises performing an environmental noise cancellation algorithm ENC by processing output signals from a primary microphone, e.g. headset mouth microphone, and output signals from an additional microphone to suppress surrounding noise from the environments where the user is located.
• the method comprises processing VAD an output signal from the microphone system, at least the primary microphone, optionally both the primary microphone and the additional microphone(s) according to a VAD algorithm by means of a processor system while the microphone system is muted.
• the steps ENC, VAD, S_D, D_AC are performed by a first processor in a first device such as a headset, while step P_MSN is performed by a second processor in a second device such as a computer executing a call with a distal participant. In some embodiments, all five mentioned steps are performed by a processor in one device.
• the additional condition may be based on one or more separate VAD algorithms operating on additional microphones arranged to determine if speech is present in the environments around the user, and/or a separate VAD algorithm operating on incoming audio from the call to determine if other participants are speaking. This can be helpful in providing information important for determining the actual situation the user is in and thus determine if it is appropriate to provide a mute state notification or not.
• the performance of the VAD algorithm, or VAD algorithms is/are improved.
• FIG. 3 shows a block diagram to illustrate a part of a headset embodiment.
• a determining algorithm D_A determines whether to send a mute state notification MT_N to a user when certain conditions are met, and in case the user's mouth microphone MM is in a mute state MT, i.e. blocking sound from the user during an ongoing call.
• a first VAD algorithm VAD1 operates on the signal from the mouth microphone MM of the headset and determines a first input to the determining algorithm D_A, namely if speech is present.
• a second VAD algorithm VAD2 operates on an input from one or more microphones arranged to capture sound from the environments around the user, e.g. one or several microphones positioned on an exterior part of the headset, and it is then provided to the determining algorithm D_A if speech is present in the environments.
• a third VAD algorithm VAD3 operates on the sound input from the call CS, thus the third VAD algorithm serves to determine if the other participants in the call speak or are silent.
• the determining algorithm D_A thus has two inputs from the VAD2, VAD3 in addition to the input from VAD1 that the user can be assumed to speak.
• the input from VAD2 can be used to determine if a person in the environments speaks, while the use speaks, which most likely means that the user may be in a conversation with the person present in the environments and thus not intends to speak to participants in the call, and thus a mute state notification MT_N should in such case be avoided. Further, when it is detected that the user speaks, and the call sound CS indicates that the other participants do not speak, then it is likely that the user wants to speak in the call, and thus it is appropriate to provide a mute state notification MT_N.
• FIG. 4 illustrates a headset system embodiment with a headset HS to be worn by a user during a call, and it has a primary microphone in the form of a mouth microphone MM to capture the user's voice, and two earcups each with a loudspeaker to provide audio to the user from the call CL.
• the mouth microphone MM and loudspeakers of the headset HS are connected to a processor P, e.g. integrated into one or both earcups of the headset HS.
• the processor P handles two-way audio communication in connection with a call CL, such as in a wireless format.
• the headset HS has a mute activation function MT which can be activated by the user to mute sound from the mouth microphone MM in a mute state MT during the call CL.
• the mute state MT is provided as input to the processor P which determines to provide a mute state notification MT_N to the user only when it is appropriate according to a method as described in the foregoing, when it is detected by means of a VAD algorithm, that the user speaks while the mouth microphone MM is in the mute state MT.
• the headset system embodiment shown is arranged for wired or wireless communication of two-way audio call CL to a communication device serving to provide the call connection via a communication channel.
• the headset system simply itself mutes the primary microphone when it is found likely that the user intends that the primary microphone should be muted.
• the headset system simply itself mutes the primary microphone when it is found likely that the user intends that the primary microphone should be muted.
• such embodiments are compatible with existing communication devices or computer programs serving to provide the call connection via a communication channel, since such devices or programs will only be prompted to send a mute notification in case the headset system has passed sound which is likely to be the user's speech which is intended for the call, and thus the mute notification of the devices or programs will function as intended, i.e. with an improved quality compared to using a standard headset system.
• the processing and mute notification decision can, in other embodiments, be entirely performed by the device or program facilitating the call.
• Context awareness by beamforming Use of additional microphones placed on the headset to act as a microphone array. Beamforming techniques are then used to directionally locate a person, e.g. a colleague, speaking in the environments of the user. If the person is detected within a certain angle of acceptance, the method may be arranged to find the context likely to be a conversation with the person, thus it will be determined that a mute state notification should not be provided.
• a beamforming setup can also be used to detect if the user points his/her attention towards the person. This is done by using beamforming to detect if the user turns his/her head towards the person speaking. When the person starts speaking the headset detects the person at a certain angle. When the user turns his/her head towards the person, the headset will detect the person at another angle, and thus it may be determined that a conversation is a likely context, and thus a mute state notification should not be provided.
• Noise cancellation algorithm to optimize VAD performance E.g. an environmental noise cancellation (ENC) algorithm may use the input of the primary microphone (e.g. mouth microphone) and one or more separate microphones to filter out surrounding noise.
• EEC environmental noise cancellation
• the VAD algorithm will not be affected as much by surrounding noise, thus the present invention will decrease the risk of an environmental sound falsely activating the mute state notification.
• a primary microphone e.g. mouth microphone
• a secondary microphone or microphones
• VAD algorithm detects if speech is present to let the headset know when the user is speaking and when someone is speaking in the user's surroundings.
• a model may be used to estimate the likelihood that the speech captured at the two microphones are part of the same conversation. This estimate can then be used to determine a mute state notification should be provided.
• Call activity context awareness Using two separate running VAD algorithms when the user is in a call, where one VAD algorithm detects speech in the signal from the primary microphone (e.g. mouth microphone). The other VAD algorithm detects speech by processing incoming audio from the call to determine call activity, i.e. speech activity in the call. Presence of speech in the call activity is used to estimate the likelihood that the user unintendedly is speaking into a muted microphone. If speech is not detected in the call activity and the user speaks into a muted microphone, it is estimated likely that the call participants are waiting for the user to contribute, thus a mute state notification is provided. If speech is detected in the call activity and the user speaks into a muted microphone, it is estimated less likely that the call participants are waiting for the user to contribute, thus a mute state notification is not provided in such case.
• the primary microphone e.g. mouth microphone
• the other VAD algorithm detects speech by processing incoming audio from the call to determine call activity, i.e. speech activity in the call. Presence of speech
• Fig. 5 shows a block diagram to illustrate a part of a headset embodiment with a mouth microphone MM as primary microphone and an additional microphone M2.
• a determining algorithm D_A determines whether to mute audio from the mouth microphone MM or to pass audio from the mouth microphone MM to an audio output A_0 depending on whether certain conditions are met.
• the audio outputs from the mouth microphone MM and the additional microphone M2 are both processed by a noise cancellation algorithm NC to cancel possible noise in the audio output from the mouth microphone MM, and a noise suppressed audio signal from the mouth microphone MM is then provided as input to a VAD algorithm VAD1.
• the audio output from the additional microphone M2 is processed by a separate VAD algorithm VAD2. It is to be understood that separate noise calculation algorithms may alternatively be provided to outputs from the two microphones MM, M2, if preferred.
• Each of the VAD algorithms VAD1, VAD2 provide results which are provided as inputs to the determining algorithm D_A, namely an algorithm determining whether speech is present at the two microphones MM, M2, respectively.
• these inputs may especially be used to determine, if it is likely that the user speaks with a person in the environments, i.e. performs a physical conversation with another person.
• the determining algorithm D_A determines to mute audio from the mouth microphone, while providing speech from the mouth microphone at the audio output A_0 in case it is detected that the user speaks, based on VAD1, while VAD2 indicates, over a period of time, that there is no additional speech in the surroundings.
• Fig. 6 shows a variant of the headset system (dashed box) of Fig. 4.
• the headset HS has a primary microphone, here shown as a mouth microphone MM, and an additional microphone AM to capture environmental sounds, here shown placed on an earcup of the headset HS.
• a processor system PI e.g. implemented integral with one of the earcups of the headset HS, is arranged to perform a noise cancellation algorithm by processing output signals from the mouth microphone MM and the additional microphone AM so as to suppress surrounding noise. Further, the processor system PI is arranged to process an output from the mouth microphone MM according to a VAD, optionally also performing an output from the additional microphone according to a separate VAD algorithm, e.g. as in Fig. 5.
• the processor system PI is arranged to determine if speech is present in accordance with an output of the VAD performed on the output from the mouth microphone MM, and further determining if an additional condition is met.
• the processor system PI is arranged to generate an audio output A_0 from the mouth microphone MM only in case it is determined that the mouth microphone MM captures speech and that the additional condition is met.
• the additional condition may be that it is determined to be likely that the user speaks, and that the user is not at the same time involved in a physical conversation with a person in the surroundings.
• the determination of the additional condition may be based on processing sound captured by the additional microphone AM.
• a separate processor system P2 facilitates the call and thus provides two-way audio connectivity to call participants CL_P.
• This processor system P2 may comprise a personal computer, a laptop, a tablet or a smartphone, or a dedicated device, serves to process the audio output A_0 from the headset system and to generate an audio input A_I with audio from distal participants CL_P in the call to the headset system.
• a more intelligent mute notification MT_N is obtained, since the separate processor system P2 provides the mute notification MT_N in the traditional way, as known from existing call systems, e.g. when the audio level in the audio output A_0 exceeds a certain level when in the mute state.
• the notification MT_N is .e.g. as a visual notification and/or an audible notification.
• the processor system PI in the headset system serves to provide an intelligent muting of the mouth microphone MM, it is ensured that the audio output A_0 to the separate processor system P2 is provided only, when the headset system has determined that it is likely that the user intends to speak in the ongoing call, thus eliminating annoying mute state notifications MT_N even with existing call systems.
• FIG. 7 illustrates an example of a noise cancellation algorithm for processing audio signals A_MM from a primary microphone and audio signals A_M2 from an additional microphone to generate a noise cancelled audio signal A_MM_NC from the primary microphone.
• the algorithm operates on frequency domain representations X, X2 of the respective audio input signals A_MM, A_M2.
• a gain vector G is multiplied with the frequency representation of the primary microphone audio signal X.
• the gain vector G is generated such that low gains are set on frequency bins of the frequency representation of the primary microphone signal X not containing speech.
• the resulting output Y of the multiplication of X and G is then transformed to a time signal A_MM_NC which represents the noise cancelled version of the original audio signal from the primary microphone A_MM.
• the block diagram of FIG. 7 illustrates initial short time analyses STA performed on the respective audio signals A_MM, A_M2 and based thereon, the two audio signals A_MM, A_M2 are transform into respective frequency domain representations X, X2.
• X is applied to a noise estimator NE which estimates noise N, and finally a gain estimator GE generates the gain vector G based on the estimated noise N and X.
• the noise estimator NE receives an input V from a Voice Activity Detector VAD operating with both X and X2 as inputs, and the input V indicates to the noise estimator NE when there is speech or not, and the noise estimator NE then updates its noise estimate N in periods where there is not speech.
• VAD Voice Activity Detector
• FIG. 8 illustrates a block diagram of another example of a noise cancellation algorithm based on simple adaptive noise cancellation. This algorithm is based on the assumption that the audio signal from the primary microphone x contains the intended speech as well as noise, and that the audio signal x2 from the additional microphone contains the same noise, which may not be completely valid in practice due to the two microphones being positioned at different locations.
• the objective for the adaptive noise canceller is to minimize the output power z. This is achieved using the output signal as the error signal e in an adaptive filter AF. It can be proven that the smallest possible output power is achieved when y equals the noise, meaning the output signal z equals the desired signal x.
• NLMS normalized least mean square
• LMS least mean square
• NLMS normalizes the power of the input and uses a time-varying step size to converge faster.
• noise cancellation examples described merely serve to illustrate that noise cancellation to suppress noise of the audio signal from the primary microphone can be implemented in various ways.
• the effect of improving the reliability of the VAD performed on the noise cancelled primary microphone signal can be obtained with various implementations.
• additional embodiments E1-E15 will be defined.
• VAD Voice Activity Detection
• S_D determining if speech is present in accordance with an output of the Voice Activity Detection algorithm
• the method according to El comprising determining if it is likely that determined speech comes from a speech source in the user's surroundings, and providing the mute state notification to the user only if it is not likely that determined speech comes from a speech source in the user's surroundings.
• E3. comprising processing output signals from a plurality of microphones to so as to allow discrimination between speech from the user and speech from the user's surroundings.
• E4. The method according to E3, processing the output signals from the plurality of microphones to provide a beamforming sensitivity pattern so as to allow discrimination between speech from the user and speech from the user's surroundings.
• E5. The method according to any of E1-E4, comprising determining if it is likely that the user has a physical conversation, and providing the mute state notification to the user only if it is not likely that the user has a physical conversation. E6.
• the method according to E5 comprising performing a first Voice Activity Detection algorithm on output signals from a microphone capturing the user's speech, such as a mouth microphone, and performing a second Voice Activity Detection algorithm on output signals from at least one additional microphone to determine speech from another source.
• E7 The method according to E5 or E6, comprising determining a timing between speech from the user and speech from another source so as to determine if it is likely that the user has a physical conversation.
• E8 The method according to any of E1-E7, comprising performing a Voice Activity Detection algorithm on a signal indicative of sound from the at least one other participant in the call, so as to detect speech from the at least one other participant in the call.
• E9 The method according to E8, providing a mute state notification to the user only in case it is detected that the user speaks, while at the same time there is no speech detected from the at least one other participant in the call.
• E10 The method according to any of E1-E9, comprising performing a noise cancellation algorithm (ENC) by processing output signals from a primary microphone, e.g. headset mouth microphone, and output signals from an additional microphone to suppress surrounding noise.
• EEC noise cancellation algorithm
• the device according to Ell comprising a headset system arranged for two- way audio communication, such as in a wireless format, the headset system comprising
• a headset arranged to be worn by the user, the headset (HS) comprising a microphone system comprising at least a mouth microphone (MM) and at least one ear cup with a loudspeaker, - a mute activation function (MT) which can be activated by the user to mute sound from the mouth microphone (MM) in a mute state during the call, and
• a microphone system comprising at least a mouth microphone (MM) and at least one ear cup with a loudspeaker
• MT mute activation function
• processor system arranged to perform the method according to any of E1-E10 so as to determine if it is appropriate to notify the user of a mute state, when the user speaks while the mouth microphone (MM) is in the mute state.
• E14 The device according to E12 or E13, wherein the processor system (P) is arranged to provide the notification to the user as an audible notification via the loudspeaker.
• E15 Use of the method according to any of E1-E10 for performing one or more of: a telephone call, an on-line call, and a tele conference call.
• the invention provides a method and device, e.g. a headset, for notifying a user of a mute state of a primary microphone during a call, in case the user speaks while the primary microphone is muted.
• the method comprises performing a noise cancellation algorithm (ENC) on output signals from the primary microphone and on output signals from an additional microphone capturing sound in the user's surroundings to suppress surrounding noise at the user location. Further processing output signals from the primary microphone according to a Voice Activity Detection (VAD) algorithm by means of a processor system while the primary microphone is muted.
• the VAD algorithm is used to determine if speech is present, and next it is determined if an additional condition if fulfilled.

Landscapes

• Engineering & Computer Science (AREA)
• Signal Processing (AREA)
• Multimedia (AREA)
• Physics & Mathematics (AREA)
• Acoustics & Sound (AREA)
• Health & Medical Sciences (AREA)
• Audiology, Speech & Language Pathology (AREA)
• Human Computer Interaction (AREA)
• Computational Linguistics (AREA)
• Theoretical Computer Science (AREA)
• Quality & Reliability (AREA)
• General Health & Medical Sciences (AREA)
• General Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• Otolaryngology (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Telephone Function (AREA)
• Circuit For Audible Band Transducer (AREA)
• Soundproofing, Sound Blocking, And Sound Damping (AREA)
• Alarm Systems (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=75539153

Family Applications (1)

Country Status (5)

Families Citing this family (4)

Citations (8)

Family Cites Families (4)

• 2021
  • 2021-04-15 EP EP21168551.6A patent/EP4075822B1/en active Active
• 2022
  • 2022-03-24 EP EP22717610.4A patent/EP4324215A1/en active Pending
  • 2022-03-24 US US18/555,018 patent/US20240195916A1/en active Pending
  • 2022-03-24 WO PCT/EP2022/057830 patent/WO2022218673A1/en not_active Ceased
  • 2022-03-24 JP JP2023562986A patent/JP7759401B2/ja active Active
  • 2022-03-24 AU AU2022259391A patent/AU2022259391A1/en active Pending

Patent Citations (8)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events