EP3945729A1 - System and method for headphone equalization and space adaptation for binaural reproduction in augmented reality - Google Patents

Abstract

A system is provided. The system includes an analyzer (152) for determining a plurality of binaural room impulse responses and a loudspeaker signal generator (154) for generating at least two loudspeaker signals dependent on the plurality of binaural room impulse responses and dependent on the audio source signal from at least one audio source. The analyzer (152) is designed to determine the plurality of binaural spatial impulse responses such that each of the plurality of binaural spatial impulse responses takes into account an effect resulting from the wearing of headphones by a user.

Description

The present invention relates to headphone equalization and spatial adaptation of binaural playback in augmented reality (AR).

Selective Hearing (SH) refers to the listener's ability to focus their attention on a particular sound source or on a plurality of sound sources in their auditory scene. This in turn implies that the listener's focus on uninteresting sources is reduced.

In this way, human listeners are able to communicate even in noisy environments. As a rule, various aspects are exploited: When listening with two ears, there are direction-dependent time and level differences and a direction-dependent different spectral coloring of the sound. With the latter, the ear is already able to determine the direction of a sound source when listening with one ear and thus separate different sound sources.

Time and level differences alone are not sufficient to determine the exact position of a sound source: The locations with the same time and level difference are on a hyperboloid. The resulting ambiguity in localization is called cone-of-confusion. In rooms, every sound source is reflected by the boundary surfaces. Each of these so-called mirror sources lies on another hyperboloid. The human sense of hearing combines the information about the direct sound and the associated reflections into one auditory event and thus resolves the ambiguity of the cone-of-confusion. At the same time, the reflections associated with a sound source increase the perceived loudness of the sound source.

Furthermore, in the case of natural sound sources, such as speech in particular, the signal components of different frequencies are coupled in time. Binaural hearing uses all of these aspects together. Furthermore, loud sources of interference that are easy to localize can be actively ignored.

The concept of selective listening is related in the literature to other terms such as assisted listening [1], virtual and amplified auditory environments [2]. Assisted hearing is an umbrella term that includes virtual, augmented, and SH applications.

According to the state of the art, classic hearing aids usually work monaurally, i.e. the signal processing for the right and left ear is completely independent in terms of frequency response and dynamic compression. As a result, time, level and frequency differences between the ear signals are lost.

Modern, so-called binaural hearing aids couple the correction factors of the two hearing aids. They often have several microphones, but usually only the microphone with the "most speech-like" signal is selected, but no explicit beamforming is calculated. In complex listening situations, desired and undesired sound signals are amplified in the same way, and concentration on desired sound components is therefore not encouraged.

In the area of hands-free systems, e.g. for telephones, several microphones are already used today and so-called beams are calculated from the individual microphone signals: Sound coming from the direction of the beam is amplified, sound from other directions is reduced. Today's methods learn the constant background noise (e.g. engine and wind noise in a car), learn loud disturbances that can be easily localized by another beam and subtract them from the useful signal (example: Generalized Sidelobe Canceler). Recognizers are sometimes used in telephony systems that recognize the static properties of speech and everything that is not structured like speech is suppressed. In the case of hands-free devices, however, only a mono signal is transmitted at the end, the spatial information, which is quite interesting for capturing the situation and in particular for creating the illusion that "one is there", especially when several speakers are on the phone together, is lost on the transmission path. By suppressing non-speech signals, important information about the acoustic environment of the interlocutor is lost, which can impede communication.

Humans are naturally able to "hear selectively" and consciously focus on individual sound sources in their environment. An automatic system for selective listening using artificial intelligence (AI) must understand the concepts behind it first learn. The automatic decomposition of acoustic scenes (Scene Decomposition) first requires a detection and classification of all active sound sources, followed by a separation in order to be able to further process, amplify or attenuate them as separate audio objects.

In the Auditory Scene Analysis research field, an attempt is made to use a recorded audio signal to detect and classify both time-localized sound events such as footsteps, clapping or screams and more global acoustic scenes such as concerts, restaurants or supermarkets. Current methods only use methods from the field of artificial intelligence (AI) and deep learning. This involves data-driven learning of deep neural networks, which learn to recognize characteristic patterns in the audio signal on the basis of large amounts of training [70]. Primarily inspired by advances in the research areas of image processing (computer vision) and language processing (natural language processing), mixtures of convolutional networks (convolutional neural networks) for two-dimensional pattern recognition in spectrogram representations and recurrent layers (recurrent neural networks) for temporal modeling of sounds used.

Audio analysis has a number of specific challenges that need to be addressed. Due to their complexity, deep learning models are very data-hungry. Compared to the research areas of image processing and speech processing, only relatively small data sets are currently available for audio processing. The largest data set is the AudioSet data set from Google [83] with around 2 million sound samples and 632 different sound event classes, although most of the data sets used in research are much smaller. This small amount of training data can be addressed, for example, with transfer learning (transfer learning), in which a model pre-trained on a large data set is then fine-tuned to a smaller data set with new classes intended for the application (fine tuning) [77 ]. Furthermore, methods from partially supervised learning (semi-supervised learning) are used in order to include the generally available large amount of non-annotated audio data in the training.

Another essential difference to image processing is that when acoustic events are audible at the same time, there is no masking of sound objects (as in the case of images) but to a complex phase-dependent superimposition. Current algorithms in deep learning use so-called "attention" mechanisms that enable the models, for example, to focus on certain time segments or frequency ranges during classification [23]. The recognition of sound events is further complicated by the high variance in their duration. Algorithms should robustly recognize both very short events such as a pistol shot and long events such as a passing train.

Due to the strong dependency of the models on the acoustic conditions when recording the training data, they often show unexpected behavior in new acoustic environments that differ, for example, in room reverberation or miking. Various approaches have been developed to alleviate this problem. Using data augmentation methods, attempts are made, for example, to achieve greater robustness and invariance of the models by simulating various acoustic conditions [68] and also artificially superimposing various sound sources. Furthermore, the parameters in complex neural networks can be regularized differently, so that overtraining and specialization on the training data is prevented and at the same time a better generalization on unseen data is achieved. In recent years, various algorithms for "domain adaptation" [67] have been proposed in order to adapt models that have already been trained to new application conditions. Real-time capability of the sound source detection algorithms is of elementary importance in the scenario of use planned in this project within a headphone. In this case, a trade-off between the complexity of the neural network and the maximum possible number of arithmetic operations on the underlying computing platform must be carried out. Even if a sound event has a longer duration, it still has to be recognized as quickly as possible in order to start an appropriate source separation.

In recent years, a large number of research projects have been carried out at the Fraunhofer IDMT in the field of automatic sound source detection. In the "StadtLärm" research project, a distributed sensor network was developed that can use recorded audio signals to measure noise levels at various locations within a city and to classify between 14 different acoustic scene and event classes [69]. The processing in the sensors on the embedded platform Raspberry Pi 3 takes place in real time. In a preparatory work investigated novel approaches for data compression of spectrograms based on autoencoder networks [71]. The application of methods from deep learning in the field of music signal processing (music information retrieval) has recently been made in applications such as music transcription [76], [77], chord recognition [78] and instrument recognition [79]. In the field of industrial audio processing, new data sets were established and deep learning methods were used, e.g. for acoustic condition monitoring of electric motors [75].

In the scenario addressed in this exemplary embodiment, several sound sources must be assumed, the number and type of which is initially unknown and can change constantly. For sound source separation, multiple sources with similar characteristics, such as multiple speakers, pose a major challenge [80].

In order to achieve high spatial resolution, multiple microphones must be used in the form of an array [72]. In contrast to the usual audio recordings in mono (1 channel) or stereo (2 channels), such a recording scenario allows an exact localization of the sound sources around the listener.

Source separation algorithms usually leave behind artifacts such as distortion and crosstalk between the sources [5], which are generally perceived as annoying by the listener. However, such artefacts can be partially masked and thus reduced by mixing the tracks again (re-mixing) [10].

To improve the "blind" source separation (Blind Source Separation), additional information such as the detected number and type of sources or their estimated spatial position is often used (Informed Source Separation [74]). For meetings in which several speakers are active, current analysis systems can simultaneously estimate the number of speakers, determine their respective temporal activity and then isolate them via source separation [66].

In recent years, many studies on the perception-based evaluation of sound source separation algorithms have been carried out at Fraunhofer IDMT. [73]
In the field of music signal processing, a real-time capable algorithm for separating the solo instrument and the accompanying instruments was developed, which uses an estimate of the fundamental frequency of the solo instrument as additional information [81]. A An alternative approach to separating vocals from complex pieces of music based on deep learning methods was presented in [82]. Specialized source separation algorithms have also been developed for use in industrial audio analysis [7].

Headphones have a significant influence on the acoustic perception of the environment. Depending on the design of the headphones, the sound incidence on the way to the ears is attenuated to different degrees. In-ear headphones completely block the ear canals [85]. The closed headphones enclosing the auricle also acoustically cut the listener off from the outside environment. Open and semi-open headphones, on the other hand, still let sound through completely or partially [84]. In many everyday applications, it is desirable for headphones to seal off unwanted ambient noise more than their design allows.

With Active Noise Control (ANC), disturbing external influences can also be dampened. This is achieved by recording incoming sound signals from the microphones in the headphones and reproducing them from the loudspeakers in such a way that these sound components are canceled out by interference with the sound components penetrating the headphones. Overall, a strong acoustic insulation from the environment can be achieved in this way. However, this harbors dangers in numerous everyday situations, which is why there is a desire to switch this function intelligently if required.

The first products allow the microphone signals to also be passed through to the headphones in order to reduce passive isolation. In addition to prototypes [86], there are already products that advertise the "transparent hearing" function. For example, Sennheiser offers the function with the AMBEO headset [88] and Bragi in the product "The Dash Pro". However, this option is just the beginning. In the future, this function is to be greatly expanded so that not only can the full ambient noise be switched on or off, but individual signal components (such as only speech or alarm signals) can be made exclusively audible if required. The French company Orosound allows the wearer of the "Tilde Earphones" [89] headset to adjust the strength of the ANC with a slider. In addition, the voice of a conversation partner can also be passed through during activated ANCs. However, this only works if the interlocutor is in a 60° cone in front of you. A direction-independent adjustment is not possible.

In the disclosure document U.S. 2015 195641 A1 (see [91]) a method was disclosed which is designed to generate a listening environment for a user. The method includes receiving a signal representing an ambient listening environment of the user, further processing the signal using a microprocessor to identify at least one of a plurality of sound types in the ambient listening environment. The method further includes receiving user preferences for each of the plurality of sound types, modifying the signal for each sound type in the ambient listening environment, and outputting the modified signal to at least one speaker to create a listening environment for the user.

A major problem is headphone equalization and room adaptation of binaural playback in augmented reality (AR):
In a typical scenario, a human listener wears acoustically (partially) transparent headphones and hears his surroundings through them. In addition, additional sound sources are played back via the headphones, which are embedded in the real environment in such a way that it is not possible for the listener to distinguish between the real sound scene and the additional sound.

As a rule, tracking is used to determine in which direction the head is turned and where the listener is in the room (six degrees of freedom (6DoF)). It is known from research that good results (i.e. externalization and correct localization) are achieved when the room acoustics of the recording and playback rooms match or when the recording is adapted to the playback room.

An exemplary solution can be implemented as follows:
In a first step, the BRIR is measured without headphones, either individually or with an artificial head using a probe microphone.

In a second step, the spatial properties of the recording room are analyzed based on the measured BRIR.

In a third step, the headphone transfer function is then measured individually or with an artificial head using a probe microphone at the same location. This determines an equalization function.

Optionally, in a fourth step, the room properties of the playback room can be measured, the acoustic properties of the playback room can be analyzed and the BRIR can be adapted to the playback room.

Then, in a further step, a source to be augmented is convolved with the position-correct, optionally adjusted, BRIR in order to obtain two raw channels. Convolve the raw channels with the equalization function to get the headphone signals.

Finally, in a further step, the headphone signals are reproduced via headphones.

However, there is a problem that when the headphone is put on, the influence of the pinna on the BRIR disappears. I.e. the BRIRs are different than without headphones. As a result, natural sound sources sound different than without headphones, but the virtual augmented sound sources are reproduced as if no headphones were present.

It would be desirable for concepts to be provided that enable the room properties of the reproduction room to be determined simply, quickly and efficiently.

Embodiments of the invention are provided below.

Thus, claim 1 provides a system, claim 19 a method and claim 20 a computer program according to embodiments of the invention
A system according to an embodiment of the invention comprises an analyzer for determining a plurality of binaural room impulse responses and a loudspeaker signal generator for generating at least two loudspeaker signals dependent on the plurality of binaural room impulse responses and dependent on the audio source signal from at least one audio source. The analyzer is designed to determine the plurality of binaural spatial impulse responses in such a way that each of the plurality of binaural spatial impulse responses takes into account an effect resulting from the wearing of headphones by a user.

• Bestimmung einer Mehrzahl von binauralen Raumimpulsantworten. Und:
• Erzeugung von wenigstens zwei Lautsprechersignalen abhängig von der Mehrzahl der binauralen Raumimpulsantworten und abhängig von dem Audioquellsignal von wenigstens einer Audioquelle.

There is also provided a method according to an embodiment of the invention, the method comprising:

• Determination of a plurality of binaural spatial impulse responses. And:
• generating at least two speaker signals dependent on the plurality of binaural room impulse responses and dependent on the audio source signal from at least one audio source.

The plurality of binaural spatial impulse responses are determined such that each of the plurality of binaural spatial impulse responses takes into account an effect resulting from a user wearing a headphone.

Furthermore, a computer program according to an embodiment of the invention is provided with a program code for carrying out the method described above.

Preferred embodiments of the invention are described below with reference to the drawings.

Fig. 1

zeigt ein System gemäß einer Ausführungsform.

Fig. 2

zeigt ein weiteres System zur Unterstützung von selektivem Hören gemäß einer weiteren Ausführungsform.

Fig. 3

zeigt ein System zur Unterstützung von selektivem Hören, das zusätzlich eine Benutzeroberfläche umfasst.

Fig. 4

zeigt ein System zur Unterstützung von selektivem Hören, dass ein Hörgerät mit zwei entsprechenden Lautsprechern umfasst.

Fig. 5a

zeigt ein System zur Unterstützung von selektivem Hören, das eine Gehäusestruktur und zwei Lausprecher umfasst.

Fig. 5b

zeigt ein System zur Unterstützung von selektivem Hören, das einen Kopfhörer mit zwei Lautsprechern umfasst.

Fig. 6

zeigt ein System gemäß einer Ausführungsform, das ein entferntes Gerät 190 umfasst, das den Detektor und den Positionsbestimmer und den Audiotyp-Klassifikator und den Signalanteil-Modifizierer und den Signalgenerator umfasst.

Fig. 7

zeigt ein System gemäß einer Ausführungsform, das fünf Sub-Systeme umfasst.

Fig. 8

stellt ein entsprechendes Szenario gemäß einem Ausführungsbeispiel dar.

Fig. 9

stellt ein Szenario gemäß einer Ausführungsform mit vier externen Schallquellen dar.

Fig. 10

stellt einen Verarbeitungsworkflow einer SH-Anwendung gemäß einer Ausführungsform dar.

In the drawings is shown:

1

Figure 1 shows a system according to one embodiment.

2

FIG. 12 shows another system for supporting selective hearing according to another embodiment.

3

shows a system for supporting selective hearing, which additionally includes a user interface.

4

shows a system for supporting selective hearing that includes a hearing aid with two corresponding loudspeakers.

Figure 5a

shows a selective hearing support system that includes a housing structure and two speakers.

Figure 5b

shows a selective listening support system that includes a headset with two speakers.

6

12 shows a system according to an embodiment, including a remote device 190, which includes the detector and the position locator and the audio type classifier and the signal component modifier and the signal generator.

7

Figure 12 shows a system according to an embodiment comprising five sub-systems.

8

represents a corresponding scenario according to an exemplary embodiment.

9

12 illustrates a scenario according to an embodiment with four external sound sources.

10

Figure 12 illustrates a processing workflow of an SH application according to one embodiment.

1 Figure 1 shows a system according to one embodiment.

The system includes an analyzer 152 for determining a plurality of binaural spatial impulse responses.

Furthermore, the system comprises a loudspeaker signal generator 154 for generating at least two loudspeaker signals depending on the plurality of binaural room impulse responses and depending on the audio source signal from at least one audio source.

The analyzer 152 is configured to determine the plurality of binaural spatial impulse responses such that each of the plurality of binaural spatial impulse responses takes into account an effect resulting from a user wearing headphones.

In one embodiment, the system can include the headphones, for example, it being possible for the headphones to be designed, for example, to output the at least two loudspeaker signals.

According to one embodiment, the headphone can comprise e.g. two headphone capsules and e.g. at least one microphone for measuring sound in each of the two headphone capsules, wherein in each of the two headphone capsules e.g. The analyzer 152 can be designed, for example, to carry out the determination of the plurality of binaural room impulse responses using the measurement of the at least one microphone in each of the two headphone capsules. Headphones intended for binaural playback always have at least two headphone capsules (one each for the left and right ear), although more than two capsules (e.g. for different frequency ranges) can also be provided.

In one embodiment, the at least one microphone in each of the two headphone capsules can be configured, for example, to generate one or more recordings of a sound situation in a playback room before the start of playback of the at least two loudspeaker signals through the headphones, from the one or more recordings an estimate of a To determine the raw audio signal of at least one audio source and to determine a binaural room impulse response of the plurality of binaural room impulse responses for the audio source in the playback room.

According to one embodiment, the at least one microphone in each of the two headphone capsules can be designed, for example, to generate one or more additional recordings of the sound situation in the reproduction room during the playback of the at least two loudspeaker signals through the headphones, of which one or more additional recordings an augmented one subtract signal and determine the estimate of the raw audio signal from one or more audio sources and determine the binaural room impulse response of the plurality of binaural room impulse responses for the audio source in the playback room.

For example, in one embodiment, the analyzer 152 may be configured to determine acoustic space properties of the playback room and to adjust the plurality of binaural room impulse responses depending on the acoustic space properties.

According to one embodiment, the at least one microphone can be arranged, for example, in each of the two headphone capsules for measuring the sound close to the entrance of the ear canal.

In one embodiment, the system can include, for example, one or more further microphones outside the two headphone capsules for measuring the sound situation in the reproduction room.

According to one embodiment, the headphone can, for example, comprise a bracket, with at least one of the one or more further microphones being arranged, for example, on the bracket.

For example, in one embodiment, the speaker signal generator 154 may be configured to generate the at least two speaker signals by convolving each of the plurality of binaural room impulse responses with an audio source signal of a plurality of one or more audio source signals.

According to one embodiment, the analyzer 152 can be configured, for example, to determine at least one of the plurality of binaural spatial impulse responses (or several or all binaural spatial impulse responses) as a function of a movement of the headphones.

In one embodiment, the system can include a sensor in order to determine a movement of the headphones. For example, the sensor may be a sensor, such as an accelerometer, having at least 3 DoF (three degrees of freedom) to detect head rotations. For example, a 6 DoF sensor (English: six degrees of freedom sensor) can be used.

Certain embodiments of the invention address the technical challenge that it is often too loud in a listening environment, certain noises are annoying in the listening environment, and selective listening is desired. The human brain itself is capable of selective hearing, but intelligent technical aids can significantly improve selective hearing. Just as glasses help many people in today's life to perceive their surroundings better, there are hearing aids for hearing, but in many situations people with normal hearing can also benefit from the support of intelligent systems. In order to realize "intelligent hearables" (hearing aids), the (acoustic) environment must be analyzed by the technical system, individual sound sources must be identified in order to be able to treat them separately. On these issues there There is preliminary work, but real-time (transparent to our ears) and high sound quality (so that what is heard is indistinguishable from a normal acoustic environment) analysis and processing of the entire acoustic environment has not yet been realized in the prior art.

Improved machine listening concepts are provided below.

In a first step, the BRIR is measured with headphones either individually or with headphones using a probe microphone.

In a second step, the spatial properties of the recording room are analyzed based on the measured BRIR.

Optionally, in a third step, for example, at least one built-in microphone in each shell records the real sound situation in the playback room before playback begins. From these recordings, an estimate of the raw audio signal from one or more sources is determined and the respective BRIR of the sound source/audio source in the playback room is determined. The acoustic room properties of the playback room are determined from this estimate and the BRIR of the recording room is thus adjusted.

Also optionally, in a further step, at least one built-in microphone in each shell records the real sound situation in the playback room during playback. The augmented signal is first subtracted from these recordings, then an estimate of the raw audio signal from one or more sources is determined and the respective BRIR of the sound source/audio source in the playback room is determined. The acoustic room properties of the playback room are determined from this estimate and the BRIR of the recording room is thus adjusted.

In a further step, a source to be augmented is convolved with the position-correct, optionally adjusted, BRIR in order to obtain the headphone signals.

Finally, in a further step, the headphone signals are reproduced via headphones.

For example, in one embodiment, at least one microphone is placed in each headphone capsule to measure sound near the entrance of the ear canal.

According to one embodiment, additional microphones are optionally arranged on the outside of the headphones, possibly also on top of the bracket, for measuring and analyzing the sound situation in the playback room.

In embodiments, sound from natural and augmented sources is realized to be the same.

Embodiments realize that no measurement of headphone characteristics is required.

Embodiments thus provide concepts for measuring the spatial properties of the rendering space.

Some embodiments provide a seed and (post) optimization of the spatial adaptation. The concepts provided also work if the room acoustics of the playback room change, e.g. if the listener changes to another room.

Among other things, embodiments are based on installing different techniques for hearing assistance in technical systems and combining them in such a way that an improvement in the quality of sound and quality of life (e.g. desired sound is louder, undesired sound is quieter, better speech intelligibility) both for people with normal hearing and for people with damage to the hearing is achieved.

2 FIG. 12 shows a system for supporting selective hearing according to an embodiment.

The system includes a detector 110 for detecting an audio source signal portion of one or more audio sources using at least two received microphone signals of a listening environment.

The system also includes a position determiner 120 for assigning position information to each of the one or more audio sources.

The system also includes an audio type classifier 130 for assigning an audio signal type to the audio source signal portion of each of the one or more audio sources.

The system also includes a signal component modifier 140 for changing the audio source signal component of at least one audio source of the one or more audio sources depending on the audio signal type of the audio source signal component of the at least one audio source in order to obtain a modified audio signal component of the at least one audio source.

The analyzer 152 and the speaker signal generator 154 of 1 together form a signal generator 150.

The analyzer 152 of the signal generator 150 is designed to generate the plurality of binaural spatial impulse responses, the plurality of binaural spatial impulse responses being a plurality of binaural spatial impulse responses for each audio source of the one or more audio sources which, depending on the position information of this audio source and one Orientation of a user's head.

The loudspeaker signal generator 154 of the signal generator 150 is designed to generate the at least two loudspeaker signals as a function of the plurality of binaural room impulse responses and as a function of the modified audio signal component of the at least one audio source.

For example, according to one embodiment, the detector 110 may be configured to detect the audio source signal portion of the one or more audio sources using deep learning models.

In one embodiment, the position determiner 120 can be designed, for example, to determine the position information for each of the one or more audio sources depending on a recorded image or on a recorded video.

According to one embodiment, the position determiner 120 can be designed, for example, to determine the position information for each of the one or more audio sources as a function of the video by detecting a lip movement of a person in the video and depending on the lip movement, the audio source signal component is assigned to one of the one or more audio sources.

For example, in one embodiment, the detector 110 may be configured to determine one or more acoustic properties of the listening environment as a function of the at least two received microphone signals.

According to one embodiment, the signal generator 150 can be configured, for example, to determine the plurality of binaural room impulse responses depending on the one or more acoustic properties of the listening environment.

In one embodiment, the signal component modifier 140 can be configured, for example, to select the at least one audio source whose audio source signal component is modified depending on a previously learned user scenario and to modify it depending on the previously learned user scenario.

For example, according to one embodiment, the system may include a user interface 160 for selecting the previously learned user scenario from a set of two or more previously learned user scenarios. 3 16 shows such a system according to an embodiment, which additionally comprises such a user interface 160. FIG.

In one embodiment, the detector 110 and/or the position determiner 120 and/or the audio type classifier 130 and/or the signal component modifier 140 and/or the signal generator 150 can be implemented, for example, using a Hough transform or using parallel signal processing a plurality of VLSI chips or using a plurality of memristors.

According to one embodiment, the system can include a hearing aid 170, for example, which serves as a hearing aid for users with limited hearing ability and/or hearing impairment, the hearing aid including at least two loudspeakers 171, 172 for outputting the at least two loudspeaker signals. 4 12 shows such a system according to an embodiment, comprising such a hearing aid 170 with two corresponding loudspeakers 171,172.

In one embodiment, the system may include, for example, at least two speakers 181, 182 for outputting the at least two speaker signals and a housing structure 183 accommodating the at least two speakers, the at least one housing structure 183 being adapted to be attached to a head 185 of a user or another to be attached to any part of the user's body. Figure 5a shows a corresponding system, which includes such a housing structure 183 and two loudspeakers 181, 182.

According to one embodiment, the system can include a headphone 180, for example, which includes at least two loudspeakers 181, 182 for outputting the at least two loudspeaker signals. Figure 5b 18 shows a corresponding headphone 180 with two loudspeakers 181, 182 according to an embodiment.

For example, in one embodiment, the detector 110 and the position determiner 120 and the audio type classifier 130 and the signal component modifier 140 and the signal generator 150 can be integrated into the headset 180 .

According to an embodiment shown in 6 For example, the system may include a remote device 190 that includes detector 110 and position determiner 120 and audio type classifier 130 and signal component modifier 140 and signal generator 150 . The remote device 190 can be spatially separated from the headphones 180, for example.

For example, in one embodiment, remote device 190 may be a smartphone.

Embodiments do not necessarily use a microprocessor, but use parallel signal processing steps, such as Hough transformation, VLSI chips or memristors for the power-saving implementation, including artificial neural networks.

In embodiments, the auditory environment is spatially recorded and reproduced, which on the one hand uses more than one signal to represent the input signal and on the other hand also uses a spatial reproduction.

In embodiments, the signal separation is performed using Deep Learning (DL) models (e.g. CNN, RCNN, LSTM, Siamese Network) and simultaneously processes the information from at least two microphone channels, with at least one microphone being in each hearable. According to the invention, a number of output signals (corresponding to the individual sound sources) together with their respective spatial position are determined by the joint analysis. If the recording device (microphones) is connected to the head, then the positions of the objects change when the head moves. This enables a natural focusing on important/unimportant sound, eg by turning the listener towards the sound object.

In some embodiments, the signal analysis algorithms are based on a deep learning architecture, for example. Alternative variants with an analysis unit or variants with separate networks are used for the aspects of localization, detection and source separation. The alternative use of generalized cross-correlation (correlation versus time offset) takes account of the frequency-dependent shadowing by the head and improves localization, detection and source separation.

According to one embodiment, different source categories (e.g. speech, vehicles, male/female/child's voice, warning tones, etc.) are learned by the recognizer in a training phase. Here, the source separation networks are also trained for high signal quality, as well as the localization networks with targeted stimuli for high localization accuracy.

The training steps mentioned above use, for example, multi-channel audio data, with a first training run usually being carried out in the laboratory using simulated or recorded audio data. This is followed by a training session in different natural environments (e.g. living room, classroom, train station, (industrial) production environment, etc.), i.e. transfer learning and domain adaptation takes place.

Alternatively or additionally, the position detector could be coupled to one or more cameras to also determine the visual position of sound/audio sources. In the case of speech, lip movement and the audio signals coming from the source separator are correlated and thus a more precise localization is achieved.

After training, a DL model with network architecture and the associated parameters exists.

In some embodiments, the auralization is performed using binaural synthesis. The binaural synthesis offers the further advantage that it is possible not to delete unwanted components completely, but only to reduce them to the extent that they are perceptible but not disturbing. This has the further advantage of receiving unexpected additional sources (warning signals, calls,...) which would be ignored if the system was switched off completely.

According to some embodiments, the analysis of the auditory environment is not only used to separate the objects, but also to analyze the acoustic properties (e.g. reverberation time, initial time gap). These properties are then used in the binaural synthesis to adapt the pre-stored (possibly also individualized) binaural room impulse responses (BRIR) to the actual room. Due to the reduction in room divergence, the listener has a significantly reduced listening effort when understanding the optimized signals. Minimizing room divergence affects the externalization of auditory events and thus the plausibility of spatial audio reproduction in the listening room. There are no known solutions in the prior art for speech understanding or for the general understanding of optimized signals.

In embodiments, a user interface is used to determine which sound sources are selected. According to the invention, this is done here by prior learning of different user scenarios, such as "amplify speech right from the front" (conversation with one person), "amplify speech in the range of +-60 degrees" (conversation in a group), "suppress music and amplify music " (I don't want to hear concert goers), "make everything quiet" (I want my peace), "suppress all calls and warning tones", etc.

Some embodiments are independent of the hardware used, i.e. both open and closed headphones can be used. The signal processing can be integrated in the headphones, in an external device, or integrated in a smartphone. Optionally, in addition to the playback of acoustically recorded and processed signals, signals from the smartphone (e.g. music, telephony) can also be played back directly.

In other embodiments, an ecosystem for "selective listening with AI support" is provided. Exemplary embodiments relate to "Personalized Auditory Reality" (PARty). In such a personalized environment, the listener is able to amplify, attenuate, or modify defined acoustic objects. To create a sound experience tailored to individual needs, a number of analysis and to carry out synthesis processes. The work of the envisaged implementation phase forms an essential building block for this.

Some embodiments implement the analysis of the real sound environment and the detection of the individual acoustic objects, the separation, tracking and editability of the existing objects and the reconstruction and playback of the modified acoustic scene.

In exemplary embodiments, a detection of sound events, a separation of the sound events, and a suppression of some of the sound events is implemented.

In embodiments, AI methods (meaning in particular deep learning-based methods) are used.

Embodiments of the invention contribute to the technological development for recording, signal processing and reproduction of spatial audio.

Embodiments create, for example, spatiality and three-dimensionality in multimedia systems when the user interacts
Exemplary embodiments are based on researched knowledge of perceptual and cognitive processes of spatial hearing.

Some embodiments use two or more of the following concepts:
Scene decomposition: This includes a room-acoustic recording of the real environment and parameter estimation and/or a position-dependent sound field analysis.

Scene Representation: This includes representation and identification of the objects and the environment and/or efficient representation and storage.

Scene composition and rendering: This includes object and environment adjustment and manipulation and/or rendering and auralization.

Quality evaluation: This includes technical and/or auditory quality measurement.

Miking: This involves an application of microphone arrays and appropriate audio signal processing.

Signal processing: This includes feature extraction and dataset generation for ML (machine learning).

Estimation of room and environment acoustics: This includes in-situ measurement and estimation of room acoustic parameters and/or provision of room acoustic characteristics for source separation and ML.

Auralization: This includes a spatial audio reproduction with an auditory fit to the environment and/or a validation and evaluation and/or a proof of function and a quality assessment.

8 represents a corresponding scenario according to an exemplary embodiment.

Embodiments combine concepts for detecting, classifying, separating, locating, and enhancing sound sources, highlighting recent advances in each area and showing relationships between them.

Unified concepts are provided that can combine detect/classify/locate and separate/enhance sound sources to provide both the flexibility and robustness required for real-life SH.

Furthermore, embodiments for real-time performance provide appropriate low-latency concepts when dealing with the dynamics of real-life auditory scenes.

Some of the embodiments utilize concepts of deep learning, machine hearing, and smart hearables that allow listeners to selectively modify their auditory scene.

Embodiments provide the possibility for a listener to selectively improve, dampen, suppress or modify sound sources in the auditory scene using a hearing device such as headphones, earphones, etc.

9 represents a scenario according to an embodiment with four external sound sources. (In 9 mean: Keep - Maintain; Suppress - Suppress; alarm - alarm; cellphone - mobile phone; Speaker X - Speaker X; City Noise - city sounds; Source Control - source control).

In 9 the user represents the center of the auditory scene. In this case, four external sound sources (S1-S4) are active around the user. A user interface allows the listener to influence the auditory scene. Sources S1-S4 can be attenuated, enhanced or suppressed with their respective sliders. As in 2 can be seen, the listener can define sound sources or events to be retained or suppressed in the auditory scene. In 2 is designed to suppress the background noise of the city while preserving alarms or the ringing of phones. The user always has the option of playing an additional audio stream such as music or radio via the hearing device.

The user is usually the center of the system and controls the auditory scene via a control unit. The user can control the auditory scene with a user interface like the one in 9 displayed or modified with any kind of interaction such as voice control, gestures, direction of gaze, etc. Once the user has provided feedback to the system, the next step is a capture/classification/localization stage. In some cases only the acquisition is necessary, e.g. B. when the user wants to keep every speech utterance occurring in the auditory scene. In other cases classification might be necessary, e.g. B. if the user wants to keep fire alarms in the auditory scene, but not phone rings or office noise. In some cases only the location of the source is relevant to the system. This is the case, for example, with the four springs in 9 the case: the user can choose to remove or attenuate the sound source coming from a certain direction, regardless of the type or characteristics of the source.

10 Figure 12 illustrates a processing workflow of an SH application according to one embodiment.

The auditory scene is first in the stage of separation / improvement in 10 modified. This is done either by suppressing, dampening, or enhancing one specific sound source (or from specific sound sources). As in 10 As shown, an additional processing alternative in the SH is noise control, where the goal is to remove or minimize background noise from the auditory scene. Perhaps the most popular and widely used noise control technology today is Active Noise Control (ANC) [11].

One distinguishes selective listening from virtual and amplified auditory environments by restricting selective listening to those applications where only real audio sources are modified in the auditory scene without attempting to add any virtual sources to the scene.

From a machine hearing perspective, selective hearing applications require technologies to automatically detect, locate, classify, separate, and enhance sound sources. To further clarify the terminology related to selective hearing, we define the following terms, emphasizing their differences and interrelationships:
For example, embodiments utilize sound source localization, which refers to the ability to detect the position of a sound source in the auditory scene. In the context of audio processing, a source location usually refers to the direction of arrival (DOA) of a given source, which can be given either as a 2D coordinate (azimuth) or, if it includes an elevation, as a 3D coordinate . Some systems also estimate the distance from the source to the microphone as location information [3]. In the context of music processing, location often refers to the panning of the source in the final mix and is usually specified as an angle in degrees [4].

For example, embodiments utilize sound source detection, which refers to the ability to determine whether any instance of a given type of sound source is present in the auditory scene. An example of a detection process is to determine if any speaker is present in the scene. In this context, determining the number of speakers in the scene or the identity of the speakers goes beyond the scope of sound source detection. Detection can be thought of as a binary classification process where the classes correspond to source present and source absent.

For example, embodiments use Sound Source Classification, which assigns a class designation from a group of predefined classes to a given sound source or sound event. An example of a classification process is to determine whether a given sound source corresponds to speech, music, or ambient noise. Sound source classification and detection are closely related concepts. In some cases, classification systems include a level of coverage by considering "no class" as one of the possible designations. In these cases, the system implicitly learns to detect the presence or absence of a sound source and is not forced to assign a class designation if there is insufficient evidence that any of the sources is active.

For example, embodiments utilize sound source separation, which refers to the extraction of a given sound source from an audio mix or an auditory scene. An example of sound source separation is the extraction of a singing voice from an audio mix in which other musical instruments are played simultaneously in addition to the singer [5]. Sound source separation becomes relevant in a selective listening scenario, as it allows for the suppression of sound sources that are of no interest to the listener. Some sound separation systems implicitly perform a detection process before extracting the sound source from the mix. However, this is not necessarily the rule and so we emphasize the distinction between these operations. In addition, the separation often serves as a pre-processing stage for other types of analysis such as source enhancement [6] or classification [7].

For example, embodiments use Sound Source Identification, which goes one step further and aims to identify specific instances of a sound source in an audio signal. Speaker identification is perhaps the most common use of source identification today. The goal of this process is to identify whether a specific speaker is present in the scene. In the example in 1 the user has selected "speaker X" as one of the sources to keep in the auditory scene. This requires technologies that go beyond speech capture and classification, and requires speaker-specific models that enable this precise identification.

For example, embodiments utilize sound source enhancement, which refers to the process of increasing the prominence of a given sound source in the auditory scene [8]. In the case of speech signals, the goal is often to increase their perception of quality and intelligibility. A common scenario for speech enhancement is denoising noise-tainted speech utterances [9]. In the context of music processing, source enhancement refers to the concept of making remixes and is often done to make a musical instrument (sound source) stand out more in the mix. Remixing applications often use sound separation front-ends to gain access to the individual sound sources and to change the characteristics of the mix [10]. Although sound enhancement may be preceded by a sound source separation stage, this is not always the case and so we also emphasize the distinction between these two terms.

In the field of sound source detection, classification and identification, some of the embodiments use, for example, one of the following concepts, such as the detection and classification of acoustic scenes and events [18]. In this context, methods for audio event detection (AED) in home settings have been proposed, where the goal is to capture the time boundaries of a given sound event within 10-second recordings [19], [20]. In this particular case, 10 sound event classes were considered, including cat, dog, speech, alarm, and running water. Methods for detecting polyphonic sound events (several simultaneous events) have also been proposed in the literature [21], [22]. In [21] a method for the detection of polyphonic sound events is proposed, in which a total of 61 sound events from real-life situations are detected using binary activity detectors based on a recurrent neural network (RNN). using bidirectional long short-term memory (BLSTM).

For example, to deal with sparsely labeled data, some embodiments integrate transient attention mechanisms to focus on particular regions of the signal for classification [23]. The problem of noise labels in classification is particularly relevant for applications to selective listening, where the class designations can be so different that high quality designations are very costly [24]. Noise labels in sound event classification processes have been addressed in [25], where noise-robust loss functions based on categorical cross-entropy are presented, as well as ways to evaluate data with noise labels as well as manually labeled data. Likewise, [26] presents a system for audio event classification based on a convolutional neural network (CNN) that includes a verification step for sound labels based on a CNN prediction consensus on several segments of the test example.

For example, some embodiments implement simultaneous detection and localization of sound events. Thus, some embodiments, as in [27], perform the detection as a multi-label classification process and the location is given as the 3D coordinates of the direction of arrival (DOA) for each sound event.

Some embodiments use concepts of voice activity detection and speaker recognition/identification for SH. Voice activity detection has been addressed in noisy environments using denoising autoencoders [28], recurrent neural networks [29] or as an end-to-end system using raw waveforms [30]. For speaker recognition applications, many schemes have been proposed in the literature [31], with the vast majority focusing on increasing robustness to different conditions, for example with data augmentation or with improved embeddings that facilitate recognition [32]-[34]. So some of the embodiments make use of these concepts.

Further embodiments use concepts for the classification of musical instruments for sound event detection. The classification of musical instruments in both monophonic and polyphonic environments has been discussed in the literature [35], [36]. In [35], the predominant instrument in 3-second audio segments is classified under 11 instrument classes, with some aggregation methods proposed. Similarly, [37] proposes a method for detecting musical instrument activity that is capable of detecting instruments at a finer temporal resolution of 1 s. A considerable amount of research has been done in the area of singing voice analysis. In particular, methods such as [38] for proposed the process of capturing segments in an audio recording where the singing voice is active. Some embodiments make use of these concepts.

Some of the embodiments use one of the concepts discussed below for sound source localization. Thus, sound source localization is closely related to the problem of source counting, since the number of sound sources in the auditory scene is usually not known in real-life applications. Some systems operate on the assumption that the number of sources in the scene is known. This is the case, for example, with the model presented in [39], which uses histograms of active intensity vectors to locate the sources. [40] proposes, from a controlled perspective, a CNN-based algorithm to estimate the DOA of multiple speakers in the auditory scene using phase maps as input representations. In contrast, several works in the literature collectively estimate the number of sources in the scene and their location information. This is the case in [41], where a system for multi-speaker localization in noisy and reverberant environments is proposed. The system uses a complex-valued Gaussian Mixture Model (GMM) to estimate both the number of sources and their location information. The concepts described there are employed by some of the embodiments.

Sound source localization algorithms can be computationally demanding as they often involve scanning a large space around the auditory scene [42]. In order to reduce computational demands on the location algorithms, some of the embodiments use concepts that expand the search space by using clustering algorithms [43] or by performing multi-resolution searches [42] relative to best practices such as those based on the steered-response phase transform (steered response power phase transform, SRP-PHAT). Other methods place requirements on the sparsity of the matrix and assume that only one sound source is dominant in a given time-frequency range [44]. Recently, [45] proposed an end-to-end system for azimuth detection directly from the raw waveforms. Some of the embodiments utilize these concepts.

Some of the embodiments utilize Sound Source Separation (SSS) concepts described below, particularly in the areas of speech separation and music separation.

In particular, some embodiments employ concepts of speaker independent separation. There, a separation occurs without any prior information about the speakers in the scene [46]. Some embodiments also evaluate the speaker's spatial location to perform a separation [47].

Given the importance of computational performance in selective listening applications, research with the specific goal of achieving low latency is particularly relevant. Some work has been proposed to perform low-latency (<10 ms) speech separation with small available learning data [48]. To avoid delays caused by framing analysis in the frequency domain, some systems approach the separation problem by carefully designing filters to be applied in the time domain [49]. Other systems achieve low latency separation by modeling the time domain signal directly using a coder-decoder frame [50]. In contrast, some systems attempted to reduce framing delay in frequency domain separation approaches [51]. These concepts are employed by some of the embodiments.

Some embodiments employ music sound separation (MSS) concepts to extract a music source from an audio mix [5], such as main instrument and accompaniment separation concepts [52]. These algorithms take the most prominent sound source in the mix, regardless of its class designation, and attempt to separate it from the rest of the accompaniment. Some embodiments use concepts for singing voice separation [53]. In most cases, either specific source models [54] or data-driven models [55] are used to capture the characteristics of the singing voice. Although systems like the one proposed in [55] do not explicitly include a classification or a detection stage to achieve separation, the data-driven nature of these approaches allows these systems to implicitly learn to detect the singing voice with some accuracy before separation . Another class of algorithms in music attempts to perform a separation using only the location of the sources [4] without attempting to classify or detect the source prior to separation. Some of the embodiments employ anti-noise (ANC) concepts, such as Active Noise Cancellation (ANC). ANC systems mainly aim to reduce background noise for headphone users by using an anti-noise signal to cancel it [11]. ANC can be viewed as a special case of SH and faces an equally stringent requirement [14]. Some work has focused on antinoise in specific environments such as automotive interiors [56] or operational scenarios [57]. The work in [56] analyzes the cancellation of different types of noise, such as road noise and engine noise, and requires unified systems capable of dealing with different types of noise. Some work has focused on developing ANC systems for canceling noise over specific spatial regions. In [58], ANC is discussed over a spatial region using spherical harmonics as basis functions to represent the noise field. Some of the embodiments employ the concepts described herein.

Some of the embodiments use sound source enhancement concepts.

In the context of speech enhancement, one of the most common applications is to improve speech affected by noise. Many works focused on phase processing of single-channel speech enhancement [8]. From the perspective of the deep neural network domain, the problem of denoising speech utterances was addressed in [59] using denoising decoders, in [60] as a non-linear regression problem between clean and noisy speech utterances using a deep deep neural network (DNN) and in [61] as an end-to-end system using generative adversarial networks (GAN). In many cases, speech enhancement is used as a precursor to automatic speech recognition (ASR) systems, as is the case in [62] where speech enhancement is addressed with an LSTM RNN. Speech enhancement is often performed in conjunction with sound source separation approaches, where the basic idea is to first extract the speech utterance and then apply enhancement techniques to the isolated speech signal [6]. The concepts described herein are employed by some of the embodiments.

Source enhancement in the context of music mostly refers to applications for making music remixes. In contrast to speech enhancement, where the assumption is often that speech is affected only by noise sources, music applications mostly assume that other sound sources (musical instruments) are playing simultaneously with the source to be enhanced. Therefore, music remix applications are always provided preceded by a source separation application. For example, in [10] early jazz recordings were remixed using techniques to separate lead and accompaniment, harmonic and percussion instruments to achieve better tonal balance in the mix. Similarly, [63] investigated the use of different vocal separation algorithms to change the relative loudness of the vocal and backing track, showing that an increase of 6 dB is possible by introducing slight but audible distortions into the final mix. In [64] the authors explore ways to improve music perception for cochlear implant users by applying sound source separation techniques to achieve new mixes. The concepts described there are used by some of the embodiments.

One of the major challenges in selective listening applications relates to the stringent processing time requirements. The complete processing workflow must be executed with minimal delay to preserve the naturalness and quality perception for the user. The maximum acceptable latency of a system depends heavily on the application and the complexity of the auditory scene. For example, McPherson et al. 10 ms as an acceptable latency reference for interactive music interfaces [12]. For music performances over a network, the authors in [13] report that delays become noticeable in the range between 20-25 and 50-60 ms. However, anti-noise/active noise cancellation (ANC) technologies require ultra-low latency processing for better performance. In these systems, the amount of acceptable latency is both frequency and attenuation dependent, but can be as low as 1 ms for around 5 dB attenuation from frequencies below 200 Hz [14]. A final consideration regarding SH applications relates to the quality perception of the modified auditory scene. A significant amount of work has been done on methodologies for reliable audio quality assessment various applications [15], [16], [17]. However, the challenge with SH is to manage the clear trade-off between processing complexity and quality perception. Some of the embodiments use the concepts described there.

In some embodiments concepts for counting and localization in [41], for localization and detection in [27], for separation and classification in [65] and for separation and counting in [66] are used as described there.

Some embodiments employ concepts to improve the robustness of current machine hearing methods as described in [25], [26], [32], [34], new emerging directions range adaptation [67] and learning based on datasets recorded with multiple devices [68].

Some of the embodiments employ concepts for improving the computational efficiency of machine hearing as described in [48], or concepts described in [30], [45], [50], [61] that are able to deal with unprocessed waveforms.

Some embodiments implement a unified optimization scheme that combines detection/classification/location and separation/enhancement to selectively modify sound sources in the scene, with independent detection, separation, localization, classification, and enhancement methods being reliable and applicable to SH provide the required robustness and flexibility.

Some embodiments are suitable for real-time processing, with a good trade-off between algorithmic complexity and performance.

Some embodiments combine ANC and machine hearing. For example, the auditory scene is first classified and then ANC is selectively applied.

Further embodiments are provided below.

In order to enrich a real listening environment with virtual audio objects, the transfer functions from each of the positions of the audio objects to each of the listener's positions in a room must be known with sufficient accuracy.

The transfer functions map the properties of the sound sources, as well as the direct sound between the objects and the user, as well as all reflections that occur in the room. In order to ensure correct spatial audio reproductions for the room acoustics of a real room in which the listener is currently located, the transfer functions must also represent the room acoustic properties of the listening room with sufficient accuracy.

In audio systems that are suitable for the presentation of individual audio objects at different positions in the room, the challenge lies in the appropriate recognition and separation of the individual audio objects when a large number of audio objects are present. Furthermore, the audio signals of the objects in the recording position or in the listening position of the room overlap. Both the room acoustics and the superimposition of the audio signals change when the objects and/or the listening positions in the room change.

Room acoustics parameters must be estimated quickly enough in the case of relative movement. A low latency of the estimation is more important than a high accuracy. On the other hand, if the position of the source and receiver do not change (static case), a high degree of accuracy is required. In the proposed system, room acoustics parameters, as well as room geometry and listener position are estimated or extracted from a stream of audio signals. The audio signals are recorded in a real environment in which the source(s) and receiver(s) can move in any direction, and in which the source(s) and/or receiver(s) change their orientation in any way be able.

The audio signal stream can be the result of any microphone setup that includes one or more microphones. The streams are fed to a signal processing stage for pre-processing and/or further analysis. Thereafter, the output is fed to a feature extraction stage. This stage estimates the room acoustics parameters, eg T60 (Reverberation Time), DRR (Direct to Reverberation Ratio) and others.

A second data stream is generated by a 6DoF ("six degrees of freedom" - degrees of freedom: three dimensions each for position in space and line of sight) sensor that records the orientation and position of the microphone setup. The position data stream is fed into a 6DoF signal processing stage for pre-processing or further analysis.

The output of the 6DoF signal processing, the audio feature extraction stage and the pre-processed microphone streams is fed into a machine learning block by estimating the listening room (size, geometry, reflective surfaces) and the position of the microphone field in the room. In addition, a user behavior model is applied to enable a more robust estimation. This model takes into account limitations of human movements (e.g. continuous movement, speed, etc.), as well as the probability distribution of different types of movements.

Some of the embodiments realize a blind estimation of room acoustics parameters by using arbitrary microphone arrays and by adding position and pose information of the user, and by analyzing the data with machine learning methods.

Systems according to embodiments may be used for acoustic augmented reality (AAR), for example. There a virtual room impulse response has to be synthesized from the estimated parameters.

Some embodiments involve removing reverberations from the recorded signals. Examples of such embodiments are hearing aids for people with normal hearing and those who are hard of hearing. The reverberation can be removed from the input signal of the microphone setup with the help of the estimated parameters.

Another application is the spatial synthesis of audio scenes created in a room other than the current listening room. For this purpose, the room-acoustic parameters, which are part of the audio scenes, are adapted to the room-acoustic parameters of the listening room.

In the case of binaural synthesis, the available BRIRs are adapted to the acoustic parameters of the listening room.

In one embodiment, an apparatus for determining one or more room acoustics parameters is provided.

The device is designed to receive microphone data that includes one or more microphone signals.

Furthermore, the device is designed to receive tracking data relating to a position and/or an orientation of a user.

In addition, the device is designed to determine the one or more room acoustics parameters as a function of the microphone data and as a function of the tracking data.

For example, according to one embodiment, the device may be configured to use machine learning to determine the one or more room acoustic parameters based on the microphone data and based on the tracking data.

For example, in one embodiment, the device may be configured to employ machine learning in that the device may be configured to employ a neural network.

For example, according to one embodiment, the device may be configured to use cloud-based processing for machine learning.

For example, in one embodiment, the one or more room acoustic parameters may include reverberation time.

For example, according to one embodiment, the one or more room acoustic parameters may include a direct-to-reverberation ratio.

In one embodiment, the tracking data to indicate the user's location may include, for example, an x-coordinate, a y-coordinate, and a z-coordinate.

According to one embodiment, the tracking data to indicate the user's orientation may include, for example, a pitch coordinate, a yaw coordinate, and a roll coordinate.

In one embodiment, the device can be designed, for example, to transform the one or more microphone signals from a time domain into a frequency domain, wherein the device can be designed, for example, to extract one or more features of the one or more microphone signals in the frequency domain, and wherein the Device can be designed, for example, to determine the one or more room acoustics parameters depending on the one or more features.

For example, according to one embodiment, the device may be configured to use cloud-based processing to extract the one or more features.

For example, in one embodiment, the device may include a microphone array of multiple microphones to pick up the multiple microphone signals.

According to one embodiment, the microphone arrangement can be designed, for example, to be worn on the body by a user.

In one embodiment, the system described above may further comprise, for example, a device as described above for determining one or more room acoustic parameters.

According to one embodiment, the signal portion modifier 140 can be configured, for example, to change the audio source signal portion of the at least one audio source of the one or more audio sources as a function of at least one of the one or more room acoustics parameters; and/or the signal generator 150 can be designed, for example, to generate at least one of the plurality of binaural room impulse responses for each audio source of the one or more audio sources depending on the at least one of the one or more room acoustics parameters.

7 Figure 12 shows a system according to an embodiment comprising five sub-systems (sub-systems 1-5).

Sub-system 1 includes a microphone setup of one, two or more individual microphones that can be combined into a microphone array if more than one microphone is available. The positioning and relative arrangement of the microphone(s) to one another can be arbitrary. The microphone assembly can be part of a device worn by the user or may be a separate device positioned in the space of interest.

Furthermore, sub-system 1 comprises a tracking device to measure the user's translational positions and the user's head pose in space. Up to 6-DOF (x-coordinate, y-coordinate, z-coordinate, pitch angle, yaw angle, roll angle) can be measured. The tracking device can be positioned on a user's head, or it can be split into different sub-devices to measure the required DOFs and placed on the user or not on the user.

Subsystem 1 thus represents an input interface that includes a microphone signal input interface 101 and a position information input interface 102 .

Sub-system 2 includes signal processing for the captured microphone signal(s). This includes frequency transformations and/or time domain based processing. Furthermore, this includes methods for combining different microphone signals in order to realize field processing. It is possible to feed back from subsystem 4 in order to adapt parameters of the signal processing in subsystem 2. The signal processing block of the microphone signal(s) can be part of the device in which the microphone(s) are built or it can be part of a separate device. It can also be part of cloud-based processing.

Furthermore, sub-system 2 includes signal processing for the recorded tracking data. This includes frequency transforms and/or time domain based processing. It also includes methods to improve the technical quality of the signals using noise reduction, smoothing, interpolation and extrapolation. It also includes procedures to derive information from higher levels. This includes speeds, accelerations, travel directions, rest times, movement areas, movement paths. Further, this includes predicting a near-future trajectory and a near-future velocity. The signal processing block of the tracking signals can be part of the tracking device or it can be part of a separate device. It can also be part of cloud-based processing.

Sub-system 3 involves the extraction of features of the processed microphone(s).

The feature extraction block can be part of the user's handheld device, or it can be part of a separate device. It can also be part of cloud-based processing.

Sub-systems 2 and 3 with their modules 111 and 121 together implement, for example, the detector 110, the audio type classifier 130 and the signal component modifier 140. For example, sub-system 3, module 121 can be the result of an audio type classification on sub-system 2, pass module 111 (feedback). Sub-system 2, module 112 implements a position determiner 120, for example. Furthermore, in one embodiment, sub-systems 2 and 3 can also implement the signal generator 150, for example by sub-system 2, module 111 generating the binaural room impulse responses and generating the loudspeaker signals.

Sub-system 4 includes methods and algorithms to estimate room acoustic parameters using the processed microphone signal(s), the extracted features of the microphone signal(s), and the processed tracking data. The output of this block is the room acoustic parameters as rest data and a control and modification of the parameters of the microphone signal processing in subsystem 2. The machine learning block 131 can be part of the user's device or it can be part of a separate device. It can also be part of cloud-based processing.

Furthermore, sub-system 4 includes post-processing of the room-acoustic resting data parameters (e.g. in block 132). This includes a detection of outliers, a combination of single parameters to a new parameter, smoothing, extrapolation, interpolation and plausibility check. This block also gets information from subsystem 2. This includes near-future positions of the user in the room to estimate near-future acoustic parameters. This block can be part of the user's device or it can be part of a separate device. It can also be part of cloud-based processing.

Sub-system 5 includes the storage and allocation of the room acoustic parameters for downstream systems (e.g. in memory 141). The allocation of the parameters can be done just-intime can be realized and/or the time course can be stored. Storage can be done on the device that is on or near the user, or can be done on a cloud-based system.

Use cases for exemplary embodiments of the invention are described below.

One use case of an embodiment is home entertainment and relates to users in a home environment.

For example, a user would like to concentrate on certain playback devices such as TV, radio, PC, tablet and block out other sources of interference (from other users' devices or children, construction noise, street noise). The user is in the vicinity of the preferred playback device and selects the device or its position. Regardless of the user's position, the selected device or sound source positions are acoustically highlighted until the user cancels their selection.

For example, the user goes near the target sound source. The user selects the target sound source via a suitable interface, and the hearable adjusts the audio playback based on the user's position, user's line of sight and the target sound source, so that the target sound source can be clearly understood even in the presence of background noise.

Alternatively, the user moves close to a particularly disruptive sound source. The user selects this noise source via a suitable interface, and the hearable (hearing aid) adjusts the audio playback based on the user's position, user's line of sight and the source of the noise in order to explicitly suppress the source of the noise.

Another use case of another embodiment is a cocktail party where a user is between multiple speakers.
For example, when many speakers are present, a user would like to concentrate on one (or more) speakers and block out or attenuate other sources of interference. In this application, the control of the hearable may only require little active interaction from the user. Optional would be to control the strength of the selectivity using Biosignals or recognizable indicators of conversational difficulties (frequent inquiries, foreign languages, strong dialects).

For example, the speakers are randomly distributed and move relative to the listener. In addition, there are regular pauses in speaking, new speakers join, other speakers move away. Noise such as music can be comparatively loud under certain circumstances. The selected speaker is highlighted acoustically and recognized again even after pauses in speaking, changes in position or pose.

For example, a hearable recognizes a speaker in the user's environment. The user can use a suitable control option (e.g. line of sight, attention control) to select preferred speakers. The hearable adapts the audio playback according to the user's line of sight and the selected target sound source in order to be able to understand the target sound source even with background noise.

Alternatively, the user is addressed directly by a (previously) non-preferred speaker, who must at least be audible to ensure natural communication.

Another use case of another embodiment is in the automobile, where a user is in his (or in) a car. While driving, the user would like to actively direct their acoustic attention to certain playback devices such as navigation devices, radio or conversation partners in order to be able to better understand them in addition to the background noise (wind, engine, passengers).

For example, the user and the target sound sources are in fixed positions inside the vehicle. The user is static in relation to the reference system, but the vehicle itself moves. An adapted tracking solution is therefore necessary. The selected sound source position is acoustically highlighted until the user cancels the selection or until warning signals stop the device from functioning.

For example, a user gets into the car and the device recognizes the surroundings. The user can switch between the target sound sources using a suitable control option (e.g. speech recognition), and the hearable adjusts the audio playback according to the user's viewing direction and the selected target sound source in order to be able to understand the target sound source well even with background noise.

Alternatively, e.g. traffic-related warning signals interrupt the normal process and cancel the user's selection. The normal process is then restarted.

Another application of a further exemplary embodiment is live music and relates to a visitor to a live music event. For example, the visitor to a concert or live music performance would like to use the hearable to increase the focus on the performance and block out distracting listeners. In addition, the audio signal itself can be optimized, for example to compensate for an unfavorable listening position or room acoustics.

For example, the visitor is between many sources of interference, but the performances are usually relatively loud. The target sound sources are in fixed positions or at least in a defined area, but the user can be very mobile (e.g. dancing). The selected sound source position is acoustically highlighted until the user cancels the selection or until warning signals stop the device from functioning.

For example, the user selects the stage area or the musician(s) as the target sound source(s). The user can use a suitable control option to define the position of the stage/musicians, and the hearable adapts the audio playback to the target sound source according to the user's viewing direction and the selected target sound source to be able to understand well even with background noise.

Alternatively, e.g. warning information (e.g. evacuation, imminent thunderstorm at outdoor events) and warning signals can interrupt the normal process and cancel the user's selection. The normal process then restarts.

A further application of another exemplary embodiment is for large events and concerns visitors at large events. For example, at major events (e.g. football stadium, ice hockey stadium, large concert hall, etc.), a hearable can be used to emphasize the voices of family members and friends who would otherwise be lost in the noise of the crowds.

For example, a major event takes place in a stadium or a large concert hall where a large number of visitors go. A group (family, friends, school class) visits the event and is in front of or in the event area, where a large crowd of visitors is walking around. One or more children lose eye contact with the group and, despite the high noise level, call out to the group due to the surrounding noise. Then the user turns off voice recognition, and Hearable no longer amplifies the voice(s).
For example, one person from the group on the hearable selects the voice of the missing child. The hearable localizes the voice. Then the hearable amplifies the voice and the user can find the missing item again (quicker) using the amplified voice.

Alternatively, the missing child also wears a hearable, for example, and selects the voice of their parents. The hearable amplifies the parents' voice(s). The reinforcement then allows the child to locate its parents. So the child can walk back to his parents. Or, alternatively, the missing child also wears a hearable and selects the voice of their parents. The hearable locates the parent's voice(s) and the hearable announces the distance to the voices. The child can find its parents more easily. An optional playback of an artificial voice from the hearable for the distance announcement is provided.

For example, the hearables are coupled for a targeted amplification of the voice(s) and voice profiles are stored.

A further application of a further exemplary embodiment is leisure sports and relates to leisure athletes. Listening to music while exercising is popular, but it also poses risks. Warning signals or other road users may not be heard. In addition to music playback, the hearable can react to warning signals or shouts and temporarily interrupt music playback. Another use case in this context is sport in small groups. The sports group's hearables can be connected to ensure good communication with each other during sports while other noise is suppressed.

For example, the user is mobile and any warning signals are overlaid by numerous sources of interference. The problem is that not all warning signals may affect the user (far away sirens in the city, horns on the street). The Hearable automatically suspends music playback and acoustically highlights the warning signal or the communication partner until the user cancels his selection. The music will then continue to play normally.

For example, a user does sports and listens to music through Hearable. Warning signals or shouts affecting the user are automatically recognized and the hearable interrupts the music playback. The hearable adjusts the audio playback in order to be able to clearly understand the target sound source\the acoustic environment. The hearable then continues playing music automatically (e.g. after the end of the warning signal) or at the request of the user.

Alternatively, athletes in a group can connect their hearables, for example. The speech intelligibility between the group members is optimized and at the same time other disturbing noises are suppressed.

Another application of another embodiment is snoring suppression and affects all sleep seekers disturbed by snoring. People whose partners snore, for example, are disturbed in their nightly rest and have problems sleeping. The Hearable provides relief by suppressing the snoring noises, thus ensuring night-time rest and domestic peace. At the same time, the hearable allows other noises (crying babies, alarm sirens, etc.) to pass through so that the user is not completely acoustically isolated from the outside world. A snoring detection is provided, for example.

For example, the user has trouble sleeping due to snoring noises. By using the hearable, the user can then sleep better again, which has a stress-reducing effect.

For example, the user wears the hearable while sleeping. He switches the hearable to sleep mode, which suppresses all snoring noises. After sleeping, he turns the hearable off again.

Alternatively, other noises such as construction noise, lawn mower noise, etc. can be suppressed while sleeping.

Another application of a further exemplary embodiment is a diagnostic device for users in everyday life. The hearable records the preferences (e.g. which sound sources, which amplification/damping are selected) and creates them over the period of use a profile with tendencies. This data can be used to draw conclusions about changes in hearing ability. The goal is the early detection of hearing loss.

For example, the user wears the device in everyday life or in the use cases mentioned for several months or years. The hearable creates analyzes based on the selected setting and gives warnings and recommendations to the user.

For example, the user wears the hearable over a long period of time (months to years). The device automatically creates analyzes based on hearing preferences, and the device provides recommendations and warnings when hearing loss begins.

A further application of another exemplary embodiment is a therapy device and affects users with hearing impairments in everyday life. In its role as a transitional device to the hearing aid, potential patients are treated at an early stage and dementia is thus treated preventively. Other possibilities are use as a concentration trainer (e.g. for ADHD), treatment of tinnitus and stress reduction.

For example, the user has hearing or attention problems and uses the hearable temporarily/transitionally as a hearing aid. Depending on the hearing problem, this is reduced by the hearable, for example by: amplification of all signals (hearing impairment), high selectivity for preferred sound sources (attention deficits), reproduction of therapy noises (tinnitus treatment).

The user selects a form of therapy independently or on the advice of a doctor and makes the preferred settings, and the hearable carries out the selected therapy.

Alternatively, the Hearable detects hearing problems from UC-PRO1, and the Hearable automatically adjusts playback based on the problems detected and notifies the user.

Another use case of another embodiment is public sector work and relates to public sector workers. Employees in the public sector (hospitals, paediatricians, airport counters, educators, gastronomy, service counters, etc.) who are exposed to a high level of noise during work wear a hearable to improve the speech of one or just a few people communication and for better occupational safety by e.g. stress reduction.

For example, employees are exposed to a high level of noise in their work environment and, despite the background noise, have to talk to customers, patients or work colleagues without being able to move to quieter surroundings. Hospital staff are exposed to high levels of noise from the noise and beeping of medical equipment (or other work noise) and still need to be able to communicate with patients or colleagues. Paediatricians and educators work in the midst of children's noise, if necessary screaming, and need to be able to talk to the parents. At the airport counter, staff have difficulty understanding passengers due to the high level of noise in the airport hall. In gastronomy, the cellars have a hard time hearing their guests' orders in the noise level of busy restaurants. Then the user turns off voice selection, for example, and the hearable no longer amplifies the voice(s).

For example, a person switches on the attached hearable. The user sets the hearable to select nearby voices, and the hearable amplifies the closest voice or a few nearby voices while suppressing background noise. The user understands the relevant voice(s) better.

Alternatively, a person puts the hearable on permanent noise suppression. The user turns on the function of recognizing occurring voices and then amplifying them. This allows the user to continue working with less noise. When addressed directly from a radius of x meters, the hearable then amplifies the voice/s. The user can thus converse with the other person(s) at low noise levels. After the conversation, the hearable switches back to noise-cancelling mode alone, and after work, the user turns the hearable back off.

Another application of another exemplary embodiment is passenger transport and relates to users in a motor vehicle for passenger transport. For example, a user and driver of a passenger transporter would like to be distracted as little as possible by the people being transported while driving. Although the passengers are the main source of interference, communication with them is also necessary at times.

For example, there is a user or driver and the sources of interference are in fixed positions inside the vehicle. The user is static to the reference system, but the car itself moves. An adapted tracking solution is therefore necessary. Normally, noises and conversations between passengers are acoustically suppressed, unless communication is to take place.

For example, the Hearable suppresses background noise from the occupants by default. The user can manually override the suppression using a suitable control option (e.g. voice recognition, button in the vehicle). The Hearable adjusts the audio playback according to the selection.

Alternatively, the hearable detects that a passenger is actively addressing the driver and temporarily disables noise cancellation.

Another application of a further embodiment is school and training and relates to teachers and students in the classroom. In one example, the hearable has two roles, with the functions of the devices being partially coupled. The teacher's/presenter's device suppresses background noise and amplifies speech/questions from the ranks of the students. Furthermore, the hearables of the listeners can be controlled via the teacher's device. In this way, particularly important content can be highlighted without having to speak louder. Students can adjust their Hearable to better understand the teacher and block out disruptive classmates.

For example, teachers and students are in defined areas in closed rooms (this is the norm). When all devices are paired together, the relative positions are interchangeable, which in turn simplifies source separation. The selected sound source is acoustically highlighted until the user (teacher/student) cancels their selection or until warning signals stop the device from functioning.

For example, a teacher or lecturer presents content and the device suppresses background noise. The teacher wants to hear a student's question and changes the focus of the hearable to the questioner (automatically or through a suitable control option). After communication, all noises are suppressed again. In addition, it can be provided that, for example, a student who feels disturbed by classmates hides them acoustically. Furthermore, for example, a student sitting far away from the teacher can amplify his voice.

Alternatively, teacher and student devices can be paired, for example. The selectivity of the student devices can be temporarily controlled by the teacher device. For particularly important content, the teacher changes the selectivity of the student devices to amplify their voice.

Another use case of another embodiment is in the military and pertains to soldiers. Verbal communication between soldiers on deployment takes place on the one hand via radios and on the other hand via shouts and direct addressing. Radio is mostly used when greater distances have to be bridged and when communication between different units and subgroups is to be carried out. A fixed radio etiquette is often applied. Shouting and direct addressing is mostly used for communication within a squad or group. Difficult acoustic conditions can arise during the deployment of soldiers (e.g. screaming people, noise from weapons, storms), which can impair both communication channels. A soldier's equipment often includes a radio set with earphones. In addition to the purpose of audio reproduction, these also protect against excessive sound pressure levels. These devices are often equipped with microphones to bring environmental signals to the wearer's ears. Active noise suppression is also part of such systems. An expansion of the range of functions enables soldiers to be called out and spoken to directly in a noisy environment through intelligent damping of background noise and selective emphasis of speech with a directional reproduction. To do this, the relative positions of the soldiers in space/terrain must be known. Furthermore, speech signals and background noise must be separated from one another spatially and in terms of content. The system must also be able to cope with high SNR levels from soft whispers to screams and explosion sounds. The advantages of such a system are: verbal communication between soldiers in a noisy environment, retention of hearing protection, no need for radio etiquette, security against eavesdropping (since there is no radio solution).

For example, shouting out and direct addressing between soldiers in action can be made more difficult by background noise. This problem is currently being addressed by radio solutions in the short range and for longer distances. The new system enables calling out and direct addressing at close range through an intelligent and Spatial emphasis of the respective speaker with simultaneous attenuation of ambient noise.

For example, the soldier is on duty. Shouts and speech are automatically recognized and the system amplifies them while simultaneously dampening background noise. The system adjusts the spatial audio reproduction in order to be able to clearly understand the target sound source.

Alternatively, for example, the soldiers in a group can be known to the system. Only audio from those group members will pass through.

Another use case of another embodiment relates to security personnel and security officers. For example, the hearable can be used at confusing large events (celebrations, protests) for preventive crime detection. The selectivity of the hearable is controlled by keywords, e.g. calls for help or calls for violence. This requires an analysis of the content of the audio signal (e.g. speech recognition).

For example, the security officer is surrounded by many loud sound sources, and the officer and all of the sound sources may be in motion. A caller for help is not audible or only faintly audible under normal hearing conditions (poor SNR). The manually or automatically selected sound source is acoustically highlighted until the user cancels the selection. Optionally, a virtual sound object is placed at the position/direction of the interesting sound source in order to be able to easily find the location (e.g. in the event of a one-time call for help).

For example, the hearable recognizes sound sources with potential sources of danger. A security officer chooses which sound source or event he would like to investigate (e.g. by selecting it on a tablet). The hearable then adjusts the audio playback in order to be able to understand and locate the target sound source even with background noise.

Alternatively, for example if the target sound source has stopped, a locating signal can be placed in the direction/distance of the source.

Another use case of another embodiment is stage communication and relates to musicians. On stages at rehearsals or concerts (e.g. band, orchestra, choir, musical) due to difficult acoustic conditions, individual instruments (groups) that could still be heard in other surroundings cannot be heard. This affects the interaction, since important (accompanying) voices are no longer perceptible. The hearable can emphasize these voices and make them audible again and thus improve or ensure the interaction of the individual musicians. The use of this could also reduce the noise exposure of individual musicians and thus prevent hearing loss, for example by muting the drums, and at the same time the musicians could still hear everything important.

For example, a musician without Hearable can no longer hear at least one other voice on stage. The hearable can then be used here. When the rehearsal or the concert is over, the user puts the hearable back down after switching it off.

In one example, the user turns on the hearable. He selects one or more desired musical instruments to be amplified. When playing music together, the Hearable now amplifies the selected musical instrument and thus makes it audible again. After making music, the user switches the hearable off again.
In an alternate example, the user turns on the hearable. He selects the desired musical instrument whose volume is to be reduced. 7. When making music together, the Hearable now reduces the volume of the selected musical instrument so that the user only hears it at a moderate volume.

Musical instrument profiles, for example, can be stored in the hearable.

Another application of a further exemplary embodiment is source separation as a software module for hearing aids in terms of the ecosystem and relates to hearing aid manufacturers and hearing aid users. Hearing aid manufacturers can use source separation as an additional tool for their hearing aids and offer it to customers. Hearing aids could also benefit from the development. A license model for other markets/devices (headphones, mobile phones, etc.) is also conceivable.

For example, hearing aid users find it difficult to separate different sources from each other in a complex auditory situation, for example to focus on to assign a specific speaker. In order to be able to hear selectively without external additional systems (e.g. transmission of signals from mobile phone systems via Bluetooth, targeted signal transmission in classrooms via an FM system or inductive hearing systems), the user uses a hearing aid with the additional function for selective hearing. In this way, he can also focus on individual sources by separating the sources without having to do anything else. In the end, the user turns off the additional function and continues to hear normally with the hearing aid.

For example, a hearing device user buys a new hearing device with an integrated additional function for selective hearing. The user sets the selective hearing function on the hearing aid. Then the user selects a profile (e.g. amplify loudest/nearest source, amplify voice recognition of specific voices from the personal environment (as with the UC-CE5 at major events). The hearing aid amplifies the respective source/s according to the set profile and at the same time suppresses background noise if necessary , and the hearing aid user hears individual sources from the complex auditory scene instead of just a "noise mush"/muddle of acoustic sources.

Alternatively, the hearing device user buys the additional function for selective listening as software or the like for his own hearing device. The user installs the add-on feature for their hearing aid. Then the user sets the selective listening function on the hearing aid. The user selects a profile (amplify loudest/closest source, amplify voice recognition of specific voices from their personal environment (like the UC-CE5 at major events), and the hearing aid amplifies the source(s) according to the set profile, while suppressing background noise if necessary. The hearing aid user hears individual sources from the complex auditory scene instead of just a "noise mush"/muddle from acoustic sources.

For example, the hearable can provide voice profiles that can be stored.

Another use case of another embodiment is professional sports and relates to athletes in competition. In sports such as biathlon, triathlon, cycling, marathon, etc., professional athletes rely on information from their coaches or communication with teammates. However, there are also situations in which you want to protect yourself from loud noises (shooting at a biathlon, loud cheering, party horns, etc.) in order to be able to concentrate. The hearable could be for the respective sport/athlete be adjusted to enable a fully automatic selection of relevant sound sources (recognition of specific voices, loudness limitation for typical background noise).

For example, the user may be very mobile and the nature of the noise depends on the sport. Due to the intense sporting activity, the athlete is not able to control the device actively or only to a limited extent. However, in most sports there is a fixed procedure (biathlon: running, shooting) and the important discussion partners (coaches, team members) can be defined in advance. Noise is suppressed in general or in certain phases of the sport. Communication between athletes and team members and coaches is always emphasized.

For example, the athlete uses a hearable specially adapted to the sport. The Hearable suppresses background noise fully automatically (preset), especially in situations where a high degree of attention is required for the sport in question. Furthermore, the Hearable automatically (default) highlights coaches and team members when they are within hearing range.

A further application of a further exemplary embodiment is ear training and relates to music students, professional musicians, amateur musicians. For music rehearsals (e.g. in an orchestra, in a band, in an ensemble, in music lessons), a hearable is used in a targeted manner in order to be able to follow individual voices filtered out. Especially at the beginning of rehearsals it is helpful to listen to finished recordings of the pieces and to follow your own voice. Depending on the composition, the voices in the background can't be clearly heard because you only hear the voices in the foreground. With the hearable you could then emphasize a voice of your choice using the instrument or similar in order to be able to practice it more specifically.

(Aspiring) music students can also use the Hearable to train their ear training skills in order to prepare for entrance exams by minimizing individual highlights step by step until they finally extract the individual voices from complex pieces without help.

Another possible application is karaoke, for example if there is no singing star or similar in the vicinity. Then you can suppress the vocal part(s) from a piece of music at will in order to only hear the instrumental version for karaoke singing.

For example, a musician begins to relearn a voice from a piece of music. He listens to the recording of the piece of music on a CD system or another playback medium. When the user is done practicing, they turn the hearable back off.
In one example, the user turns on the hearable. He selects the desired musical instrument to be amplified. When listening to the piece of music, the hearable amplifies the voice(s) of the musical instrument and turns down the volume of the other musical instruments, allowing the user to hear their own voice better

In an alternate example, the user turns on the hearable. He selects the desired musical instrument to be suppressed. When listening to the song, the voice(s) of the selected song will be muted so that only the remaining voices can be heard. The user can then practice the voice on their own instrument with the other voices without being distracted by the voice from the recording.

In the examples, the hearable may provide stored musical instrument profiles.

Another use case of another embodiment is occupational safety and concerns workers in noisy environments. Workers in noisy environments, for example in machine halls or on construction sites, must protect themselves from noise, but also be able to perceive warning signals and communicate with employees.

For example, the user is in a very noisy environment and the target sound sources (warning signals, employees) may be significantly quieter than the interfering signals. The user may be mobile, but the noise interference is mostly stationary. As with hearing protection, noise is permanently reduced and the hearable automatically highlights a warning signal. Communication with employees is ensured by amplification of speaker sources
For example, the user goes about his work and uses Hearable as hearing protection. Warning signals (eg fire alarm) are acoustically emphasized and the user interrupts his work if necessary.

Alternatively, the user goes about his work, for example, and uses Hearable as hearing protection. If there is still a need for communication with employees, the communication partner is selected with the help of suitable interfaces (here, for example: gaze control) and highlighted acoustically
Another use case of another embodiment is source separation as a software module for live translators and concerns users of a live translator. Live translators translate spoken foreign languages in real time and can benefit from an upstream source separation software module. Especially when multiple speakers are present, the software module can extract the target speaker and potentially improve the translation.

For example, the software module is part of a live translator (dedicated device or smartphone app). For example, the user can select the target speaker via the device display. It is advantageous that the translator and the target sound source usually do not move or move very little during the translation. The selected sound source position is acoustically emphasized and thus potentially improves the translation.

For example, a user wants to have a conversation in a foreign language or listen to a foreign speaker. The user selects the target speaker through a suitable interface (e.g. GUI on the display) and the software module optimizes the audio recording for further use in the translator.

A further application of another exemplary embodiment is occupational safety for emergency services and relates to the fire brigade, THW, possibly the police, rescue services. For emergency services, good communication is essential for successful operation management. It is often not possible for the emergency services to wear hearing protection despite loud ambient noise, since then no communication with each other is possible. For example, firefighters must be able to communicate and understand commands precisely despite the loud engine noise, some of which is happening over radios. For this reason, emergency services are exposed to a high level of noise pollution, where the Hearing Protection Ordinance cannot be implemented. On the one hand, a hearable would offer hearing protection for the emergency services and, on the other hand, would continue to enable communication between the emergency services. Other points are that the emergency services are not acoustically isolated from the environment with the help of the hearable when wearing helmets/protective equipment and thus can help better. You can then communicate better and also better assess dangers for yourself (e.g. hear what kind of fire there is).

For example, the user is exposed to high ambient noise and therefore cannot wear hearing protection and still needs to be able to communicate with others. He uses the hearable. After the operation or the dangerous situation is over, the user can put the hearable back down.

For example, the user wears the hearable during an operation. He turns on the hearable. The hearable suppresses ambient noise and amplifies the speech of colleagues and other nearby speakers (e.g. fire victims).

Alternatively, the user wears the hearable during an operation. He turns on the Hearable, and the Hearable blocks out ambient noise and amplifies co-workers' speech over the radio.

If necessary, the hearable is specially designed to meet a structural suitability for use in accordance with a use regulation. The hearable may have an interface to a radio device.

Although some aspects have been described in the context of a device or a system, it is understood that these aspects also represent a description of the corresponding method, so that a block or a component of a device or a system can also be used as a corresponding method step or as a Feature of a process step is to be understood. Similarly, aspects described in connection with or as a method step also constitute a description of a corresponding block or detail or feature of a corresponding apparatus or system. Some or all of the method steps may be performed by hardware apparatus (or using a hardware apparatus) such as a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some or more of the essential process steps can be performed by such an apparatus.

Depending on particular implementation requirements, embodiments of the invention may be in hardware or in software, or at least partially in hardware or be at least partially implemented in software. Implementation can be performed using a digital storage medium such as a floppy disk, a DVD, a BluRay disk, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, a hard disk or other magnetic or optical Memory are carried out on which electronically readable control signals are stored, which can interact with a programmable computer system in such a way or interact that the respective method is carried out. Therefore, the digital storage medium can be computer-readable.

Thus, some embodiments according to the invention comprise a data carrier having electronically readable control signals capable of interacting with a programmable computer system in such a way that one of the methods described herein is carried out.

In general, embodiments of the present invention can be implemented as a computer program product with a program code, wherein the program code is effective to perform one of the methods when the computer program product runs on a computer.

The program code can also be stored on a machine-readable carrier, for example.

Other exemplary embodiments include the computer program for performing one of the methods described herein, the computer program being stored on a machine-readable carrier. In other words, an exemplary embodiment of the method according to the invention is therefore a computer program that has a program code for performing one of the methods described herein when the computer program runs on a computer.

A further exemplary embodiment of the method according to the invention is therefore a data carrier (or a digital storage medium or a computer-readable medium) on which the computer program for carrying out one of the methods described herein is recorded. The data carrier or digital storage medium or computer-readable medium is typically tangible and/or non-transitory.

A further exemplary embodiment of the method according to the invention is therefore a data stream or a sequence of signals which represents the computer program for carrying out one of the methods described herein. the The data stream or the sequence of signals can, for example, be configured to be transferred over a data communication link, for example over the Internet.

Another embodiment includes a processing device, such as a computer or programmable logic device, configured or adapted to perform any of the methods described herein.

Another embodiment includes a computer on which the computer program for performing one of the methods described herein is installed.

A further exemplary embodiment according to the invention comprises a device or a system which is designed to transmit a computer program for carrying out at least one of the methods described herein to a recipient. The transmission can take place electronically or optically, for example. For example, the recipient may be a computer, mobile device, storage device, or similar device. The device or the system can, for example, comprise a file server for transmission of the computer program to the recipient.

In some embodiments, a programmable logic device (e.g., a field programmable gate array, an FPGA) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, in some embodiments, the methods are performed on the part of any hardware device. This can be hardware that can be used universally, such as a computer processor (CPU), or hardware that is specific to the method, such as an ASIC.

The embodiments described above are merely illustrative of the principles of the present invention. It is understood that modifications and variations to the arrangements and details described herein will occur to those skilled in the art. Therefore, it is intended that the invention be limited only by the scope of the following claims and not by the specific details presented in the description and explanation of the embodiments herein.

References:

1. [1] V Valimaki, A Franck, J Ramo, H Gamper, and L Savioja, "Assisted listening using a headset: Enhancing audio perception in real, augmented, and virtual environments," IEEE Signal Processing Magazine, Vol. 32, No. 2, pp. 92-99, March 2015 .
2. [2] Brandenburg K, Cano E, Klein F, Köllmer T, Lukashevich H, Neidhardt A, Sloma U, and Werner S, "Plausible augmentation of auditory scenes using dynamic binaural synthesis for personalized auditory realities," in proc. of AES International Conference on Audio for Virtual and Augmented Reality, Aug 2018 .
3. [3] S Argentieri, P Dans, and P Soures, "A survey on sound source localization in robotics: From binaural to array processing methods," Computer Speech Language, Vol. 34, No. 1, pp. 87-112, 2015 .
4. [4] D FitzGerald, A Liutkus, and R Badeau, "Projection-based demixing of spatial audio," IEEE/ACM Trans. on Audio, Speech, and Language Processing, Vol. 24, No. 9, pp. 1560-1572 , 2016 .
5. [5] Cano E, FitzGerald D, Liutkus A, Plumbley MD, and Stöter F, "Musical source separation: An introduction," IEEE Signal Processing Magazine, Vol 36, No 1, pp 31-40, Jan 2019 .
6. [6] S Gannot, E Vincent, S Markovich-Golan, and A Ozerov, "A consolidated perspective on multimicrophone speech enhancement and source separation," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no 4, pp. 692-730, April 2017 .
7. [7] E Cano, J Nowak, and S Grollmisch, "Exploring sound source separation for acoustic condition monitoring in industrial scenarios," in Proc. of 25th European Signal Processing Conference (EUSIPCO), Aug 2017, pp. 2264-2268 .
8. [8th] T Gerkmann, M Krawczyk-Becker, and J Le Roux, "Phase processing for single-channel speech enhancement: History and recent advances," IEEE Signal Processing Magazine, Vol. 32, No. 2, pp. 55-66 , March 2015 .
9. [9] Vincent E, Virtanen T, and Gannot S, Audio Source Separation and Speech Enhancement. Wiley, 2018 .
10. [10] D Matz, E Cano, and J Abeßer, "New sonorities for early jazz recordings using sound source separation and automatic mixing tools," in Proc. of the 16th International Society for Music Information Retrieval Conference. Malaga, Spain: ISMIR, Oct. 2015, pp. 749-755 .
11. [11] SM Kuo and DR Morgan, "Active noise control: a tutorial review," Proceedings of the IEEE, Vol. 87, No. 6, pp. 943-973, June 1999 .
12. [12] A McPherson, R Jack, and G Moro, "Action-sound latency: Are our tools fast enough?" in Proceedings of the International Conference on New Interfaces for Musical Expression, July 2016 .
13. [13] C Rottondi, C Chafe, C Allocchio, and A Sarti, "An overview on networked music performance technologies," IEEE Access, Vol. 4, pp. 8823-8843, 2016 .
14. [14] S Liebich, J Fabry, P Jax, and P Vary, "Signal processing challenges for active noise cancellation headphones," in Speech Communication; 13th ITG Symposium, Oct 2018, pp. 1-5 .
15. [15] E Cano, J Liebetrau, D Fitzgerald, and K Brandenburg, "The dimensions of perceptual quality of sound source separation," in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), April 2018, pp. 601-605 .
16. [16] PM Delgado and J. Herre, "Objective assessment of spatial audio quality using directional loudness maps," in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2019, pp. 621-625 .
17. [17] CH Taal, RC Hendriks, R Heusdens, and J Jensen, "An algorithm for intelligibility prediction of time-frequency weighted noisy speech," IEEE Transactions on Audio, Speech, and Language Processing, Vol. 19, No. 7, p 2125-2136, Sep. 2011 .
18. [18] Plumbley, MD, Kroos, C, Bello, JP, Richard, G, Ellis, DP, and Mesaros, A, Proceedings of the Detection and Classification of Acoustic Scenes and Events 2018 Workshop (DCASE2018). Tampere University of Technology. Laboratory of Signal Processing, 2018 .
19. [19] R Serizel, N Turpault, H Eghbal-Zadeh, and A Parag Shah, "Large-Scale Weakly Labeled Semi-Supervised Sound Event Detection in Domestic Environments," July 2018, submitted to DCASE2018 Workshop .
20. [20] L. JiaKai, "Mean teacher convolution system for dcase 2018 task 4," DCASE2018 Challenge, Tech. Republic, September 2018 .
21. [21] G Parascandolo, H Huttunen, and T Virtanen, "Recurrent neural networks for polyphonic sound event detection in real life recordings," in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), March 2016, pp. 6440-6444 .
22. [22] E C, Çakir and T Virtanen, "End-to-end polyphonic sound event detection using convolutional recurrent neural networks with learned time-frequency representation input," in Proc. of International Joint Conference on Neural Networks (IJCNN), July 2018, pp. 1-7 .
23. [23] Xu Y, Kong Q, Wang W, and Plumbley MD, "Large-Scale Weakly Supervised Audio Classification Using Gated Convolutional Neural Network," in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary , AB, Canada, 2018, pp. 121-125 .
24. [24] B. Frenay and M. Verleysen, "Classification in the presence of label noise: A survey," IEEE Transactions on Neural Networks and Learning Systems, Vol. 25, No. 5, pp. 845-869, May 2014 .
25. [25] E Fonseca, M Plakal, DPW Ellis, F Font, X Favory, and X Serra, "Learning sound event classifiers from web audio with noisy labels," in Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, 2019 .
26. [26] M Dorfer and G Widmer, "Training general-purpose audio tagging networks with noisy labels and iterative self-verification," in Proceedings of the Detection and Classification of Acoustic Scenes and Events 2018 Workshop (DCASE2018), Surrey, UK, 2018 .
27. [27] Adavanne S, Politis A, Nikunen J, and Virtanen T, "Sound event localization and detection of overlapping sources using convolutional recurrent neural networks," IEEE Journal of Selected Topics in Signal Processing, pp 1-1, 2018 .
28. [28] Y Jung, Y Kim, Y Choi, and H Kim, "Joint learning using denoising variational autoencoders for voice activity detection," in Proc. of Interspeech, September 2018, pp. 1210-1214 .
29. [29] Eyben F, Weninger F, Squartini S, and Schuller B, "Real-life voice activity detection with LSTM recurrent neural networks and an application to hollywood movies," in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing, May 2013, pp. 483-487 .
30. [30] R Zazo-Candil, TN Sainath, G Simko, and C Parada, "Feature learning with rawwaveform CLDNNs for voice activity detection," in Proc. of INTERSPEECH, 2016 .
31. [31] M McLaren, Y Lei, and L Ferrer, "Advances in deep neural network approaches to speaker recognition," in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), April 2015, pp. 4814-4818 .
32. [32] Snyder, D, Garcia-Romero, D, Seil, G, Povey, D, and Khudanpur, S, "X-vectors: Robust DNN embeddings for speaker recognition," in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), April 2018, pp. 5329-5333 .
33. [33] M McLaren, D Castán, MK Nandwana, L Ferrer, and E Yilmaz, "How to train your speaker embeddings extractor," in Odyssey, 2018 .
34. [34] SO Sadjadi, JW Pelecanos, and S. Ganapathy, "The IBM speaker recognition system: Recent advances and error analysis," in Proc. of Interspeech, 2016, pp. 3633-3637 .
35. [35] Han Y, Kim J, and Lee K, "Deep convolutional neural networks for predominant instrument recognition in polyphonic music," IEEE/ACM Transactions on Audio, Speech, and Language Processing, Vol. 25, No. 1, p. 208-221, Jan 2017 .
36. [36] V. Lonstanlen and C.-E. Cella, "Deep convolutional networks on the pitch spiral for musical instrument recognition," in Proceedings of the 17th International Society for Music Information Retrieval Conference. New York, USA: ISMIR, 2016, pp. 612-618 .
37. [37] S Gururani, C Summers, and A Lerch, "Instrument activity detection in polyphonic music using deep neural networks," in Proceedings of the 19th International Society for Music Information Retrieval Conference. Paris, France: ISMIR, Sep. 2018, pp. 569-576 .
38. [38] J Schlütter and B Lehner, "Zero mean convolutions for level-invariant singing voice detection," in Proceedings of the 19th International Society for Music Information Retrieval Conference. Paris, France: ISMIR, Sep. 2018, pp. 321-326 .
39. [39] S. Delikaris-Manias, D. Pavlidi, A. Mouchtaris, and V. Pulkki, "DOA estimation with histogram analysis of spatially constrained active intensity vectors," in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), March 2017, pp. 526-530 .
40. [40] S. Chakrabarty and EAP Habets, "Multi-speaker DOA estimation using deep convolutional networks trained with noise signals," IEEE Journal of Selected Topics in Signal Processing, Vol. 13, No. 1, pp. 8-21, March 2019 .
41. [41] X Li, L Girin, R Horaud, and S Gannot, "Multiple-speaker localization based on direct-path features and likelihood maximization with spatial sparsity regularization," IEEE/ACM Transactions on Audio, Speech, and Language Processing, Vol. 25, No. 10, pp. 1997-2012, Oct 2017 .
42. [42] F Grondin and F Michaud, "Lightweight and optimized sound source localization and tracking methods for open and closed microphone array configurations," Robotics and Autonomous Systems, Vol. 113, pp. 63 - 80, 2019 .
43. [43] D Yook, T Lee, and Y Cho, "Fast sound source localization using two-level search space clustering," IEEE Transactions on Cybernetics, Vol 46, No 1, pp 20-26, Jan 2016 .
44. [44] D Pavlidi, A Griffin, M Puigt, and A Mouchtaris, "Real-time multiple sound source localization and counting using a circular microphone array," IEEE Transactions on Audio, Speech, and Language Processing, Vol. 21, No 10, pp. 2193-2206, Oct 2013 .
45. [45] P Vecchiotti, N Ma, S Squartini, and GJ Brown, "End-to-end binaural sound localization from the raw waveform," in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2019, pp. 451-455 .
46. [46] Luo Y, Chen Z, and Mesgarani N, "Speaker-independent speech separation with deep attractor network," IEEE/ACM Transactions on Audio, Speech, and Language Processing, Vol. 26, No. 4, p. 787- 796, April 2018 .
47. [47] Z Wang, J Le Roux, and JR Hershey, "Multi-channel deep clustering: Discriminative spectral and spatial embeddings for speaker-independent speech separation," in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), April 2018, pp. 1-5 .
48. [48] G. Naithani, T. Barker, G. Parascandolo, L. Bramsl
    
    w, NH Pontoppidan, and T Virtanen, "Low latency sound source separation using convolutional recurrent neural networks," in Proc. of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), Oct 2017, pp. 71-75.
49. [49] M Sunohara, C Haruta, and N Ono, "Low-latency real-time blind source separation for hearing aids based on time-domain implementation of online independent vector analysis with truncation of non-causal components," in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), March 2017, pp. 216-220 .
50. [50] Y Luo and N Mesgarani, "TaSNet: Time-domain audio separation network for real-time, single-channel speech separation," in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), April 2018, pp. 696-700 .
51. [51] J Chua, G Wang, and WB Kleijn, "Convolutive blind source separation with low latency," in Proc. of IEEE International Workshop on Acoustic Signal Enhancement (IWAENC), Sep. 2016, pp. 1-5 .
52. [52] Z Rafii, A Liutkus, F Stöter, SI Mimilakis, D FitzGerald, and B Pardo, "An overview of lead and accompaniment separation in music," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol 26, No. 8, pp. 1307-1335, Aug 2018 .
53. [53] F.-R. Stöter, A Liutkus, and N Ito, "The 2018 signal separation evaluation campaign," in Latent Variable Analysis and Signal Separation, Y Deville, S Gannot, R Mason, MD Plumbley, and D Ward, Eds. Cham: Springer International Publishing, 2018, pp. 293-305 .
54. [54] J.-L. Durrieu, B David, and G Richard, "A musically motivated midlevel representation for pitch estimation and musical audio source separation," Selected Topics in Signal Processing, IEEE Journal of, Vol. 5, No. 6, pp. 1180-1191 , Oct 2011 .
55. [55] S Uhlich, M Porcu, F Giron, M Enenkl, T Kemp, N Takahashi, and Y Mitsufuji, "Improving music source separation based on deep neural networks through data augmentation and network blending," in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017 .
56. [56] PN Samarasinghe, W. Zhang, and TD Abhayapala, "Recent advances in active noise control inside automobile cabins: Toward quieter cars," IEEE Signal Processing Magazine, Vol. 33, No. 6, pp. 61-73, Nov 2016 .
57. [57] GS Papini, RL Pinto, EB Medeiros, and FB Coelho, "Hybrid approach to noise control of industrial exhaust systems," Applied Acoustics, Vol. 125, pp. 102 - 112, 2017 .
58. [58] Zhang J, Abhayapala TD, Zhang W, Samarasinghe PN, and Jiang S, "Active noise control over space: A wave domain approach," IEEE/ACM Transactions on Audio, Speech, and Language Processing, Vol. 26, No 4, pp. 774-786, April 2018 .
59. [59] X Lu, Y Tsao, S Matsuda, and C Hori, "Speech enhancement based on deep denoising autoencoder," in Proc. of Interspeech, 2013 .
60. [60] Xu Y, Du J, Dai L, and Lee C, "A regression approach to speech enhancement based on deep neural networks," IEEE/ACM Transactions on Audio, Speech, and Language Processing, Vol. 23, No. 1, pp. 7-19, Jan 2015 .
61. [61] S. Pascual, A. Bonafonte, and J. Serra, "SEGAN: speech enhancement generative adversarial network," in Proc. of Interspeech, August 2017, pp. 3642-3646 .
62. [62] F Weninger, H Erdogan, S Watanabe, E Vincent, J Le Roux, J R Hershey, and B Schuller, "Speech enhancement with LSTM recurrent neural networks and its application to noise-robust ASR," in Latent Variable Analysis and Signal Separation, E Vincent, A Yeredor, Z Koldovsk
    
    , and P. Tichavsk
    
    , Eds. Cham: Springer International Publishing, 2015, pp. 91-99.
63. [63] H Wierstorf, D Ward, R Mason, EM Grais, C Hummersone, and MD Plumbley, "Perceptual evaluation of source separation for remixing music," in Proc. of Audio Engineering Society Convention 143, Oct 2017 .
64. [64] J Pons, J Janer, T Rode, and W Nogueira, "Remixing music using source separation algorithms to improve the musical experience of cochlear implant users," The Journal of the Acoustical Society of America, Vol. 140, No. 6, pp. 4338-4349, 2016 .
65. [65] Q Kong, Y Xu, W Wang, and MD Plumbley, "A joint separation-classification model for sound event detection of weakly labeled data," in Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), March 2018 .
66. [66] tv Neumann, K. Kinoshita, M. Delcroix, S. Araki, T. Nakatani, and R. Haeb-Umbach, "All-neural online source separation, counting, and diarization for meeting analysis," in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2019, pp. 91-95 .
67. [67] Gharib, S, Drossos, K, Cakir, E, Serdyuk, D, and Virtanen, T, "Unsupervised adversarial domain adaptation for acoustic scene classification," in Proceedings of the Detection and Classification of Acoustic Scenes and Events Workshop (DCASE), November 2018, pp. 138-142 .
68. [68] A Mesaros, T Heittola, and T Virtanen, "A multi-device dataset for urban acoustic scene classification," in Proceedings of the Detection and Classification of Acoustic Scenes and Events Workshop, Surrey, UK, 2018 .
69. [69] Abeßer J, Götze M, Kühnlenz S, Graefe R, Kühn C, Clauß T, Lukashevich H, "A Distributed Sensor Network for Monitoring Noise Level and Noise Sources in Urban Environments," in Proceedings of the 6th IEEE International Conference on Future Internet of Things and Cloud (FiCloud), Barcelona, Spain, pp. 318-324., 2018 .
70. [70] Virtanen T, Plumbley MD, Ellis D (Eds.), "Computational Analysis of Sound Scenes and Events," Springer, 2018 .
71. [71] J Abeßer, S Loannis Mimilakis, R Gräfe, H Lukashevich, "Acoustic scene classification by combining autoencoder-based dimensionality reduction and convolutional neural net-works," in Proceedings of the 2nd DCASE Workshop on Detection and Classification of Acoustic Scenes and Events, Munich, Germany, 2017 .
72. [72] A Avni, J Ahrens, M Geierc, S Spors, H Wierstorf, B Rafaely, "Spatial perception of sound fields recorded by spherical microphone arrays with varying spatial resolution," Journal of the Acoustic Society of America, 133 (5), pp. 2711-2721, 2013 .
73. [73] E Cano, D FitzGerald, K Brandenburg, "Evaluation of quality of sound source separation algorithms: Human perception vs quantitative metrics," in Proceedings of the 24th European Signal Processing Conference (EUSIPCO), pp. 1758-1762, 2016 .
74. [74] S. Marchand, "Audio scene transformation using informed source separation," The Journal of the Acoustical Society of America, 140(4), p. 3091, 2016 .
75. [75] S. Grollmisch, J. Abeßer, J. Liebetrau, H. Lukashevich, "Sounding industry: Challenges and datasets for industrial sound analysis (ISA)," in Proceedings of the 27th European Signal Processing Conference (EUSIPCO) (submitted), A Coruna , Spain, 2019 .
76. [76] J. Abeßer, M. Müller, "Fundamental frequency contour classification: A comparison between hand-crafted and CNN-based features, " in Proceedings of the 44th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2019 .
77. [77] J Abeßer, S Balke, M Müller, "Improving bass saliency estimation using label propagation and transfer learning," in Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, pp. 306-312 , 2018 .
78. [78] C.-R. Nagar, J. Abeßer, S. Grollmisch, "Towards CNN-based acoustic modeling of seventh chords for recognition chord recognition," in Proceedings of the 16th Sound & Music Computing Conference (SMC) (submitted), Malaga, Spain, 2019 .
79. [79] JS Gómez, J. Abeßer, E. Cano, "Jazz solo instrument classification with convolutional neural networks, source separation, and transfer learning", in Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, pp 577-584, 2018 .
80. [80] JR Hershey, Z Chen, J Le Roux, S Watanabe, "Deep clustering: Discriminative embeddings for segmentation and separation," in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 31- 35, 2016 .
81. [81] Cano E, Schuller G, Dittmar C, "Pitch-informed solo and accompaniment separation towards its use in music education applications", EURASIP Journal on Advances in Signal Processing, 2014:23, pp. 1-19 .
82. [82] Mimilakis SI, Drossos K, Santos JF, Schuller G, Virtanen T, Bengio Y, "Monaural Singing Voice Separation with Skip-Filtering Connections and Recurrent Inference of Time-Frequency Mask," in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Calgary, Canada, pp.721-725, 2018 .
83. [83] Gemmeke JF, Ellis DPW, Freedman D, Jansen A, Lawrence W, Moore RC, Plakal M, Ritter M. "Audio Set: An ontology and human-labeled dataset for audio events," in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, USA, 2017 .
84. [84] Kleiner, M. "Acoustics and Audio Technology," . 3rd ed. USA: J Ross Publishing, 2012 .
85. [85] M. Dickreiter, V. Dittel, W. Hoeg, M. Wöhr, M. "Handbuch der Tonstudiotechnik," A. Medienakademie (Eds). 7th ed. Vol. 1. Munich: KG Saur Verlag, 2008 .
86. [86] F. Müller, M. Karau. "Transparent hearing," in: CHI ,02 Extended Abstracts on Human Factors in Computing Systems (CHI EA '02), Minneapolis, USA, pp. 730-731, April 2002 .
87. [87] L.Vieira. "Super hearing: a study on virtual prototyping for hearables and hearing aids," Master Thesis, Aalborg University, 2018. Available at: https://projekter.aau.dk/projekter/files/287515943/MasterThesis_Luis.pdf .
88. [88] Sennheiser, "AMBEO Smart Headset," [Online]. Available: https://de-de.sennheiser.com/finalstop [Accessed: March 1, 2019 ].
89. [89] Orosound "Tilde Earphones" [Online]. Available: https://www.orosound.com/tilde-earphones/ [Accessed; March 1, 2019 ].
90. [90] K Brandenburg, E Cano Ceron, F Klein, T Köllmer, H Lukashevich, A Neidhardt, J Nowak, U Sloma, and S Werner, "Personalized auditory reality," in 44th Annual Conference for Acoustics (DAGA), Garching near Munich, German Society for Acoustics (DEGA), 2018 .
91. [91] U.S. 2015 195641 A1 , filing date: January 6, 2014; published July 9, 2015.

Claims (20)

System comprising: an analyzer (152) for determining a plurality of binaural spatial impulse responses, a loudspeaker signal generator (154) for generating at least two loudspeaker signals depending on the plurality of binaural room impulse responses and depending on the audio source signal from at least one audio source, wherein the analyzer (152) is configured to determine the plurality of binaural spatial impulse responses such that each of the plurality of binaural spatial impulse responses takes into account an effect resulting from a user wearing headphones. System according to claim 1,
the system comprising the headset,
wherein the headphones are designed to output the at least two loudspeaker signals. System according to claim 1 or 2,
wherein the headphone comprises two headphone capsules and at least one microphone for measuring sound in each of the two headphone capsules,
wherein the at least one microphone for measuring the sound is arranged in each of the two headphone capsules,
wherein the analyzer (152) is designed to carry out the determination of the plurality of binaural room impulse responses using the measurement of the at least one microphone in each of the two headphone capsules. System according to claim 3,
wherein the at least one microphone in each of the two headphone capsules is designed to generate one or more recordings of a sound situation in a reproduction room before the start of playback of the at least two loudspeaker signals through the headphones, from the one or more recordings an estimate of a raw audio signal at least to determine an audio source and to determine a binaural room impulse response of the plurality of binaural room impulse responses for the audio source in the playback room. System according to claim 4,
wherein the at least one microphone in each of the two headphone capsules is designed to produce one or more additional recordings of the sound situation in the reproduction room during the playback of the at least two loudspeaker signals through the headphones, to subtract an augmented signal from these one or more additional recordings and to estimate the raw audio signal from one or more audio sources and determine the binaural spatial impulse response of the plurality of binaural spatial impulse responses for the audio source in the playback room. System according to claim 4 or 5,
wherein the analyzer (152) is designed to determine acoustic room properties of the reproduction room and to adapt the plurality of binaural room impulse responses depending on the acoustic room properties. System according to one of claims 4 to 6,
wherein the at least one microphone is located in each of the two earphone capsules for measuring the sound close to the entrance of the ear canal. System according to one of claims 4 to 7,
the system comprising one or more further microphones outside the two headphone capsules for measuring the sound situation in the reproduction room. System according to claim 8,
wherein the headset comprises a headband, at least one of the one or more further microphones being arranged on the headband. System according to one of the preceding claims,
wherein the speaker signal generator (154) is configured to generate the at least two speaker signals by convolving each of the plurality of binaural room impulse responses with an audio source signal of a plurality of one or more audio source signals. System according to one of the preceding claims,
wherein the analyzer (152) is designed to determine at least one of the plurality of binaural spatial impulse responses as a function of a movement of the headphones. System according to claim 11,
wherein the system includes a sensor to determine movement of the headset. A system as claimed in any preceding claim, the system further comprising: a detector (110) for detecting an audio source signal portion of one or more audio sources using at least two received microphone signals of a listening environment, a position determiner (120) for assigning position information to each of the one or more audio sources, an audio type classifier (130) for assigning an audio signal type to the audio source signal portion of each of the one or more audio sources, and a signal component modifier (140) for changing the audio source signal component of at least one audio source of the one or more audio sources depending on the audio signal type of the audio source signal component of the at least one audio source in order to obtain a modified audio signal component of the at least one audio source, and wherein the analyzer (152) and the loudspeaker signal generator (154) together form a signal generator (150), wherein the analyzer (152) of the signal generator (150) is designed to generate the plurality of binaural spatial impulse responses, the plurality of binaural spatial impulse responses being a plurality of binaural spatial impulse responses for each audio source of the one or more audio sources, which are dependent on the Position information of this audio source and an orientation of a user's head, and wherein the loudspeaker signal generator (154) of the signal generator (150) is designed to generate the at least two loudspeaker signals depending on the plurality of binaural room impulse responses and depending on the modified audio signal component of the at least one audio source. System according to claim 13,
wherein the detector (110) is designed to detect the audio source signal portion of the one or more audio sources using deep learning models. System according to claim 13 or 14,
wherein the position determiner (120) is designed to determine the position information for each of the one or more audio sources as a function of a recorded image or of a recorded video. System according to one of claims 13 to 15,
wherein the signal component modifier (140) is designed to select the at least one audio source whose audio source signal component is modified depending on a previously learned user scenario and to modify it depending on the previously learned user scenario. System according to one of claims 13 to 16,
the system comprising a remote device (190) comprising the detector (110) and the position determiner (120) and the audio type classifier (130) and the signal component modifier (140) and the signal generator (150),
wherein the remote device is physically separated from the headset. The system of claim 17, wherein the remote device (190) is a smartphone. Method comprising: Determination of a plurality of binaural room impulse responses, Generation of at least two loudspeaker signals depending on the plurality of binaural room impulse responses and depending on the audio source signal from at least one audio source, wherein the plurality of binaural spatial impulse responses are determined such that each of the plurality of binaural spatial impulse responses accounts for an effect resulting from a user wearing headphones. Computer program with a program code for carrying out the method according to Claim 19.

Images