EP2206113A1

EP2206113A1 - Device and method for generating a multi-channel signal using voice signal processing

Info

Publication number: EP2206113A1
Application number: EP08802737A
Authority: EP
Inventors: Christian Uhle; Oliver Hellmuth; Jürgen HERRE; Harald Popp; Thorsten Kastner
Original assignee: Fraunhofer Gesellschaft zur Forderung der Angewandten Forschung eV
Current assignee: Fraunhofer Gesellschaft zur Forderung der Angewandten Forschung eV
Priority date: 2007-10-12
Filing date: 2008-10-01
Publication date: 2010-07-14
Anticipated expiration: 2028-10-01
Also published as: DE502008003378D1; KR101100610B1; CN101842834A; RU2010112890A; BRPI0816638B1; JP2011501486A; DE102007048973A1; CN101842834B; AU2008314183A1; BRPI0816638A2; KR20100065372A; PL2206113T3; HK1146424A1; JP5149968B2; RU2461144C2; WO2009049773A1; MX2010003854A; ATE507555T1; EP2206113B1; CA2700911A1

Abstract

In order to generate a multi-channel signal having a number of output channels greater than a number of input channels, a mixer is used for upmixing the input signal to form at least a direct channel signal and at least an ambience channel signal. A speech detector is provided for detecting a section of the input signal, the direct channel signal or the ambience channel signal in which speech portions occur. Based on this detection, a signal modifier modifies the input signal or the ambience channel signal in order to attenuate speech portions in the ambience channel signal, whereas such speech portions in the direct channel signal are attenuated to a lesser extent or not at all. A loudspeaker signal outputter then maps the direct channel signals and the ambience channel signals to loudspeaker signals which are associated to a defined reproduction scheme, such as, for example, a 5.1 scheme.

Description

Device and method for generating a multi-channel signal with a voice signal processing

description

The present invention relates to the field of audio signal processing, and more particularly to the generation of multiple output channels from fewer input channels, such as audio channels. B. one (mono) channel or two (stereo) input channels.

Multi-channel audio is becoming more and more popular. As a result, many end users now have multi-channel playback systems. This is mainly due to the growing popularity of DVDs, and many DVD users now have 5.1 multichannel equipment. Such playback systems generally consist of three speakers L (left), C (center) and R (right), which are typically located in front of the user, and two speakers Ls and Rs located behind the user, and typically one of them LFE channel, also called the low frequency effect channel or subwoofer. Such a channel scenario is indicated in Fig. 5b and in Fig. 5c. While the positioning of the loudspeakers L, C, R, Ls, Rs should be made with respect to the user as shown in FIGS. 10 and 11 in order to give the user the best possible sound impression, the positioning of the LFE channel (in FIGS. 5b and 5c) is not so critical because the ear can not locate at such low frequencies and thus the LFE channel can be located anywhere where it does not bother due to its considerable size.

Such a multi-channel system provides several advantages over a typical stereo reproduction, which is a two-channel reproduction such as shown in Fig. 5a. Even outside of the optimal central listening position results in improved stability of the front listening experience, which is also referred to as a "front image", due to the center channel. This results in a larger "sweet spot", where "sweet spot" stands for the optimal listening position.

Further, the listener has a better feeling of "immersing" in the audio scene due to the two rear speakers Ls and Rs.

However, there is a huge amount of user-owned or commonly available audio material that only exists as stereo material, so it only has two channels, the left channel and the right channel. Typical sound carriers for such stereo pieces are compact disks.

To play such a stereo material over a 5.1 multi-channel audio system, you have two options, which are recommended according to the ITU.

The first option is to play the left and right channels through the left and right speakers of the multi-channel playback system. A disadvantage of this solution, however, is that you do not exploit the variety of existing speakers, so that you do not take advantage of the presence of the center speaker and the two rear speakers advantageous.

Another option is to convert the two channels into a multi-channel signal. This can be done during playback or by a special preprocessing, which advantageously takes advantage of all six loudspeakers of the existing 5.1 reproduction system, for example, and thus leads to an improved listening impression when the upmixing or the "upmixing" of two channels to 5 or 6 Channels is carried out faultlessly. Only then does the second option, ie the use of all loudspeakers of the multichannel system, have an advantage over the first solution, if one does not commit Üpmix errors. Such upmix errors can be especially troublesome if signals for the rear speakers, which are also known as ambience signals or ambient signals, are not generated without error.

One way to perform this so-called upmix process is known under the keyword "Direct Ambience Concept". The direct sound sources are reproduced by the three front channels so that they are perceived by the user at the same position as in the original two-channel version. The original two-channel version is shown schematically in Fig. 5a, using the example of various drum instruments.

Fig. 5b shows a highly mixed version of the concept in which all the original sound sources, ie the drum instruments, are again reproduced by the three front loudspeakers L, C and R, wherein in addition special environmental signals are output from the two rear loudspeakers. The term "direct sound source" is thus used to describe a sound coming only and directly from a discrete sound source, such as a drum instrument or other instrument, or generally a particular audio object, as shown schematically, eg, in FIG. 5a is shown using a drum instrument. Any additional sounds, such as due to wall reflections, etc. are not present in such a direct sound source. In this scenario, the sound signals output from the two rear speakers Ls, Rs in Fig. 5b consist only of environmental signals present or not in the original record. Such ambient signals or "ambience" signals do not belong to a single sound source but contribute to the reproduction of the room acoustics. a recording and thus lead to the so-called "immersion" feeling of the listener.

Another alternative concept, which is referred to as "in the band" concept, is shown schematically in FIG. 5c. Each type of sound, ie direct sound sources and ambient sounds, are all positioned around the listener. The position of a sound is independent of its characteristics (direct sound sources or ambient sounds) and depends only on the specific design of the algorithm, as described e.g. in Fig. 5c is shown. Thus, in Fig. 5c, it has been determined by the upmix algorithm that the two instruments 1100 and 1102 are positioned laterally with respect to the listener while the two instruments 1104 and 1106 are positioned in front of the user. As a result, the two rear speakers Ls, Rs now also contain portions of the two instruments 1100 and 1102 and no longer just ambient sounds, as was the case in Fig. 5b, where the same instruments are all positioned in front of the user have been.

The technical bulletin "C. Avendano and JM Jot:" Audio Extraction and Synthesis from Stereo Signals for MultiChannel Audio Upmix ", IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 02, Orlando, FL, May 2002" discloses a frequency domain technique to identify and extract environmental information in stereo audio signals. This concept is based on the computation of inter-channel coherence and a non-linear mapping function designed to allow time-frequency regions in the stereo signal to be determined, consisting mainly of environmental components. Ambient signals are then synthesized and used to store the back channels or "surround" channels Ls, Rs (Figs 10 and 11) of a multi-channel playback system.

In the specialist publication "R. Irwan and Ronald M. Aarts:" A method to convert stereo to multi-channel sound ", The proceedings of the AES 19th International Conference, Schloss Elmau, Germany, June 21-24, pp. 139-143, 2001 "a method is presented for converting a stereo signal to a multi-channel signal, the signal for the surround channels is determined using a cross-correlation technique A principal component analysis (PCA) is used to compute a vector indicating a direction of the dominant signal, which is then mapped from a two-channel representation to a three-channel representation. to create the three front channels.

All known techniques try in various ways to extract the ambience signals or ambient signals from the original stereo signal or even to synthesize from noise or other information, which also used for the synthesis of ambience signals, which are not in the stereo signal can be. Ultimately, however, it is always about extracting information from the stereo signal or feed information into a playback scenario, which are not explicitly available, since typically only a two-channel stereo signal and possibly any additional information or meta-information are available.

In the following, further known upmix or high-mix methods are described, which work without control parameters. Such high-mixing processes are also referred to as blind high-mixing processes or "blind-upmixing" processes.

Most such techniques for producing a so-called pseudo-stereophonic signal from a mono-channel (ie, a 1-to-2 upmix) are not signal adaptive. This means that they always process a mono signal the same regardless of which content is contained in the mono signal. Such systems often operate with simple filter structures and / or time delays to generate the generated ones To decorrelate signals, for example, by processing the single-channel input signal through a pair of so-called complementary comb filters, as described in M. Schroeder, "An artificial stereophonic effect obtained from using a single signal", JAES, 1957. Another review Such systems can be found in C. Faller, "Pseudo stereophony revisited", Proceedings of the AES 118nd Convention, 2005.

In addition, the Ambience Extraction technique also exists using non-negative matrix factorization, especially in the context of a 1-up-N upmix, where N is greater than two. Here, a time-frequency distribution (TFD) of the input signal is calculated, for example by means of a short-time Fourier transformation. An estimate of the TFD of the direct signal components is derived by a numerical optimization technique called non-negative matrix factorization. An estimate of the TFD of the ambient signal is determined by calculating the difference of the TFD of the input signal and the estimate of the TFD for the direct signal. The re-synthesis of the time signal of the surround signal is performed using the phase spectrogram of the input signal. Additional post processing is optionally performed to enhance the listening experience of the generated multichannel signal. This method is described in detail in C. Uhle, A. Walther, O. Hellmuth and J. Herre in "Ambience Separation from mono recordings using non-negative matrix factorizing", Proceedings of the AES 30th Conference 2007.

When mixing up stereo recordings, there are various techniques. One technique is the use of matrix decoders. Matrix decoders are known under the heading Dolby Pro Logic II, DTS Neo: 6 or HarmanKardon / Lexicon Logic 7 and in almost every Au- contained in the dio / video receiver sold today. As a by-product of their intended functionality, these processes are also able to perform a blind upmix. These decoders use interchannel differences and signal adaptive control mechanisms to produce multichannel output signals.

As already stated, frequency domain techniques described by Avendano and Jot are also used to identify and extract the ambience information in stereo audio signals. This method is based on the calculation of an interchannel coherence index and a non-linear mapping function, thereby making it possible to determine the time-frequency regions which are mainly composed of ambient signal components. The surround signals are subsequently synthesized and used to feed the surround channels of the multi-channel playback system.

One component of the direct / ambient high-mix process is the extraction of an environmental signal that is injected into the two back channels Ls, Rs. There are certain requirements for a signal that it is used as an environment-like signal in the context of a direct / environment high-racking process. A prerequisite is that no relevant parts of the direct sound sources should be audible in order to be able to locate the direct sound sources safely in front of the listener. This is especially important if the audio signal contains speech or one or more distinguishable speakers. Speech signals generated by a crowd, on the other hand, do not necessarily disturb the listener unless they are located in front of the listener.

If a particular set of speech components were reproduced through the posterior channels, this would result in the position of the one or more speakers of placed front to back or a short distance to the user or even behind the user, resulting in a very disturbing sound perception. Especially in the case where audio and video material is presented simultaneously, such as in a movie theater, such an impression is particularly disturbing.

A prerequisite for the sound signal of a movie (a soundtrack) is that the listening experience should conform to the impression created by the images. Audible clues to the localization should therefore not be in contrast to visible clues to the localization. Consequently, if a speaker is seen on the screen, the corresponding language should also be placed in front of the user.

The same applies to all other audio signals, d. H. is not necessarily limited to situations where both audio and video signals are presented simultaneously. Such other audio signals are for example broadcast signals or audiobooks. A listener is accustomed to producing speech from the front channels, and would likely turn around to restore his usual impression if speech were coming from the back channels at once.

In order to improve the quality of the ambient signals, it is proposed in the German patent application DE 102006017280.9-55 to subject a once extracted ambient signal to transient detection and to bring about a transient suppression without achieving significant losses of energy in the ambient signal. To this end, a signal substitution is made to replace areas with transients by corresponding signals without transients, but with approximately the same energy.

The AES Convention Paper "Descriptor-based specialization", J. Monceaux, F. Pachet et al., 28-31 May 2005, Barcelo Spain, discloses a descriptor-based spatialization in which speech detected on the basis of extracted descriptors is to be attenuated by muting only the center channel. For this purpose, a language extractor is used. An attack and settling time are used to smooth out modifications of the output signal. So a multi-channel soundtrack without language can be extracted from a movie. If a particular stereo reverberation feature is present in the original stereo downmix signal, this causes a high-mix tool to distribute that reverberation to each channel except for the center channel, so that reverberation is heard. To prevent this, dynamic level control is performed on L, R, Ls and Rs to attenuate the reverberation of a voice.

The object of the present invention is to provide a concept for generating a multi-channel signal with a number of output channels, which on the one hand provides flexibility and on the other hand, a high-quality product.

This object is achieved by a device for generating a multi-channel signal according to claim 1, a method for generating a multi-channel signal according to claim 23 or a computer program according to claim 24.

The present invention is based on the finding that speech components are suppressed in the rear channels, ie in the surrounding channels, so that the rear channels are speech component-free. To accomplish this, an input signal is highly mixed with one or more channels to provide a direct signal channel and to provide an environmental signal channel or, depending on the implementation, the modified surround signal channel. A speech detector is provided to search for speech components in the input signal, the direct channel or the surround channel, such speech components being temporal and / or frequency sections, or even in components of orthogonal decomposition, for example. A signal modifier is provided to modify the direct signal produced by the high mixer or a copy of the input signal to suppress the speech signal components there while less or not attenuating the direct signal components in the corresponding sections comprising speech signal components. Such a modified surround channel signal is then used to generate loudspeaker signals for corresponding loudspeakers.

However, if the input signal has been modified, the surround signal generated by the high mixer is used directly because the speech components are already suppressed there since the underlying audio signal also had already suppressed speech components. In this case, however, if the high-mix process also generates a direct channel, the direct channel is calculated not based on the modified input signal but on the basis of the unmodified input signal to selectively suppress the speech components, only in the environment channel, but not in the direct channel, in which the speech components are explicitly desired.

This prevents the reproduction of speech components in the back channels or surround signal channels that would otherwise disturb or even confuse the listener. Thus, according to the invention, it is ensured that dialogues and other speech understandable by a listener, that is, having a spectral characteristic typical of speech, are placed in front of the listener.

The same requirements exist for the in-band concept, in which it is also desired that direct signals are not placed in the rear channels, but rather in front of the listeners and, if necessary, to the side of the listener, but not behind the listener, as shown in Fig. 5c, where the direct signal components (and also the surround signal components) are all placed in front of the listener.

According to the invention, a signal-dependent processing is thus carried out in order to remove or suppress the speech components in the rear channels or in the ambient signal. For this purpose, two essential steps are taken, namely the detection of the occurrence of speech and the suppression of speech, wherein the detection of the occurrence of speech in the input signal, in the direct channel or in the surrounding channel can be made, and wherein the suppression of speech in the surrounding channel directly or indirectly can be made in the input signal, which is then used to generate the surround channel, this modified input signal is not used to generate the direct channel.

Thus, according to the invention, when one generates a multichannel surround signal from an audio signal with fewer channels containing speech components, it is ensured that the resulting signals for the rear channels viewed by the user comprise a minimal amount of speech, to get the original sound image before the user (front image). If a particular set of speech components were to be reproduced through the back channels, the position of the speakers would be positioned outside the front area, somewhere between the listener and the front speakers or, in extreme cases, even behind the listener. This would result in a very disturbing sound perception, especially if the audio signals are presented simultaneously with visual signals, as is the case for instance in films. Therefore, many multi-channel movie soundtracks contain hardly any speech components in the back channels. According to the invention, speech signal detected components and suppressed at a suitable location.

Preferred embodiments of the present invention will be explained below in detail with reference to the accompanying drawings. Show it:

Fig. 1 is a block diagram of an embodiment of the present invention;

Fig. 2 shows an assignment of time / frequency sections of an analysis signal and an environmental channel or input signal to explain the "corresponding sections";

FIG. 3 is an environmental signal modification according to a preferred embodiment of the present invention; FIG.

4 shows a cooperation between a speech detector and an environment signal modifier according to a further embodiment of the present invention;

5a shows a stereo reproduction scenario with direct sources (percussion instruments) and diffuse components;

Fig. 5b shows a multi-channel playback scenario in which all direct-switching sources are reproduced by the front channels and diffuse components are reproduced by all channels, this scenario also being referred to as a direct-environment concept;

5c shows a multi-channel reproduction scenario in which discrete switching sources can also be reproduced at least partially by rear channels and not or less in the transmission channels as shown in Fig. 5b by the rear speakers;

6a shows a further embodiment with a speech detection in the environment channel and a modification of the environment channel;

6b an embodiment with speech detection in the input signal and modification of the ambient channel;

6c an embodiment with a speech detection in the input signal and a modification of the input signal;

Fig. 6d shows a further embodiment with a speech detection in the input signal and a modification in the surrounding signal, wherein the modification is specially tuned to the speech;

7 shows an embodiment with band-wise gain calculation based on a bandpass signal / subband signal; and

8 is a more detailed illustration of a gain calculation block of FIG. 7. FIG.

Fig. 1 shows a block diagram of an apparatus for generating a multi-channel signal 10, which is shown in Fig. 1 such that it has a left channel L, a right channel R, a center channel C, an LFE channel, a left rear channel LS and a right rear channel RS. It should be noted, however, that the present invention is also suitable for any other representations than for this selected 5.1 representation, for example, for a 7.1 representation or for a 3.0 representation, in which case only a left channel, a right channel and a center channel is generated. The multi Channel signal 10 having the e.g. six channels shown in Fig. 1 is generated from an input signal 12 or " ^{xλλ having} a number of input channels, the number of input channels being 1 or greater than 1 and equal to 2, for example is when a stereo downmix is entered. In general, however, the number of output channels is greater than the number of input channels.

The apparatus shown in FIG. 1 includes a high mixer 14 for up-converting the input signal 12 to produce at least one direct signal channel 15 and one ambient signal channel 16 or optionally a modified ambient signal channel 16 '. Also provided is a speech detector 18 adapted to use as input the analysis signal, the input signal 12, as provided at 18a, or to use the direct signal channel 15, as provided at 18b, or to use another signal, which is similar in terms of the temporal / frequency appearance or in terms of its characteristics, as far as speech components, to the input signal 12. The speech detector detects a portion of the input signal, the direct channel or z. Also the environmental channel, as shown at 18c, in which a speech component occurs. This language component can be a significant language component, eg. For example, a speech component whose language property has been derived as a function of a specific qualitative or quantitative measure, wherein the qualitative measure and the quantitative measure exceeds a threshold, which is also referred to as speech detection threshold.

For a quantitative measure, a language property is quantified with a numeric value, and this numeric value is compared to a threshold. In the case of a qualitative measure, a decision is made per section, which can be made by one or more decision criteria. Such decision criteria may be, for example, various quantitative features that be compared / weighted with each other or processed somehow in order to come to a yes / no decision.

The apparatus shown in FIG. 1 further includes a signal modifier 20 configured to modify the original input signal, as shown at 20 a, or adapted to modify the environmental channel 16. When the control channel 16 is modified, the signal modifier 20 outputs a modified environmental channel 21, while when the input signal 20a is modified, a modified input signal 20b is output to the high mixer 14, which then modifies the modified environmental channel 16 '. B. generated by the same Hochmischvorgang that has been used for the direct channel 15. Should this hyperbolic process also lead to a direct channel due to the modified input signal 20b, this direct channel would be discarded because a direct channel derived from the unmodified (without speech suppression) input signal 12 and not from the modified input signal 20b is used as the direct channel according to the invention ,

The signal modifier is configured to modify portions of the at least one environmental channel or the input signal, which portions may be temporal or frequency portions or portions of orthogonal decomposition, for example. In particular, the portions corresponding to the portions detected by the speech detector are modified so that the signal modifier, as illustrated, generates the modified surround channel 21 or the modified input signal 20b in which a speech portion is attenuated or eliminated, wherein the speech portion in the corresponding portion of the direct channel has been less, or at best, not attenuated at all.

Moreover, the apparatus shown in Fig. 1 comprises a speaker signal output means 22 for outputting of loudspeaker signals in a reproduction scenario, such as the 5.1 scenario shown by way of example in FIG. 1, but also a 7.1 scenario, a 3.0 scenario or another or even higher scenario is also possible. In particular, to generate the loudspeaker signals for a display scenario, the at least one direct channel and the at least one modified surround channel are used, where the modified surround channel may either originate from the signal modifier 20, as shown at 21 or originate from the high mixer 14, as at 16 'is shown.

For example, if two modified surround channels 21 are provided, then these two modified surround channels could be fed directly into the two loudspeaker signals Ls, Rs, while the direct channels are fed only to the three front loudspeakers L, R, C, thus allowing complete separation between ambient signal components and direct signal components. The direct signal components are then all in front of the user and the surrounding signal components are all behind the user. Alternatively, ambient signal components can typically also be introduced to a smaller percentage in the front channels, so that z. B. the direct / ambient scenario shown in Fig. 5b is formed in which not only surround channels ambient signals are generated, but also from the front speakers z. L, C, R.

On the other hand, if the in-band scenario is preferred, then surrounding signal components will also be mainly from the front speakers z. B. L, R, C output, but also direct signal components are at least partially fed into the two rear speakers Ls, Rs. In order to achieve a placement of the two direct signal sources 1100 and 1102 in Fig. 5c at the locations shown, the proportion of the source 1100 in the speaker L will be about the same size as in the speaker Ls, so according to a typical panning rule, the source 1100 can be placed midway between L and Ls. Depending on the implementation, the loudspeaker signal output device 22 can thus effect a direct forwarding of a channel fed on the input side or can map the surrounding channels and the direct channels, for example by an in-band concept or a direct / ambient concept, such that a distribution the channels to the individual loudspeakers takes place and finally, in order to produce the actual loudspeaker signal, a summation of the components from the individual channels can take place.

Fig. 2 shows a time / frequency division of an analysis signal in the upper portion and an ambient channel or input signal in a lower portion. In particular, the time is plotted along the horizontal axis and the frequency is plotted along the vertical axis. This means that in FIG. 2 there are 15 time / frequency tiles or time / frequency sections for each signal, which have the same number in the analysis signal and in the surround channel / input signal. This means that the signal modifier 20 z. For example, if the speech detector 18 in section 22 detects a speech signal, somehow processes the portion of the surround channel / input signal, such as attenuates, completely eliminates, or substitutes a synthesis signal that has no speech property. It should be noted that in the present invention, the division need not be as selective as shown in FIG. Instead, even a temporal detection can already provide a satisfactory effect, in which case a specific time segment of the analysis signal, for example from second 2 to second 2.1 is detected as containing speech signal, and then the section of the ambient channel or the input signal also between second 2 and 2.1 to achieve speech suppression. Alternatively, an orthogonal decomposition can be performed, for. B. by means of a principal component analysis, in which case the same component decomposition is then used both in the environment channel or input signal and in the analysis signal. Then, certain components that have been detected as speech components in the analysis signal are attenuated or completely suppressed or eliminated in the ambient channel or input signal. Thus, depending on the implementation, a section is detected in the analysis signal, in which case this section is not necessarily processed in the analysis signal, but possibly also in another signal.

FIG. 3 shows an implementation of a speech detector in cooperation with an environmental channel modifier, wherein the speech detector provides only time information, that is, if FIG. 2 is considered, only wideband identifies the first, second, third, fourth or fifth time period and this information is the envelope - Channel channel modifier 20 via a control line 18d (Fig. 1) communicates. The speech detector 18 and the environmental channel modifier 20, operating synchronously or buffered, together achieve that in the signal to be modified, which may be, for example, the signal 12 or the signal 16, the speech signal is attenuated while ensuring in that such attenuation of the corresponding section in the direct channel does not occur or only occurs to a lesser extent. Depending on the implementation, this can be achieved by the high mixer 14 operating without regard to speech components, such as in a matrix method or in another method that does not perform special speech processing. The direct signal thus obtained is then supplied to the output device 22 without any further processing, while the ambient signal is processed with regard to a speech suppression. Alternatively, if the signal modifier subjects the input signal to speech suppression, the up-mixer 14 may, so to speak, operate twice to extract the direct channel component based on the original input signal, but to extract the modified surround channel 16 'based on the modified input signal 20b. Here, the same high-mix algorithm would run twice, but using a different input signal, in which one input signal the speech component is attenuated and in the other input signal the speech component is not attenuated.

Depending on the implementation, the environment channel modifier has broadband attenuation functionality or high pass filtering functionality, as set forth below.

Various implementations of the device according to the invention will be explained below with reference to FIGS. 6a, 6b, 6c and 6d.

In Fig. 6a, the environmental signal a is extracted from the input signal x, which extraction is part of the functionality of the upmixing 14. The occurrence of speech is detected in the surround signal a. The detection result d is used in the environment channel modifier 20, which computes the modified surround signal 21 in which speech components are suppressed.

FIG. 6b shows a configuration different from FIG. 6a in that the input signal and not the surrounding signal is supplied to the speech detector 18 as the analysis signal 18a. In particular, the modified surround channel signal a _{s is} calculated similarly to the configuration of FIG. 6 a, but the speech in the input signal is detected. This is motivated by the fact that the speech components generally in the input signal x signify can be found more easily than in the signal a. Thus, a higher reliability can be achieved by the configuration shown in FIG.

In Fig. 6c, the speech-modified surround signal a _{s is} extracted from a version x _{s of} the input signal which has already been subjected to speech signal suppression. Since the speech components typically emerge more prominently in x than in an extracted surround signal, their suppression is safer and more sustainable than in FIG. 6a. The disadvantage of the configuration shown in FIG. 6c compared to the configuration in FIG. 6a is that possible artifacts of the speech suppression and the environmental extraction process could still be increased depending on the type of extraction process. However, in Fig. 6c, the functionality of the environment channel extractor 14 is only used to extract the environment channel from the modified audio signal. However, the direct channel is not extracted from the modified audio signal x _s (20b), but on the basis of the original input signal x (12).

In the configuration shown in FIG. 6d, the environmental signal a is extracted from the input signal x by the high mixer. The occurrence of speech is detected in the input signal x. Furthermore, additional page information e, which additionally controls the functionality of the environment channel modifier 20, is calculated by a speech analyzer 30. This page information is calculated directly from the input signal and may be the location of speech components in a time / frequency representation, for example in the form of a spectrogram of FIG. 2, or may be additional information, which will be discussed in more detail below.

The functionality of the speech detector 18 will be discussed in greater detail below. The task of speech detection is to add a mix of audio signals analyze to estimate a likelihood that language is present. The input signal may be a signal that may be composed of a variety of different types of audio signals, such as a music signal, noise, or special sound effects, as known from movies. One way to detect speech is to use a pattern recognition system. Pattern recognition is understood to mean analyzing raw data and performing special processing based on a category of a pattern discovered in the raw data. In particular, the term "pattern" or "pattern" describes an underlying similarity that can be found between the measurements of objects of the same categories (classes). The basic operations of a pattern recognition system consist in capturing, that is to say recording the data using a converter, preprocessing, feature extraction and classification, whereby these basic operations can be performed in the given order.

Typically, microphones are used as sensors for a speech capture system. A preparation may include A / D conversion, resampling, or noise reduction. The feature extraction is the calculation of characteristic features for each object from the measurements. The features are chosen to be similar among objects of the same class, so that good intra-class compactness is achieved and that they are different for objects of different classes, so that inter-class separability is achieved. A third requirement is that the features should be robust with respect to noise, ambient conditions, and irrelevant transformations of the input signal to human perception. Feature extraction can be split into two separate stages. The first level is the feature calculation and the second level is feature projection or transformation to a generally orthogonal basis to minimize correlation between feature vectors and to reduce the dimensionality of the features by not using low energy elements.

The classification is the process of deciding whether speech is present or not based on the extracted features and a trained classifier. So let's say the following equation.

In the above equation, a set of training _vectors Sl _{x Y are} defined, where feature vectors are denoted by xi and the set of classes by Y. Thus, for a basic language acquisition, Y has two values, namely, {language, non-language} ,

In the training phase, the features Xi are calculated from labeled data, i. H. from audio signals that know which class y they belong to. After completing the training, the classifier has learned the characteristics of all classes.

In the application phase of the classifier, the features are computed from the unknown data as in the training phase and projected and classified by the classifier on the basis of the knowledge gained in training about the characteristics of the classes.

Specific implementations of the speech suppression, such as may be performed by the signal modifier 20, are discussed below. Thus, various methods can be used to suppress speech in an audio signal. There are methods known in the field of speech amplification and noise reduction for communication applications. Originally, voice amplification techniques were used to used to amplify the language in a mix of speech and background noise. Such methods can be modified to do the opposite, namely, suppression of speech as practiced for the present invention.

Thus, approaches to speech enhancement and noise reduction exist which attenuate or enhance the coefficients of a time / frequency representation according to an estimate of the degree of noise contained in such a time / frequency coefficient. If no additional background noise information is known, such as a priori information or information measured by a particular noise sensor, a time / frequency plot is obtained from a noisy measurement using, for example, special minimal statistics techniques. A noise suppression rule calculates a damping factor using the noise estimate. This principle is known as short-term spectral attenuation or spectral weighting, as is known, for example, in G. Schmid, "Single-channel noise suppression based on spectral weighting", Eurasip Newsletter 2004. Signal processing methods that are based on the principle of short-term Spectral Subtraction, Wiener Filtering, and the Ephraim-Malah Algorithm A more general formulation of the STSA approach leads to a signal subspace method, also known as a reduced-rank method in P. Hansen and S. Jensen, "Fir filter representation of reduced-rank noise reduction", IEEE TSP, 1998.

In principle, therefore, all methods that reinforce speech or suppress non-speech components can be used in the opposite way with respect to their known use in order to suppress speech or to reinforce non-speech. The general model of speech enhancement or noise suppression exists in that the input signal is a mixture of desired signal (speech) and background noise (non-speech). A suppression of the language is z. By inverting the attenuation factors in a STSA-based method, or by exchanging the definition of the desired signal and background noise.

However, an important requirement in speech suppression is that, in view of the context of upmixing, the resulting audio signal be perceived as a high audio quality audio signal. It is known that speech enhancement techniques and noise reduction techniques introduce audible artifacts into the output signal. An example of such artifact is known as music noise or musical tones and results from an erroneous estimation of noise floors and fluctuating subband attenuation factors.

Alternatively, blind source separation techniques may be used to separate the speech signal components from the surround signal and then separately manipulate both.

However, for the particular requirement of producing high quality audio signals, certain methods set forth below are preferred due to the fact that they perform significantly better in comparison to other methods. One method consists in the broadband attenuation, as indicated at 20 in FIG. The audio signal is attenuated at the intervals where speech is present. Special amplification factors range between -12 dB and -3 dB, with a preferred attenuation of 6 dB. Since other signal components / components are equally suppressed, one might think that the total loss of audio signal energy is clearly perceived. However, it has been found that this effect is not disturbing as the user in any case, concentrates on the front speakers L, C, R, in particular, when a voice sequence starts, so that the user will not perceive the energy absorption of the rear channels or of the ambient signal, if he is currently concentrating on a voice signal. This is reinforced in particular by the further typical effect that the level of the audio signal increases anyway due to an incipient speech. By introducing an attenuation in the range between -12 dB and 3 dB, the attenuation is not perceived as disturbing. Instead, the user feels much more comfortable that, due to the suppression of speech components in the back channels, an effect is achieved which results in the user's voice components being positioned exclusively in the front channels.

An alternative method, which is also indicated in Fig. 3 at 20, consists in a high-pass filtering. The audio signal is high-pass filtered where speech is present, with a cutoff frequency in the range between 600 Hz and 3000 Hz. The adjustment of the cutoff frequency results from the signal characteristic of speech with respect to the present invention. The long-term power spectrum of a speech signal focuses on an area below 2.5 kHz. The preferred range of the fundamental frequency of voiced speech is in the range between 75 Hz and 330 Hz. A range between 60 Hz and 250 Hz results for male adults. Mean values are 120 Hz for male speakers and 215 Hz for female speakers. Due to the resonances in the vocal tract certain signal frequencies are amplified. The corresponding peaks in the spectrum are also referred to as formant frequencies or simply as formants. Typically, there are about three significant formants below 3,500 Hz. Thus, speech exhibits a 1 / F nature, ie, the spectral energy decreases with increasing frequency. Therefore, for purposes of the present invention, speech components may be well-defined by a high-pass Filtering be filtered with the specified cutoff frequency range.

Another preferred implementation is sine signal modeling, which is illustrated with reference to FIG. Thus, in a first step 40, the fundamental wave of a speech is detected, which detection may take place in the speech detector 18 or, as shown in Fig. 6e, in the speech analyzer 30. Then, in a step 41, an examination is made to find the harmonics belonging to the fundamental wave. This functionality can be performed in the speech detector / speech analyzer or even in the ambient signal modifier. Thereafter, a spectrogram is calculated for the surround signal based on a block-wise forward transform as set forth at 42. Then the actual speech suppression is performed in a step 43, in which the fundamental wave and the harmonics are attenuated in the spectrogram. Then, in a step 44, the modified surround signal in which the fundamental and harmonics are attenuated or eliminated is again inverse transformed to reach the modified surround signal or the modified input signal.

This sinusoidal signal modeling is often used for tone synthesis, audio coding, source separation, tone manipulation, and noise suppression. Here, a signal is represented as a composition of sine waves with time-varying amplitudes and frequencies. Tonal speech signal components are manipulated by dividing the partial tones, i. H. the fundamental and its harmonics (harmonics) are identified and modified.

The partial tones are identified by means of a partial tone finder, as shown at 41. typically, Partial tone finding is performed in the time / frequency domain. A spectrogram is performed by means of a short-time Fourier transform, as indicated at 42. Local maxima in each spectrum of the spectrogram are detected and trajectories determined by local maxima of neighboring spectra. An estimate of the fundamental frequency may support the peak picking process, where this estimate of the fundamental frequency is performed at 40. A sinusoidal signal is then obtained from the trajectories. It should be noted that the order between step 40, 41 and step 42 can also be varied, so that first an outward transformation 42 is performed, which takes place in the speech analyzer 30 of FIG. 6d.

Various extensions of the derivation of a sinusoidal signal representation have been proposed. A multi-resolution processing approach to noise reduction is presented in D. Andersen and M. Clements, "Audio signal noise reduction using multi-resolution sinusoidal modeling", Proceedings of ICASSP 1999. An iterative process for deriving the sinusoidal representation has been described in US Pat J. Jensen and J. Hansen, "Speech enhancement using a constrained iterative sinusoidal model", IEEE TSAP 2001.

Using the sinusoidal signal representation, an improved speech signal is obtained by amplifying the sine component. The speech suppression according to the invention, however, wants to achieve exactly the opposite, namely to suppress the partial tones, the partial tones comprising the fundamental wave and its harmonics, for a speech segment with tonal speech. Typically, the high energy speech components are tonal. Thus, speech is spoken at a level of 60-75 dB for vowels and about 20-30 dB lower for consonants. For tonal speech (vocals), the excitation is a periodic pulse-like signal. The excitation signal is filtered by the vocal tract. Consequently, almost all of the energy is one tonal speech segment in the fundamental and its harmonics. By suppressing these partial tones, the speech components are significantly suppressed.

Another way to achieve speech suppression is illustrated in FIGS. 7 and 8. FIGS. 7 and 8 illustrate the basic principle of short-term spectral attenuation or spectral weighting. Here, first, the power density spectrum of the background noise is estimated. The illustrated method estimates the amount of speech contained in a time / frequency tile using so-called low-level features that provide a measure of the "language" of a signal in a particular frequency range Low level features are low level features in terms of interpretation of their meaning and the cost of their computation.

The audio signal is decomposed into a number of frequency bands by means of a filter bank or a short-time Fourier transform, which is shown at 70 in FIG. Hereupon, as exemplified at 71a and 71b, temporally varying gains for all subbands are computed from such low-level features to attenuate subband signals in proportion to the amount of speech they contain. Suitable low-level features are the spectral flatness measure (SFM) and the 4 Hz modulation energy (4HzME). The SFM measures the degree of tonality of an audio signal and, for a band, results from the quotient of the geometric mean of all spectral values in a band and the arithmetic mean of the spectral components in the band. The 4HzME is motivated by the fact that speech has a characteristic energy modulation peak at about 4 Hz, which corresponds to the average syllable rate of a speaker. FIG. 8 shows a more detailed illustration of the gain calculation block 71a and 71b of FIG. 7. On the basis of a subband Xi, a plurality of different low-level features, ie LLF1,..., LLFn, are calculated. These features are then combined in a combiner 80 to arrive at a gain gi for a subband.

It should be noted that, depending on the implementation, it is not necessarily possible to use low-order features but also any features, such as energy features, etc., which can then be combined in a combiner in accordance with the implementation of FIG a quantitative gain gi, such that each band is variably attenuated (at each time) to achieve speech suppression.

Depending on the circumstances, the method according to the invention can be implemented in hardware or in software. The implementation may be on a digital storage medium, in particular a floppy disk or CD with electronically readable control signals, which may interact with a programmable computer system such that the method is performed. In general, the invention thus also exists in a computer program product with a program code stored on a machine-readable carrier for carrying out the method according to the invention, when the computer program product runs on a computer. In other words, the invention can thus be realized as a computer program with a program code for carrying out the method when the computer program runs on a computer.

Claims

claims

An apparatus for generating a multi-channel signal (10) having a number of output channels greater than a number of input channels of an input signal

(12), where the number of input channels is the same

1 or greater, with the following features:

a high mixer (14) for up-converting the input signal to provide at least one direct signal channel and at least one ambient channel or a modified surround channel;

a voice detector (18) for detecting a portion of the input signal, the direct signal channel or the surround signal channel in which a voice portion occurs; and

a signal modifier (20) for modifying a portion of the surround channel or the input signal corresponding to the portion detected by the speech detector (18) to obtain a modified surround signal channel or a modified input signal in which the speech portion is attenuated or eliminated wherein the portion in the direct channel signal is less or not damped; and

a speaker signal output means (22) for outputting speaker signals in a reproducing scheme using the direct channel and the modified surround channel.

2. Device according to claim 1, in which the loudspeaker signal output device (22) is designed to operate according to a direct / ambient scheme in which each direct channel can be imaged on a separate loudspeaker, and each surround channel is connected to a separate loudspeaker. The loudspeaker signal output device (22) is designed so that, for loudspeaker signals for loudspeakers behind a listener in the display scheme, only the surround channel and not the direct channel are reproduced.

3. A device according to claim 1, wherein the loudspeaker signal output means (22) is adapted to operate according to an in-band scheme in which each direct signal channel is mappable to one or more loudspeakers depending on its position and wherein the loudspeaker signal output means (22). 22) is adapted to add the ambient channel and the direct channel or a portion of the ambient channel or the direct channel, which are intended for a loudspeaker, to obtain a loudspeaker output signal for the loudspeaker.

Apparatus according to any one of the preceding claims, wherein the loudspeaker signal output means is adapted to provide loudspeaker signals for at least three channels which are placeable in front of a listener in the display scheme and to generate at least two channels which are in the playback scheme behind the listener are placeable.

5. Device according to one of the preceding claims,

wherein the speech detector (18) is adapted to operate in block-by-block fashion and to frequency-selectively analyze each time block in a band-wise manner to detect a frequency band for a time block, and

in which the signal modifier (20) is designed to generate a frequency band in such a time block of the ambient signal channel or of the input signal. to the output signal corresponding to the band detected by the speech detector (18).

6. Device according to one of the preceding claims,

wherein the signal modifier is configured to attenuate the ambient channel signal or the input signal or portions of the ambient channel signal or the input signal in a time interval detected by the speech detector (18), and

wherein the high mixer (14) and the loudspeaker signal output means (22) are arranged to generate the at least one direct channel so that the same temporal portion is less or not attenuated so that the direct channel has a speech component that is more noticeable in a display is as a speech component in the modified surround channel signal or in the modified input signal.

7. Apparatus according to any one of the preceding claims, wherein the signal modifier (20) is adapted to highpass pass the at least one surround channel or the input signal when the speech detector (18) has detected a temporal portion in which Voice component occurs, with a cutoff frequency of the high-pass filter is between 400 Hz and 3,500 Hz.

8. Device according to one of the preceding claims,

wherein the speech detector (18) is adapted to detect a temporal occurrence of a speech signal component, and wherein the signal modifier (20) is adapted to detect a fundamental frequency of the speech signal component, and

to selectively attenuate (43) tones in the surround channel or the input signal at the fundamental frequency and harmonics to obtain the modified surround channel signal or the modified input signal.

9. Device according to one of the preceding claims,

wherein the speech detector (18) is adapted to determine a measure of a speech content per frequency band, and

wherein the signal modifier (20) is configured to attenuate a corresponding band of the surround channel according to the measure with a damping factor (72a, 72b), wherein a higher measure results in a higher attenuation factor and a lower measure results in a lower attenuation factor.

10. The device according to claim 9, wherein the signal modifier (20) has the following features:

a time-frequency domain converter (70) for converting the environmental signal or the input signal into a spectral representation;

a damper (72a, 72b) for frequency selective variable attenuation of the spectral representation; and

a frequency-time domain converter (73) for converting the variably attenuated spectral representation in the time domain to obtain the modified ambient channel signal or the modified input signal.

Device according to claim 9 or 10, wherein the speech detector (18) has the following features:

a time-frequency domain converter (42) for providing a spectral representation of an analysis signal;

means for calculating one or more features (71a, 71b) per band of the analysis signal; and

means (80) for calculating a measure of a speech content based on a combination of the one or more features per band.

12. The device according to claim 11, wherein the signal modifier (20) is designed to calculate as characteristics a spectral flatness measure (SFM) or a 4 Hz modulation energy (4HzME).

Apparatus as claimed in any one of the preceding claims, wherein the speech detector (18) is adapted to analyze the environmental channel signal (18c) and wherein the signal modifier (20) is adapted to modify the environmental channel signal (16).

14. An apparatus according to any one of claims 1 to 12, wherein the speech detector (18) is adapted to analyze the input signal (18a) and the signal modifier (20) is adapted to receive the environmental channel signal (16) based on control information - to modify (18d) of the speech detector (18).

Apparatus according to any one of claims 1 to 12, wherein the speech detector (18) is adapted to analyze the input signal (18a) and wherein the signal modifier (20) is adapted to receive the input signal based on control information (18d). from the speech detector (18), and in which the high mixer (14) an ambient channel extractor which is configured to determine the modified ambient channel signal (16 ') on the basis of the modified input signal, wherein the high mixer (14) is further configured to operate on the basis of the input signal (12) at the input of the signal modifier (20 ) to determine the direct channel signal (15).

16. Device according to one of claims 1 to 12,

wherein the speech detector (18) is adapted to analyze the input signal (18a), further comprising a speech analyzer (30) for subjecting the input signal to speech analysis, and

wherein the signal modifier (20) is adapted to modify the surround channel signal (16) based on control information (18d) from the speech detector (18) and based on speech analysis information (18e) from the speech analyzer (30).

17. Device according to one of the preceding claims, wherein the high mixer (14) is designed as a matrix decoder.

18. Device according to one of the preceding claims, wherein the high mixer (14) is designed as a dummy high mixer, the direct channel signal (15), the ambient channel signal (15) only on the basis of the input signal (12), but without additionally transmitted high-mix information 16) or the modified ambient channel signal (16 ') is generated.

19. Device according to one of the preceding claims,

in which the high mixer (14) is designed to carry out a statistical analysis of the input signal (12) in order to calculate the direct channel signal (15), the ambient channel signal (16) or the modified ambient channel signal (16 ').

20. Device according to one of the preceding claims, wherein the input signal is a mono signal with a

Channel and where the output signal is a multichannel signal with two or more channel signals.

21. Device according to one of claims 1 to 19, wherein the high mixer (14) is adapted to receive as input a stereo signal with two stereo channel signals, and wherein the high mixer (14) is further adapted to the ambient channel signal (16). or perform the modified surround channel signal (16 ') based on a cross-correlation calculation of the stereo channel signals.

22. A method for generating a multi-channel signal (10) having a number of output channels which is greater than a number of input channels of an input signal (12), wherein the number of input channels is equal to 1 or greater, comprising the following steps:

Upmixing (14) the input signal to provide at least one direct signal channel and at least one surround channel or a modified surround channel;

Detecting (18) a portion of the input signal, the direct signal channel or the surround signal channel in which a speech component occurs; and

Modifying (20) a portion of the surround channel or the input signal corresponding to the portion detected in the step of detecting (18) to obtain a modified surround signal channel or a modified input signal in which the speech portion attenuates or eliminates is wherein the portion in the direct channel signal is less or not damped; and

Outputting (22) loudspeaker signals in a playback scheme using the direct channel and the modified surround channel.

23. Computer program with a program for carrying out the method according to claim 22, when the program runs on a computer.