EP4646721A1

EP4646721A1 - Audio reproduction system and method

Info

Publication number: EP4646721A1
Application number: EP24700088.8A
Authority: EP
Inventors: Menno Lindwer; Alexander Augusteijn; Ronaldus M. Aarts; Mahesh Makhijani
Original assignee: Snap Inc
Current assignee: Snap Inc
Priority date: 2023-01-04
Filing date: 2024-01-02
Publication date: 2025-11-12
Also published as: KR20250130352A; WO2024146888A1; CN120457480A

Abstract

The present disclosure pertains to an audio reproduction system (SYS) comprising an audio input device (AID), a previously recorded audio source (PRAS), an audio control unit (ACU); and a plurality of audio reproduction devices (ARD1, ARD2). The audio control unit is configured to combine audio input signals (SM, L,R) from the audio input device and from the previously recorded audio source and to provide a respective audio output signal (L", R") to each of the audio reproduction devices. In particular the audio control unit (ACU) is configured to provide the respective audio output signal (L", R") with a respective magnitude weighted in accordance to a respective distance of the audio input device (AID) relative to the respective audio reproduction devices (ARD1).

Description

Title: Audio reproduction system and method

BACKGROUND

The present invention pertains to an audio reproduction system.

The present invention further pertains to an audio reproduction method.

Karaoke is a popular interactive entertainment wherein a participant using a microphone sings along to recorded music wherein the original voice is partly or fully removed. The recorded music may originate from a local data storage or may be streamed from an external source. Analogously the participant may be an instrumentalist, e.g. a guitarist playing an instrumental voice along to recorded music wherein the original instrumental voice is partly or fully removed. The human or instrumental voice of the participant in combination with the modified recorded music is reproduced by an audio reproduction system.

For a realistic experience, it is desired that the virtual position of the participant as perceived by the audience based on the output of the audio reproduction system corresponds to the actual position of the participant.

SUMMARY

According to a first aspect of the present invention an improved audio reproduction system is provided that efficiently adapts a reproduction of the participant’s voice to achieve correspondence of the virtual position with the actual position of the participant.

According to a second aspect of the present invention an audio reproduction method is provided that efficiently adapts a reproduction of the participant’s voice to achieve correspondence of the virtual position with the actual position of the participant. The improved audio reproduction system according to the first aspect comprises an audio input device, a previously recorded audio source, an audio control unit and a plurality of audio reproduction devices. The audio input device is configured to convert an acoustic signal into an audio input signal. Typically the audio input device comprises a microphone that converts the voice of a singer, but alternatively or additionally the audio input device may comprise an element that converts the acoustic output of a musical instrument into an audio input signal. The previously recorded audio source provides a further audio input of previously recorded music that the singer or the player of the musical instrument wishes to accompany. The audio control unit is configured to combine the audio input signals from the audio input device with one or more further audio input signals and to provide a respective audio output signal to each of the audio reproduction devices. The one or more further audio input signals may be streamed from an external source, and/or may originate from a local source, e.g. from a local data storage with pre-recorded audio and/or from an electronic input, e.g. of a keyboard.

For a good user experience it is desired that the virtual position of the participant as perceived by the audience based on the output of the audio reproduction system corresponds to the actual position of the participant. This could be achieved in studio with a pair of microphones at a sufficient distance from the singer. However, this is not possible in a noisy environment which is typical for a bar or club where Karaoke is performed. In such circumstances the voice of the singer or instrumentalist can only be properly registered at a receiving location close to the source of the voice. Therewith it is not possible to properly reproduce the participant’s voice with the audio reproduction devices such that the virtual position and the actual position of the participant correspond.

In the improved audio reproduction system the audio control unit is configured to provide the respective audio output signal with a respective magnitude weighted in accordance to a respective distance of the audio input device relative to the respective audio reproduction devices. In one embodiment the improved audio reproduction system SYS is configured to determine the respective distance with a first and a second correlation unit. The correlation units correlate respective signal components with a reference signal component to determine a respective acoustic signal delay. By measuring the acoustic delay in each path between the audio input device and a respective one of the audio reproduction devices, the position of the participant can be easily determined, therewith optimally using components of the audio reproduction system that would anyhow be required to reproduce the audio input signal.

In one example of this embodiment of the improved audio reproduction system the first correlation unit correlates the audio input signal obtained from the microphone with an auxiliary signal from a first auxiliary microphone arranged near a first one of the audio reproduction devices to determine a delay with which acoustic signals originating in the neighborhood of the microphone are perceived by the first auxiliary microphone. The second correlation unit correlates the audio input signal obtained from the microphone with an auxiliary signal from a second auxiliary microphone arranged near a second one of the audio reproduction devices to determine a delay with which acoustic signals originating in the neighborhood of the microphone are perceived by the second auxiliary microphone.

By determining the delays of the auxiliary signals of the auxiliary microphones the position of the participant can reliably determined, and audio input signal from the participant can be properly weighted in accordance with the actual position of the participant so that the virtual position of the participant corresponds to that actual position. The auxiliary audio input signals are not used for reproduction of the participant’s voice as such, but merely to achieve that the participant’s voice is perceived to originate from a virtual position that corresponds to the actual position of the participant. Therewith the auxiliary microphones do not introduce environmental noise in the participant’s voice as reproduced by the audio reproduction system. In another example of an embodiment of the improved audio reproduction system the audio control unit is configured to include a respective pilot signal in each of the audio output signals for the audio reproduction devices. In this example the first correlation unit determines a delay with which the pilot signal as reproduced by the first one of the audio reproduction devices is received by the audio input device and the second correlation unit determines a delay with which the pilot signal as reproduced by the second one of the audio reproduction device is received by the audio input device.

As in the preceding example the audio reproduction system therewith determines the distances of the audio input device, which indicates the location of the participant (receiving location), to each of the audio reproduction locations where the audio reproduction devices are positioned. It is an advantage of this embodiment that no additional microphones are required to measure the transmission delays.

Due to the fact that the pilot signals are of a predefined nature the corresponding components can be identified in the audio input signal so that they can be separated from the component due to the participant’s voice. Therewith the participant’s voice can be properly reproduced.

The pilot signals are preferably inaudible to the audience. In one example the pilot signals have a frequency in a frequency range that is inaudible by humans. For example the pilot signals are wavelets with a central frequency selected in a range between 25 kHz and 30 kHz. Signals in this range are not audible for a human being, but can be processed by standard audio equipment.

In another example the pilot signals are low power broadband signals having a characteristic phase-frequency relationship or are low power frequency modulated signals. In this case the signals may have frequency components that are as such in the audible range, but that are not audible due to the low level of power. Provided that the phase-frequency relationship or the frequency modulation is in a sufficiently distinct pattern, the components in the audio input signal corresponding to the pilot signals can, despite their low power level, be identified in the audio input signal so that they can be separated from the component due to the participant’s voice. In a still further example the pilot signals comprise a respective Barker sequence. The pilot signal therewith has a strongly peaked autocorrelation function.

In some examples an audio reproduction devices placed at an audio reproduction location actually is a multi-channel audio reproduction devices. In that case it is advantageous if the multi-channel audio reproduction devices are configured to reproduce the received respective audio output signal via each audio channel. An improved audio reproduction method according to the second aspect of the present invention comprises: converting an acoustic signal into an audio input signal at an audio receiving location in a space; receiving one or more further audio input signals; combining the audio input signals to provide a plurality of audio output signals; reproducing each of the audio output signals at a respective audio reproduction location in said space with a respective magnitude weighted in accordance to a respective distance of each of the audio reproduction locations to the audio receiving location.

An embodiment of the improved audio reproduction method comprises correlating respective signal components with a reference signal component to determine a respective acoustic signal delay occurring over each of the respective distances.

In one example said correlating comprises correlating the audio input signal with an auxiliary signal indicative for said acoustic signal perceived at a first one of the audio reproduction locations to determine a delay with which said acoustic signal is perceived at said first one of the audio reproduction locations, and correlating the audio input signal with an auxiliary signal indicative for said acoustic signal perceived at a second one of the audio reproduction locations to determine a delay with which said acoustic signal is perceived at said second one of the audio reproduction locations.

In another example the improved audio reproduction method further comprises: including a respective pilot signal in each of the audio output signals; reproducing a first one of the respective pilot signals at a first one of the audio reproduction locations as a first acoustic signal component; at the audio receiving location converting a version of the first acoustic signal component as perceived at said location into a first delay indication signal component; correlating the first delay indication signal component with the first of the pilot signals to determine a delay with which the first one of the pilot signals as reproduced at the first one of the audio reproduction locations is received at the audio receiving location; reproducing a second one of the respective pilot signals at a second one of the audio reproduction locations as a second acoustic signal component; at the audio receiving location converting a version of the second acoustic signal component as perceived at said location into a second delay indication signal component; correlating the second delay indication signal component with the second of the pilot signals to determine a delay with which the second one of the pilot signals as reproduced at the second one of the audio reproduction locations is received at the audio receiving location.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically shows an embodiment of the improved audio reproduction system;

FIG. 2 shows an exemplary component in an embodiment of the improved audio reproduction system; FIG. 3 shows a further exemplary component in more detail;

FIG. 4 schematically shows another embodiment of the improved audio reproduction system.

DETAILED DESCRIPTION OF EMBODIMENTS

FIG. 1 schematically shows an audio reproduction system SYS comprising an audio input device ML, a first audio reproduction device ARD 1 and a second audio reproduction device ARD2. The audio input device ML is coupled to an audio control unit (ACU, see FIG. 2) to which it provides an audio input signal SML.

FIG. 2 shows an exemplary audio control unit ACU comprising a Karaoke Processor KP and a mixer Mix. The Karaoke Processor KP is configured to receive audio signals L,R from a first and a second channel and to provide modified output L’, R’ at its output channels. In an embodiment the modified output signals L’, R’ are obtained from the audio signals L, R by fully or partially removing thereof the original singing voice. An exemplary approach with which this can be achieved in the Karaoke Processor KP is described by Cano et al. in “Musical source separation: An introduction” IEEE Signal Processing Magazine, 36(l):31-40, 2019.

Alternatively or additionally, one or more other components are fully or partially removed from the received audio signals L,R. For example contributions from accompanying musical instruments, e.g. a guitar, a piano and the like may be fully or partially removed.

The sound mixer Mix mixes the output L’, R’ with the audio input signal SML from the audio input device ML of the singer Pi and combines these to generate the respective audio output signals L”, R” for the audio reproduction devices ARD1, ARD2. Therewith the sound mixer adds the audio input signal SML according to respective weights to the respective audio output signals L”, R” to create the impression that the singer Pi is at an apparent location in a room wherein the audio reproduction system SYS is arranged. In an embodiment the apparent location corresponds to the actual location of the singer Pi.

In the embodiment shown in FIG. 1, each of the audio reproduction devices ARD1, ARD2 is provided with a respective auxiliary microphone Msi, Ms2. The auxiliary microphones render a respective auxiliary signal SMSI, SMS2 that is indicative of the sound of the singer audible at the position of the corresponding audio reproduction device ARD1, ARD2. The sound mixer Mix receives these auxiliary signals and is configured to determine the time delay of the singing voice represented by each of the respective auxiliary signals SMSI, SMS2 relative to the singing voice represented by the audio input signal SML obtained from the microphone ML held by the singer Pi. Due to the fact that the audio input signal SML is substantially determined by the singing voice, it can be relatively easily correlated with each of the auxiliary signals.

FIG. 3 shows an exemplary embodiment, wherein a first correlation unit CR1 correlates the singing voice as represented by the audio input signal SML obtained from the microphone ML held by the singer Pi with the singing voice represented by the respective auxiliary signal SMSI and determines a delay Ati with which the singing voice is received by the first auxiliary microphone Msi. Likewise, a second correlation unit CR2 correlates the singing voice as represented by the audio input signal SML obtained from the microphone ML held by the singer Pi with the singing voice represented by the auxiliary signal SMS2 from the second auxiliary microphone Ms2 and determines a delay Ats with which the singing voice is received by the second auxiliary microphone Ms2. A weight computation unit CW computes respective weights wl, w2 with which the signal originating the microphone held by the singer is weighted for each output channel.

In the embodiment of FIG. 3, the signal SML of the singer’s microphone ML is further delayed by a respective delay unit PHI, PH2 for each channel in accordance with the delay Ati, Ats measured by the correlation units. For example the delay imposed on the microphone signal by the delay units may be proportional to the delay Ati, At2 measured by the correlation units. By giving one of the channels more delay than the other the perception of the location will shift to the channel with the less delay. Even if the signal of the singer’s microphone were reproduced with the same intensity in each of the channels the audience would perceive the signal of the channel with the smaller delay at a higher intensity than the other one. This phenomenon is known as “time intensity trading” and discussed in more detail by R.M. Aarts in:

Time/intensity trading stereophony for (HD)TV and audio applications.

In 14th International Congress on Acoustics (Beijing, China), September 1992. (session L8-3). Accordingly in the first delay unit PHI the signal SML is delayed with a delay dependent on Ati to obtain a first intermediate signal SMLI. The first intermediate signal SMLI is weighted in multiplier Ml with the weight wl to obtain the further first intermediate signal SMLH that is added by adder Al to the modified output signal L’ so as to obtain the audio output signal L” for the first audio reproduction device ARD1. Similarly, in the second delay unit PH2 the signal SML is delayed with a delay dependent on At2 to obtain a second intermediate signal SML2. The second intermediate signal SML2 is weighted in multiplier M2 with the weight w2 to obtain the further second intermediate signal SML22 that is added by adder A2 to the modified output signal R’ so as to obtain the audio output signal R” for the second audio reproduction device ARD2.

It is noted that the auxiliary signals SMSI, SMS2 are only used to determine the position of the singer and to therewith control the weights wl, w2, and or the delays with which the signal of the singer’s microphone is reproduced by the audio reproduction devices. Therewith signal components in the auxiliary signals contributed by other audio sources do not affect the quality with which the singer’s voice is reproduced. The weights and/or delays can be adapted at a relatively low frequency, e.g. a frequency range of about 1 to 10 Hz, so that this process is also substantially insensitive to noise. Nevertheless, if desired noise can be suppressed by auxiliary microphones that are sufficiently direction sensitive and that are directed away from the audio reproduction devices so that they hardly receive the reproduced audio signal. Additionally or alternatively, the correlation units are configured to take into account that the received auxiliary signal comprises a signal component that originates from the corresponding audio reproduction device. This component can be easily identified as it has substantially no delay other than the delay that optionally is explicitly introduced by a delay unit PHI, PH2.

In the example shown, the first audio reproduction device ARD 1 is a stereo device that reproduces the first audio output signal L” at both its channels. Analogously, the second audio reproduction device ARD2 is a stereo device that reproduces the second audio output signal R” at both its channels. In an alternative embodiment the sound mixer Mix is configured to render a pair of respective output signals SIL and SIR to be reproduced by the left channel and the right channel of the first audio reproduction device ARD1 respectively and/or is configured to render a pair of respective output signals S2L and S2R to be reproduced by the left channel and the right channel of the second audio reproduction device ARD2 respectively.

The audio reproduction system SYS presented herewith can be easily extended to a larger plurality of channels. For each additional channel the signal SML originating from the singer’s microphone ML is properly weighted and optionally delayed according to the same approach as described above for a left and a right channel.

Alternatively or additionally, the audio reproduction system SYS can be simply extended for one or more additional singers. In that case also the microphone signal of each singer is correlated with each auxiliary signal to determine the corresponding signal delays so as to determine the corresponding position of the singer and to appropriately adapt the weights and/or the delays with which the microphone signal of the additional is reproduced through the respective channels. In the embodiment of FIG. 4, the audio control unit ACU includes a respective pilot signal SAI, SA2 in each of the audio output signals L”, R” for the audio reproduction devices ARD1, ARD2. The pilot signals SAI, SA2 are reproduced by the audio reproduction devices ARD1, ARD2 as a component AAI, AA2 in their respective reproduced audio signals Al, A2. The singer’s microphone ML outputs a microphone signal SML which is provided to a signal splitter SPL that splits the microphone signal SML into a basic component SMLO, a first delay indication component SMT.DI and a second delay indication component SMLD2. The basic component SMLO, corresponds to the singer’s voice, the first delay indication component SMT.DI corresponds to the first pilot signal SA , reproduced by the first audio reproduction device ARD1 as component AAI and the second delay indication component SMLP2 corresponds to the second pilot signal SA2 reproduced by the second audio reproduction device ARD2 as component AA2.

The first correlation unit CR1 determines a delay Ati with which the component AAI reproduced by the first audio reproduction device ARD1 is received relative to the pilot signal SA . The second correlation unit CR2 determines a delay Ats with which the component AA2 as reproduced by the second audio reproduction device ARD2 is received relative to the pilot signal SA2.

The sound mixer Mix mixes the modified output signals L’, R’ with respective weighted versions of the basic component SMLO corresponding to the singer’s voice to obtain the respective output signals L”, R”. As in the embodiment of FIG. 2, the respective weights used to obtain the respective output signals L”, R” are determined on the basis of the computed delay Ati, Ats respectively.

In an embodiment the pilot signals SA , SA2 with which the delays Ati, Ats are computed have a frequency in a frequency range that is inaudible by humans. The audio control unit ACU may for example periodically generate a wavelet with a first central frequency as the pilot signal SAI and generate a wavelet with a second central frequency as the pilot signal SA2. The signal splitter SPL can easily distinguish the first delay indication component SMLDT and the second delay indication component SMLD2 from the basic component SMLO, using bandpass filters.

Alternatively, the pilot signals SAI, SA2 are low power broadband signals having a characteristic phase-frequency relationship or low power frequency modulated signals which can be distinguished in the microphone signal. In these embodiments it is not necessary that the pilot signals SAI, SA2 are outside the range of audible frequencies as due to their low power they are not audible anyway. Although not audible, these pilot signals can be distinguished in the microphone signal SML due to their characteristic phase-frequency relationship or to their characteristic frequency modulation pattern.

In the claims the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. In this connection it is noted that in the examples presented in the present application the plurality of audio reproduction devices includes a first audio reproduction device audio reproduction device and a second audio reproduction device. However, the present invention is also applicable to audio reproduction systems having a larger plurality of audio reproduction devices. A single component or other unit may fulfill the functions of several items recited in the claims. For example FIG. 3 depicts an embodiment in terms of individual functional units CR1, CR2, CW, etc. Indeed it is conceivable to construct the audio reproduction system with respective components that implement these functional units. Alternatively, it is possible to implement two or more functional units with a single component. For example a single component may implement the correlation functions CR1, CR2 on a time-shared basis, another component may implement the delay units PHI, PH2 on a time-shared basis, and the like. Also mutually different functions may be implemented on a time-shared basis by a single component. Any component may be provided as dedicated hardware specifically having the designated functionality or may be provided as a suitably programmed programmable processor. In further embodiments the functions of the audio reproduction system are implemented as an integrated circuit or implemented by a trained neural network.

The mere fact that certain measures are recited in mutually different claims does not indicate that a combination of these measures cannot be used to advantage.

Any reference signs in the claims should not be construed as limiting the scope.

It is noted that the present invention is analogously applicable to other input sources. For example, an audio input device AID may be responsive for a musical instrument, for example a guitar or a flute instead of a voice input. Also combinations of audio input devices may be provided.

Claims

1. An audio reproduction system (SYS), comprising: an audio input device (AID) to convert an acoustic signal into an audio input signal; an audio control unit (ACU); and a plurality of audio reproduction devices (ARD1, ARD2) wherein the audio control unit is configured to combine an audio input signal (SM, L,R) from the audio input device with one or more further audio input signals and to provide a respective audio output signal (L”, R”) to each of the audio reproduction devices, wherein the audio control unit (ACU) is configured to provide the respective audio output signal (L”, R”) with a respective magnitude weighted in accordance to a respective distance of the audio input device (AID) relative to the respective audio reproduction devices (ARD1).

2. The audio reproduction system (SYS) according to claim 1 comprising a first correlation unit (CR1) and a second correlation unit (CR2), which correlation units (CR1, CR2) correlate respective signal components with a reference signal component to determine a respective acoustic signal delay.

3. The audio reproduction system (SYS) according to claim 2, wherein the first correlation unit (CR1) correlates the audio input signal (SML) obtained from the microphone (ML) with an auxiliary signal (SMSI) from a first auxiliary microphone (Msi) arranged near a first one of the audio reproduction devices (ARD1) to determine a delay (Ati) with which acoustic signals originating in the neighborhood of the microphone (ML) are perceived by the first auxiliary microphone (Msi) and wherein the second correlation unit (CR2) correlates the audio input signal (SML) obtained from the microphone (ML) with an auxiliary signal (SMS2) from a second auxiliary microphone (Mss) arranged near a second one of the audio reproduction devices (ARD2) to determine a delay (Ats) with which acoustic signals originating in the neighborhood of the microphone (ML) are perceived by the second auxiliary microphone (Mss).

4. The audio reproduction system (SYS) according to claim 2, wherein the audio control unit (ACU) is configured to include a respective pilot signal (SAI, SA2) in each of the audio output signals (L”, R”) for the audio reproduction devices (ARD1, ARD2), wherein the first correlation unit (CR1) determines a delay (Ati) with which the pilot signal as reproduced by the first one of the audio reproduction devices (ARD 1) is received by the audio input device (AID) and wherein the second correlation unit (CR2) determines a delay (Ats) with which the pilot signal as reproduced by the second one of the audio reproduction device (ARD2) is received by the audio input device (AID).

5. The audio reproduction system (SYS) according to claim 4, wherein the pilot signals are inaudible to the audience.

6. The audio reproduction system (SYS) according to claim 5, wherein the pilot signals (SAI, SA2) have a frequency in a frequency range that is inaudible by humans.

7. The audio reproduction system (SYS) according to claim 5, wherein the pilot signals (SAI, SA2) are low power broadband signals having a characteristic phase-frequency relationship or are low power frequency modulated signals.

8. The audio reproduction system (SYS) according to claim 5, wherein the pilot signals (SAI, SA2) comprise a Barker sequence.

9. The audio reproduction system according to any of the preceding claims, wherein the audio reproduction devices are multi-channel audio reproduction devices that are configured to reproduce the received respective audio output signal via each audio channel.

10. An audio reproduction method, comprising: converting an acoustic signal into an audio input signal at an audio receiving location in a space; receiving one or more further audio input signals; combining the audio input signals to provide a plurality of audio output signals (L”, R”); reproducing each of the audio output signals (L”, R”) at a respective audio reproduction location in said space with a respective magnitude weighted in accordance to a respective distance of each of the audio reproduction locations to the audio receiving location.

11. The audio reproduction method according to claim 10 comprising correlating respective signal components with a reference signal component to determine a respective acoustic signal delay occurring over each of the respective distances.

12. The audio reproduction method according to claim 11, wherein said correlating comprises correlating the audio input signal (SML) with an auxiliary signal (SMSI) indicative for said acoustic signal perceived at a first one of the audio reproduction locations to determine a delay (Ati) with which said acoustic signal is perceived at said first one of the audio reproduction locations, and correlating the audio input signal (SML) with an auxiliary signal (SMS2) indicative for said acoustic signal perceived at a second one of the audio reproduction locations to determine a delay (Ats) with which said acoustic signal is perceived at said second one of the audio reproduction locations.

13. The audio reproduction method according to claim 11, comprising: including a respective pilot signal (SAI, SA2) in each of the audio output signals (L”, R”); reproducing a first one of the respective pilot signals (SAI) at a first one of the audio reproduction locations as a first acoustic signal component (AAI); at the audio receiving location converting a version of the first acoustic signal component as perceived at said location into a first delay indication signal component (SMLDI); correlating the first delay indication signal component with the first of the pilot signals (SAI) to determine a delay (Ati) with which the first one of the pilot signals as reproduced at the first one of the audio reproduction locations is received at the audio receiving location; reproducing a second one of the respective pilot signals (SAZ) at a second one of the audio reproduction locations as a second acoustic signal component (AA2); at the audio receiving location converting a version of the second acoustic signal component as perceived at said location into a second delay indication signal component (SMLD2); correlating the second delay indication signal component (SMLD2) with the second of the pilot signals (SAZ) to determine a delay (Atz) with which the second one of the pilot signals as reproduced at the second one of the audio reproduction locations is received at the audio receiving location.

14. The audio reproduction method according to claim 13, wherein the reproduced pilot signals are inaudible for a human being.