EP4659244A1 - Refined inter-channel time difference (itd) selection for multi-source stereo signals - Google Patents
Refined inter-channel time difference (itd) selection for multi-source stereo signalsInfo
- Publication number
- EP4659244A1 EP4659244A1 EP24703301.2A EP24703301A EP4659244A1 EP 4659244 A1 EP4659244 A1 EP 4659244A1 EP 24703301 A EP24703301 A EP 24703301A EP 4659244 A1 EP4659244 A1 EP 4659244A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- itd
- threshold
- correlation
- cross
- encoder
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/008—Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
Definitions
- the present disclosure relates generally to communications, and more particularly to encoder/decoder methods and related encoders and decoders supporting encoding and/or decoding.
- BACKGROUND [0002]
- the quality relates to the similarity of the decoded output compared to the source at the input and typically refers to how humans perceive this similarity which is affected by human auditory system.
- the encoding process is made by segmenting the input signal into frames of 10-20ms which are processed to extract parameters in the encoder and the parameters extracted are then encoded. These encoded parameters are then transferred to the decoder where the signal is decoded and reconstructed. It is also common to have some overlap between consecutive frames and then use an analysis/synthesis window combination that enables perfect (or at last nearly perfect) reconstruction of the input signal from the parameters in the decoder when the synthesis window is used in an overlap add reconstruction.
- the system typically includes quantization and encoding processes which introduce impairments in the reconstruction of the signals at the decoder (sometimes also including or being connected to a renderer).
- stereo signals could be treated as two separate mono channels, this would not be as efficient as coding the two channels together, as one unit. The reason is that in a typical stereo signal, the channel pair shows a high degree of similarity, or correlation.
- State-of-the-art stereo coding schemes exploit this correlation by employing, e.g., parametric coding, where a single channel is encoded with high quality and complemented with a parametric description that allows to reconstruct the full stereo image.
- the type of parameters used as stereo parameters are, e.g., inter-channel level difference (or corresponding parameters such as side-gain), inter-channel phase difference, inter- channel time difference, and inter-channel coherence, which are typically derived through analysis in the encoder.
- the encoder may use these (potentially quantized) parameters for a downmix of the input stereo signals to a mono signal being encoded and transmitted to a decoder.
- these quantized parameters received from the encoder may be used for a stereo upmix synthesizing stereo signals based on the mono downmix signal and the stereo parameters.
- Parametric coding techniques are typically beneficial at lower bitrates.
- encoding of the audio signals of more audio channels may give a better quality, e.g., utilizing M/S stereo coding techniques that transform the left and right channel signals into a mid channel and a side channel signals, or directly encoding of the left and right audio channel signals.
- M/S stereo coding techniques that transform the left and right channel signals into a mid channel and a side channel signals, or directly encoding of the left and right audio channel signals.
- combining parametric models with encoding of residual signals may obtain a better coding efficiency, i.e., quality given the constraints in resources.
- the residual signals aim to capture the error between the input audio signals and what is described by the parametric model, so that utilizing the residual signals at the decoder will give an output that better resembles the original audio signals input to the encoder.
- the process of reducing the channel pair into a single channel is often called a down-mix and the resulting channel the down-mix channel.
- the down-mix procedure typically tries to maintain the energy by aligning the signals of the stereo channels based on estimated inter-channel time differences (ITD) and inter-channel phase differences (IPD) before mixing the channels.
- ITD inter-channel time differences
- IPD inter-channel phase differences
- an inter-channel level difference (ILD) or a side gain parameter may also be estimated.
- Figure 1 and Figure 2 show block diagrams of an example of a parametric stereo encoder and decoder.
- a stereo signal is captured by two microphones the distance from the source to the left microphone may be different from the distance to the right microphone as illustrated in Figure 3.
- Speaker 1 talks the different distances (L1 and L2) to the two microphones left and right will generate a small difference in delay and level for the two microphones. These are called inter channel time difference (ITD) and inter channel level difference (ILD) and are important parameters in the parametric stereo model.
- ITD inter channel time difference
- ILD inter channel level difference
- the amount of inter channel level difference (ILD) depends on the source’s direction, distance from the microphones and the distance between the microphones.
- the process is reversed as illustrated in Figure 2. First the bitstream from the encoder is feed to the decoder to create the decoded mono signal. This is then feed to the stereo processing and up-mixing block, where stereo parameters are used to recreate the two stereo channels.
- the ITD parameter may be used to restore the time difference between the left and right channel and the ILD or side gain to recreate the level difference.
- SUMMARY There currently exist certain challenge(s).
- For stationary signals one wants a stable ITD estimate – as there would be a degradation in the output signal if the ITD estimate is noisy or unstable. It is therefore common to use smoothing in the ITD estimate. Such smoothing can be done both in the parameter estimation as such but also in the form of decision logic with a preference of ITD estimates close to previous estimates or using a hangover logic where one keeps the ITD estimate fixed to the last ITD over short periods of time when the input signal does not contain/show any ITD. This usually works for stationary signals where there is one source active at the time.
- a method in an encoder to estimate an inter-channel time difference, ITD, between two waveforms includes obtaining an estimate of a threshold based on a shape of a cross-correlation function between the two waveforms.
- the method includes determining a second threshold based on a maximum value of the first threshold and a non-zero constant.
- the method includes determining at least one lag for which of cross- correlation estimates magnitude is above the second threshold to form a list of potential peak positions.
- the method includes determining an indication of how many true peaks there are and a list of lags of the true peaks.
- the method includes responsive to there being three or more true peaks, keeping three lags having a higher correlation magnitude values than other true peaks and setting lags and correlation magnitude values of the other true peaks to zero to form a list of three candidate ITD estimates.
- the method includes responsive to there being less than three true peaks, keeping the lags and correlation of the true peaks and setting lags and correlation values of remaining positions to zero to form the list of three candidate ITD estimates.
- the method includes estimating the ITD based on the list of three candidate ITD estimates.
- an encoder to estimate an inter-channel time difference, ITD, between two waveforms is configured to perform operations comprising obtaining an estimate of a threshold based on a shape of a cross-correlation function between the two waveforms. Determining a second threshold based on a maximum value of the first threshold and a non-zero constant, and determining at least one lag for which of cross-correlation estimates magnitude is above the second threshold to form a list of potential peak positions.
- the encoder being configured to perform determining an indication of how many true peaks there are and a list of lags of the true peaks.
- Certain embodiments may provide one or more of the following technical advantage(s).
- Figure 1 is a block diagram of a parametric stereo encoder
- Figure 2 is a block diagram of a parametric stereo decoder
- Figure 3 is an illustration of inter channel time difference (ITD) and inter-channel level difference (ILD);
- Figure 4 is an illustration of ITD estimates of separate speakers and combined with overlap;
- Figure 5 is an illustration of an example of an operating environment of an encoder according to some embodiments;
- Figure 6 is a flow chart illustrating frame processing steps for a parametric stereo encoder;
- Figure 7 is a flow chart illustrating ITD estimation with hangover logic;
- Figure 8 is a flow chart illustrating ITD estimation with detection of multiple sources according to some embodiments;
- Figure 9-10 are flow charts illustrating operations an encoder performs according to some embodiments;
- Figure 11 is a flow chart illustrating peak pruning according to some embodiments;
- Figure 12 is an illustration of how the various embodiments handle multiple
- CCF cross- correlation function
- the cross-correlation is typically estimated as [0049]
- the inter-channel coherence or inter-channel correlation (ICC) is conventionally obtained as the maximum of the CCF which is normalized by the signal energies as follows: [0050]
- the time lag ⁇ corresponding to the ICC is determined as the ITD between the channels ⁇ and ⁇ .
- DFT discrete Fourier transform
- GCC cross- correlation
- the phase transform is basically the absolute value of each frequency coefficient, i.e., [0054] This weighting will thereby whiten the cross-spectrum such that the power of each component becomes equal.
- the phase transformed GCC (GCC-PHAT) becomes just the Kronecker delta function ⁇ ( ⁇ ⁇ ⁇ ⁇ ), i.e., [0055]
- the encoding process is done on time segments called frames. Common lengths of these segments are 10 or 20 ms.
- the coding parameters like the ITD are estimated at the encoding side on a per frame basis and are transmitted to the decoder.
- the filter coefficient ⁇ can be fixed but it may also be adaptive.
- One example is to use a spectral flatness measure, ⁇ , calculated on the left or right input signal as the filter coefficient, as described in paragraphs 0079-0081 of U.S.
- Patent Application Publication No. 20200194013A1 which can be determined in accordance with: This measure will have the range 0.0 – 1.0 where a higher value would indicate a flatter spectrum. Using this coefficient may improve the robustness and accuracy of the ITD estimation. [0058] Ideally one would just have one peak in the cross correlation estimate ⁇ ⁇ ⁇ ⁇ ⁇ [ ⁇ ] and it would be close to 1.0 if there is correlation between the channels and 0.0 if it is not. Finding the peak would be just a matter of looking for the highest peak – and one would get a reading of the estimated time delay of the channels.
- a feature that measures the energy balance between the channels will show if the source is to the right or to the left.
- One such feature is the side gain parameter which represents the projection of a side signal ⁇ ( ⁇ ) on a downmix signal ⁇ ( ⁇ ).
- the scaling factor of 1 ⁇ 2 may not always be used. Thus, other scaling factors may be used.
- Side gain parameters may also be estimated separately for certain frequencies (or frequency bands) of the signal, e.g., in the Discrete Fourier Transform (DFT) domain.
- DFT Discrete Fourier Transform
- a positive time lag may be selected. This may e.g., be done if there are ITD candidates, peaks of the cross-correlation function, that are rather close (in respect of magnitude, not time lag). It may result in not selecting the time lag corresponding to the peak of highest magnitude (i.e., highest estimated correlation), instead another one is used which is more consistent with the estimated ILD. Further details of embodiments of the method is shown in U.S. Patent No.10,002,614 and U.S. Patent No.10,311,881.
- Figure 4 illustrates the problem with such existing solutions.
- Figure 4 is based on a synthesized example where two sources with different directions are processed both individually and in combination.
- the top shows the energy of the separate speakers (labeled in Figure 4 as 1st and 2nd), the middle contains the ITD estimates for the separate files.
- FIG. 4 shows the ITD track for the combined source file (where the 1st and 2nd signals are added before being sent to the codec). Conventional technology was used for the ITD estimations.
- Figure 4 illustrates how the codec misses the initial part of the 2nd speaker (start of overlap region) even though the 2nd speaker is clearly the louder of the two. There are also other regions where the 1st speaker is louder than the 2nd but the ITD used is the one for the 2nd speaker. These "misses" in ITD estimates used result in a degraded quality with diffuse of faulty position for the speaker or that the speaker appears in the faulty position and then "glides" to the correct one. [0071] Methods described in U.S.
- Patent Nos.10,002,614 and 10,311,881 may be used to improve the ITD estimation in certain cases.
- a selection based on the ILD results in a suboptimal selection of the ITD among ITD candidates, i.e., peaks of a cross-correlation function.
- the cross-correlation function may typically give some information on the number of sources and their direction(s) even though some analysis may be necessary. To extract such information, one needs to further process the cross-correlation estimate to first decide how many source directions are present. If there are more than one source direction present, further processing is needed to find which of them is dominant and should be used for the ITD estimate.
- the dominant source direction is different from the last used (e.g., for the previous frame) one can, if the new direction is strong enough (in terms of correlation), decide to immediately change the ITD estimate to use for the current frame.
- Some embodiments provide a number of identified sources and vectors of magnitudes in decreasing order for up to the three strongest ITD candidates and a corresponding vector with their lags to determine which source direction to use.
- Other embodiments improve the robustness of determining the sign of the ITD based on the level difference between signals of the stereo channels by a decision on whether or not to utilize the level difference information, e.g., side gain parameters, in determining the ITD value.
- ⁇ ( ⁇ ) functions the functions are used to only look at the sign of the value ⁇ .
- a dead zone is used which can simplify the decision logic.
- the advantage with this function is that one can check for different signs and also handle a dead zone around zero where the sign would be reported as 0.
- Figure 5 illustrates an example of an operating environment in which the various embodiments of the present disclosure may be implemented.
- the encoder 502 receives data, such as an audio file, to be encoded from an entity through network 504, such as a host 506, and/or from storage 508.
- the encoder 502 in various embodiments is a parametric stereo encoder such as, e.g., the parametric stereo encoder of Figure 1.
- the host 506 may communicate directly to the encoder 502.
- the encoder 502 encodes the audio file as described herein and either stores the encoded audio file in storage 508 or transmits the encoded audio file to a decoder 512 via network 510.
- the decoder 512 is various embodiments is a parametric stereo decoder such as, e.g., the parametric stereo decoder illustrated in Figure 2.
- the decoder 512 decodes the audio file and transmits the decoded audio file to an audio player 514 for playback.
- the audio player 514 may be or be comprised in a user equipment, a terminal, a mobile phone, and the like.
- the host 506 may transmit encoded audio files to the decoder 512 via network 510.
- the overall processing of the parametric stereo involves segmenting the input in to frames of 20ms of audio samples. Note that this refers to the amount of new data consumed by each frame processing loop. The actual data analysis for a frame is usually longer, by also including some earlier frame(s).
- the basic principle is to process each frame by deriving the ITD estimate, time align the stereo channels before down mixing it to a mono channel that is encoded using a state-of-the-art mono speech codec and then sent to a receiver.
- This processing scheme for a typical parametric stereo solution utilizing ITD estimates is shown in Figure 6. Note that additional processing blocks may be included if other stereo parameters are included in the parametric stereo model.
- the process is reversed.
- the mono bitstream is decoded and the mono signal is then up-mixed using decoded stereo parameters and the resulting output can be used for audio output.
- the various embodiments described below shall focus on ITD estimation in the encoder.
- ITD estimation There are two partly competing requirements on the system about ITD estimation. It is important to use the correct ITD estimate for the down mix, as this would prevent any signal cancelation that would happen if the two channels in the input signal are out of phase. At the same time in the decoder, the ITD contributes to the positioning of the rendered output where rapid variations can be perceived as disturbing.
- ITD Hangover logic keeps the ITD estimate even during short pauses in normal speech. Further details of the ITD Hangover logic according to an embodiment may be found in U.S. Patent No.10,832,689. [0085] While the processing shown in Figure 7 works well for cases where a single source is active at a time, there would be problems if multiple sources are active at the same time as shown in Figure 4. Here it becomes important to determine which of the speakers is the dominant part and use the ITD corresponding to that speaker position.
- the peak pruning described herein is used to determine the number of source directions available as the cross-correlation function can show many peaks – showing that the one source assumption is not true, the use of a stationary ITD would not be optimal and it should be investigated whether one should change the ITD to handle the direction of the new source.
- One feature from the peak pruning is the number of detected peaks ⁇ ( ⁇ ) , for the current frame, ⁇ . Some other features are used to judge the importance of the three largest peaks found.
- the first step in the multisource detection and ITD selection is to determine if there are multiple sources in the current input. This information is present in the cross correlation estimate, but additional processing is needed to make the determination.
- Figure 9 illustrates operations the encoder 502 performs in the multisource detection and ITD selection.
- the encoder 502 obtains an estimate of a threshold based on a shape of a cross-correlation function between two waveforms. For example, using the gcc_phat(),a threshold based on the shape of the cross correlation can be estimated - see U.S. Patent Application Publication No.20200194013, paragraphs 0097-0111 for details of the threshold.
- the encoder 502 determines a second threshold based on a maximum value of the first threshold and a non-zero constant. For example, a second threshold based on the one from gcc_phat() is formed by taking the maximum value of the one from gcc_phat() and a non-zero constant. In some embodiments, a nonzero constant in the range of 0.1 to 0.2 can be used. In one embodiment, a value of 0.12 is used for the non-zero constant. [0093] In block 905, the encoder 502 determines at least one lag for which of cross- correlation estimate magnitude is above the second threshold to form a list of potential peak positions.
- the second threshold is used to find the lags for which of the cross-correlation estimate magnitude is above said threshold, these lags form a list of potential peak positions.
- the encoder 502 determines an indication of how many true peaks there are and a list of lags of the true peaks. Then the list of potential peaks is evaluated to find the lags that are true peaks.
- Figure 10 illustrates a way to find lags that are true peaks. Turning to Figure 10, in block 1001, the encoder 502 determines that a possible peak is a true peak when a cross-correlation magnitude value is larger than its neighbors (the positions lag-1 and lag+1).
- peaks that are too close to each other.
- the encoder 502 removes possible peaks that are too close to each other by determining when lags between the possible peaks are less than a constant apart (e.g., 10), keeping only a lag corresponding to a largest cross correlation estimate magnitude value. Thus, only the lags having the larger of the cross correlation estimate magnitude values are kept. At the end of this process, an indication of how many peaks there are, and a list of their lags is obtained.
- a constant apart e.g. 10
- the encoder 502 responsive to there being three or more true peaks, keeps three lags having a higher cross correlation estimate magnitude value than other true peaks and sets lags and cross correlation estimate magnitude values of the other true peaks to zero to form a list of three candidate ITD estimates. Thus, the three lags with the largest cross correlation estimate magnitude values are kept.
- the encoder 502 responsive to there being less than three true peaks, keeps the lags and cross correlation estimate magnitude value of the true peaks and sets lags and cross correlation estimate magnitude values of what would correspond to the other true peaks (if there were at least three true peaks) to zero to form the list of three candidate ITD estimates.
- the encoder 502 estimates the ITD based on the list of three candidate ITD estimates and a consistency measure between estimated ITD values and level different parameters. The process is also illustrated in the flow chart in Figure 11, which illustrates operations to determine a number of peaks and lag and cross correlation estimate magnitude values for the three largest.
- the encoder 502 estimates the ITD in various embodiments is described below.
- ITD estimation e.g., due to smoothing of the cross- correlation function over time
- ILD or side gain parameters may earlier indicate the change of a dominant source.
- the level difference may indicate a preferred direction even when the cross-correlation function does not yet show this as a dominant source.
- the various embodiments provide a method selecting whether to utilize the level difference information for ITD estimation or not based on a consistency measure between estimated ITD values and level difference parameters.
- the method further utilizes the position of the ITD candidates, i.e., their position in the cross-correlation function, and/or information on whether the signals encoded contains speech or music.
- the position of the ITD candidates i.e., their position in the cross-correlation function, and/or information on whether the signals encoded contains speech or music.
- ⁇ _ ⁇ 0 ⁇ ( ⁇ , ⁇ h ⁇ ) makes it possible to form more compact logic that handles sign changes and minimum magnitudes of new ⁇ ⁇ values. By combining the dead zone test with the sign, one also reduces the complexity of the logic.
- AR autoregressive
- the filtered energy measure ⁇ is a state variable being maintained between frames. For the processing of the next frame ⁇ + 1, ⁇ would prior to the filtering process comprise the filtered energy measure obtained in frame ⁇ .
- the state parameter ⁇ may be initialized to zero.
- FIG. 12 illustrates how the various embodiments can handle overlapping sources as compared the techniques illustrated in Figure 4.
- the ITD ORG is the techniques illustrated in Figure 4
- ITD CUT illustrates how the various embodiments detect multiple sources.
- the first and second in the Energy (dB) indicate the two sources.
- the ITD selection includes a decision on whether to utilize the level difference information for ITD estimation or not based on a consistency measure between estimated ITD values and level difference parameters, in this case the side gain parameter. This is shown in Figure 13, where in block 1301, the encoder 502 determines whether or not to utilize level difference information for ITD estimation based on a consistency measure between estimated ITD values and level difference parameters.
- the encoder 502 determines the consistency measure.
- the updated consistency measure ⁇ _ ⁇ _ ⁇ may be obtained in accordance with where ⁇ _ ⁇ _ ⁇ ⁇ is the current consistency measure that is being updated.
- ⁇ ⁇ ( ⁇ ) is the side gain parameter
- ⁇ [0] is the ITD candidate corresponding to the largest correlation among the ITD candidates
- the consistency measure ⁇ _ ⁇ _ ⁇ ⁇ is a state variable being maintained between frames, which corresponds to the consistency measure determined in the previous frame ⁇ ⁇ 1.
- ⁇ _ ⁇ _ ⁇ ⁇ would comprise ⁇ _ ⁇ _ ⁇ obtained in frame ⁇ .
- the state parameter ⁇ _ ⁇ _ ⁇ ⁇ may be initialized to zero.
- the consistency measure is only updated this way when there is just a single source identified, i.e., there is only a single ITD candidate. Further it may be determined whether the ITD candidate is close to a lag of zero, and the level difference parameter is also close to zero, indicating the signals of the audio channels are similar in level. If so, the parameters can be seen consistent, and the consistency measure may be increased. However, if there is not just a single ITD candidate or if only one of the ITD candidate and the level difference parameter is close to zero, the consistency measure may not reliably be updated.
- the encoder 502 in block 1403 applies a forgetting factor to the consistency measures such that the consistency measure slowly approaches zero when there is no information to reliably update the consistency measure.
- Figure 15 shows a flow diagram for the update of ⁇ _ ⁇ _ ⁇ .
- Figure 16 illustrates an embodiment of using a consistency threshold.
- the encoder 502 compares the consistency measure to a consistency threshold.
- the encoder 502 responsive to the consistency measure being above the consistency threshold, uses the level difference parameter to select among positive and negative ITD candidates.
- the encoder 502 responsive to the consistency measure being below the consistency threshold, does not use the level difference parameter to select among positive and negative ITD candidates. For example, if the consistency measure is above a certain threshold ⁇ _ ⁇ _ ⁇ , e.g., being above 0.6, the level difference parameter (in this case the side gain) is considered to be relevant for the ITD estimation and will be used to select among positive and negative ITD candidates as outlined below. [0110] However, in certain embodiments as illustrated in block 1701 of Figure 17, a further condition may be added, where the level difference parameter is considered to be relevant for the ITD estimation only when speech signals are identified, or similarly when not music is identified. Blocks 1703 and 1705 are the same as blocks 1603 and 1605 of Figure 16.
- the current frame's down mix energy estimate nrgDMXdB should be sufficiently larger than the previous frames smoothed long term energy estimate that is ⁇ – ⁇ > ⁇ _ ⁇ _ ⁇ for the level difference parameter to be considered to be relevant.
- the D_DMX_THR threshold depends on the used AR-coefficient (DMXalpha) and in some embodiments, may be in the range 10 - 20, for example 15. In the description hereinbelow, the value of 15 will be used to describe the various embodiments.
- Blocks 1803 and 1805 are the same as blocks 1603 and 1605 of Figure 16. [0111] If the analysis of the cross-correlation peaks has revealed that there is more than one detected peak and that there is sufficient cross-correlation detected in the input, the logic proceeds to decide if the ITD estimate should be updated and how.
- the level difference parameter is considered to be relevant for the ITD estimation, and if there are more than one ITD candidate identified and none of them are considered close to zero (e.g., using the same criteria as for the update of the consistency measure): 1. Check whether the first and second ITD candidate have different signs (i.e. ⁇ ( ⁇ [ 0 ]) ⁇ ⁇ ( ⁇ [ 1 ] ), where ⁇ () is a function returning the sign of the input), and the corresponding cross correlation values (i.e. ⁇ [0] and ⁇ [1]) are not differing more than a certain threshold ⁇ _ ⁇ _ ⁇ (e.g.
- any of the following conditions are present: is there a certain level difference between the signals of the channels, i.e.
- ITD Update 1 in Figures 19 and 22-25. 2. If the conditions of 1 are false, check whether there are more than two ITD candidates, and if so check if the first and the third ITD candidate have different signs (i.e.
- ITD Update 2 select an ITD being the first ITD candidate ⁇ [0] if ( ⁇ ⁇ ⁇ 0 and ⁇ [0] > 0) or ( ⁇ ⁇ > 0 and ⁇ [0] ⁇ 0), otherwise select an ITD being ⁇ [ 2 ] if ⁇ ⁇ ⁇ 0 and ⁇ [ 2 ] > 0 ⁇ ⁇ ⁇ > 0 and ⁇ [ 2 ] ⁇ 0. This is illustrated as ITD Update 2 in Figures 20 and 22-25.
- ITD being the first ITD candidate ⁇ [0] if ⁇ ⁇ ⁇ 0 and ⁇ [ 0 ] > 0 ⁇ ⁇ ⁇ > 0 and ⁇ [ 0 ] ⁇ 0 or if ⁇ ⁇ and lags[0] are sufficiently close to 0, otherwise select and ITD being ⁇ [ 1 ] if ⁇ > 0.4 and any of ⁇ ⁇ ⁇ 0 and ⁇ [ 1 ] > 0 ⁇ ⁇ ⁇ > 0 and ⁇ [1] ⁇ 0. This is illustrated as ITD Update 3 in Figures 21-25.
- ITD_XCS_SUM_THR 0.525
- ITD_NON_ZERO 5
- ITD_XCS_THR 0.4
- SG_THR 0.1
- SG THR2 0.09
- IMP_MAG_THR 0.55
- NZ_EPS 0.12
- REL_MAG_THR 0.75
- ZERO 0.0
- APPROX_ZERO 0.025f
- Cond_A -sign_m0p(Gside, APPROX_ZERO) ⁇ sign_m0p(prev_Gside, ZERO) ⁇ abs(Gside- prev_Gside) > SG_THR2
- Cond_B -sign_m0p(Gside, APPROX_ZERO) ⁇ sign_m0p(prev2_Gside,ZERO) ⁇ abs(Gside- prev2_Gside) > SG_THR2
- Cond_C Not music && itd_
- Figure 23 is a flow chart of an alternative embodiment of the decision logic for the multi-source detection and ITD selection logic where the conditions Cond_A and Cond_B are handled separately without restrictions for abs(xcs[1] – xcs[0]).
- ITD_XCS_SUM_THR 0.525
- ITD_NON_ZERO 5
- ITD_XCS_THR 0.4
- SG_THR 0.1
- SG THR2 0.09
- IMP_MAG_THR 0.55
- NZ_EPS 0.12
- REL_MAG_THR 0.75
- ZERO 0.0
- APPROX_ZERO 0.025f
- Cond_A -sign_m0p(Gside, APPROX_ZERO) ⁇ sign_m0p(prev_Gside, ZERO) ⁇ abs(Gside- prev_Gside) > SG_THR2
- Cond_B -sign_m0p(Gside, APPROX_ZERO) ⁇ sign_m0p(prev2_Gside,ZERO) ⁇ abs(Gside- prev2_Gside) > SG_THR2
- Cond_C Not music && itd_
- ITD_XCS_SUM_THR 0.525
- ITD_NON_ZERO 5
- ITD_XCS_THR 0.4
- SG_THR 0.1
- SG THR2 0.09
- IMP_MAG_THR 0.55
- NZ_EPS 0.12
- REL_MAG_THR 0.75
- D_DMX_THR 15
- ZERO 0.0
- APPROX_ZERO 0.025f
- Cond_A -sign_m0p(Gside, APPROX_ZERO) ⁇ sign_m0p(prev_Gside, ZERO) ⁇ abs(Gside- prev_Gside) > SG_THR2
- Cond_B -sign_m0p(Gside, APPROX_ZERO) ⁇ sign_m0p(prev2_Gside,ZERO) ⁇ abs(Gside- prev2_Gside) > SG_THR2
- Cond_A -sign_m0
- Figure 25 is a flow chart of an alternative embodiment of the decision logic for the multi-source detection and ITD selection logic where the conditions Cond_A and Cond_B are handled separately without restrictions for abs(xcs[1] – xcs[0]) similar to Figure 23 with the addition of the energy estimation of the downmix channel for the energy tracking of the input signal.
- ITD_XCS_SUM_THR 0.525
- ITD_NON_ZERO 5
- ITD_XCS_THR 0.4
- SG_THR 0.1
- SG THR2 0.09
- IMP_MAG_THR 0.55
- NZ_EPS 0.12
- REL_MAG_THR 0.75
- D_DMX_THR 15
- ZERO 0.0
- APPROX_ZERO 0.025f
- Cond_A -sign_m0p(Gside, APPROX_ZERO) ⁇ sign_m0p(prev_Gside, ZERO) ⁇ abs(Gside- prev_Gside) > SG_THR2
- Cond_B -sign_m0p(Gside, APPROX_ZERO) ⁇ sign_m0p(prev2_Gside,ZERO) ⁇ abs(Gside- prev2_Gside) > SG_THR2
- Cond_C Not
- the ITD hangover could be performed before the ITD hangover, as shown in Figure 26. If done before the ITD hangover logic one may need to also update the ITD hangover logic so that it considers information gained in the multi- source detection step. [0124] Another option would be to allow the multi-source detection to replace ITD hangover logic in cases where multiple sources are detected, as illustrated in Figure 27. [0125] It would be possible to use an alternative way to calculate the filtered energy estimate. One option would be to use different AR-coefficients if the current frame value nrgDMXdB is large than nrgDMXdBst and another if it is lower.
- FIG 28 shows an audio encoder 502 (e.g., a parametric stereo encoder) in accordance with some embodiments where the audio encoder 502 is implemented as a stand- alone device.
- an audio encoder refers to a device capable, configured, arranged and/or operable to encode objects and communicate with network nodes, encoders, and/or decoders.
- Examples of an audio encoder include, but are not limited to, a smart phone, mobile phone, cell phone, voice over IP (VoIP) phone, wireless local loop phone, desktop computer, personal digital assistant (PDA), wireless cameras, gaming console or device, storage device, playback appliance, wearable terminal device, wireless endpoint, mobile station, tablet, laptop, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), smart device, wireless customer-premise equipment (CPE), vehicle-mounted or vehicle embedded/integrated wireless device, etc.
- VoIP voice over IP
- PDA personal digital assistant
- Wi-Fi wireless local loop phone
- PDA personal digital assistant
- gaming console or device storage device
- playback appliance wearable terminal device
- wireless endpoint mobile station
- mobile station tablet
- laptop laptop-embedded equipment
- LME laptop-mounted equipment
- CPE wireless customer-premise equipment
- An audio encoder may support device-to-device (D2D) communication, for example by implementing a 3GPP standard for sidelink communication, Dedicated Short-Range Communication (DSRC), vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), or vehicle- to-everything (V2X).
- D2D device-to-device
- DSRC Dedicated Short-Range Communication
- V2V vehicle-to-vehicle
- V2I vehicle-to-infrastructure
- V2X vehicle- to-everything
- an encoder may not necessarily have a user in the sense of a human user who owns and/or operates the relevant device.
- the audio encoder 502 includes processing circuitry 2802 that is operatively coupled via a bus 2804 to an input/output interface 2806, a power source 2808, a memory 2810, a communication interface 2812, and/or any other component, or any combination thereof.
- Certain encoders may utilize all or a subset of the components shown in Figure 28. The level of integration between the components may vary from one encoder to another encoder. Further, certain encoders may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc.
- the processing circuitry 2802 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory 2810.
- the processing circuitry 2802 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above.
- the processing circuitry 2802 may include multiple central processing units (CPUs).
- the input/output interface 2806 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and/or output devices.
- Examples of an output device include a speaker, a sound card, a video card, a display, a monitor, an actuator, an emitter, a smartcard, another output device, or any combination thereof.
- An input device may allow a user to capture information into the audio encoder 502.
- Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like.
- the presence- sensitive display may include a capacitive or resistive touch sensor to sense input from a user.
- a sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof.
- An output device may use the same type of interface port as an input device.
- a Universal Serial Bus (USB) port may be used to provide an input device and an output device.
- the power source 2808 is structured as a battery or battery pack.
- Other types of power sources such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used.
- the power source 2808 may further include power circuitry for delivering power from the power source 2808 itself, and/or an external power source, to the various parts of the audio encoder 502 via input circuitry or an interface such as an electrical power cable. Delivering power may be, for example, for charging of the power source 2808.
- Power circuitry may perform any formatting, converting, or other modification to the power from the power source 2808 to make the power suitable for the respective components of the audio encoder 502 to which power is supplied.
- the memory 2810 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read- only memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth.
- the memory 2810 includes one or more application programs 2814, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data 2816.
- the memory 2810 may store, for use by the audio encoder 502, any of a variety of various operating systems or combinations of operating systems.
- the memory 2810 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a USIM and/or ISIM, other memory, or any combination thereof.
- RAID redundant array of independent disks
- HD-DVD high-density digital versatile disc
- HDDS holographic digital data storage
- DIMM external mini-dual in-line memory module
- SDRAM synchronous dynamic random access memory
- SDRAM synchronous dynamic random access memory
- the UICC may for example be an embedded UICC (eUICC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’
- the memory 2810 may allow the audio encoder 502 to access instructions, application programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data.
- An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 2810, which may be or comprise a device-readable storage medium.
- the processing circuitry 2802 may be configured to communicate with an access network or other network using the communication interface 2812.
- the communication interface 2812 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna 2822.
- the communication interface 2812 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., a UE or a network node in an access network).
- Each transceiver may include a transmitter 2818 and/or a receiver 2820 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth).
- the transmitter 2818 and receiver 2820 may be coupled to one or more antennas (e.g., antenna 2822) and may share circuit components, software or firmware, or alternatively be implemented separately.
- communication functions of the communication interface 2812 may include cellular communication, Wi-Fi communication, LPWAN communication, data communication, voice communication, multimedia communication, short- range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof.
- GPS global positioning system
- Communications may be implemented in according to one or more communication protocols and/or standards, such as IEEE 802.11, Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol/internet protocol (TCP/IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), QUIC, Hypertext Transfer Protocol (HTTP), and so forth.
- an audio object renderer may provide an output of decoded data, through its communication interface 2812, via a wireless connection to a network node.
- An audio encoder when in the form of an Internet of Things (IoT) device may be a device for use in one or more application domains, these domains comprising, but not limited to, city wearable technology, extended industrial application and healthcare.
- IoT device are a device which is or which is embedded in: a connected refrigerator or freezer, a TV, a connected lighting device, an electricity meter, a robot vacuum cleaner, a voice controlled smart speaker, a home security camera, a thermostat, an electrical door lock, a connected doorbell, an autonomous vehicle, a surveillance system, a weather monitoring device, a vehicle parking monitoring device, an electric vehicle charging station, a smart watch, a fitness tracker, a head-mounted display for Augmented Reality (AR) or Virtual Reality (VR), a wearable for tactile augmentation or sensory enhancement.
- AR Augmented Reality
- VR Virtual Reality
- a decoder in the form of an IoT device comprises circuitry and/or software in dependence of the intended application of the IoT device in addition to other components as described in relation to the audio encoder 502 shown in Figure 28.
- Figure 29 is a block diagram of a host 506 in accordance with various aspects described herein.
- the host 506 may be or comprise various combinations hardware and/or software, including a standalone server, a blade server, a cloud-implemented server, a distributed server, a virtual machine, container, or processing resources in a server farm.
- the host 506 may provide one or more services to one or more UEs.
- the host 506 includes processing circuitry 2902 that is operatively coupled via a bus 2904 to an input/output interface 2906, a network interface 2908, a power source 2910, and a memory 2912.
- Other components may be included in other embodiments. Features of these components may be substantially similar to those described with respect to the devices of previous figures, such as Figure 28, such that the descriptions thereof are generally applicable to the corresponding components of host 506.
- the memory 2912 may include one or more computer programs including one or more host application programs 2914 and data 2916, which may include user data, e.g., data generated by a UE for the host 506 or data generated by the host 506 for a UE.
- Embodiments of the host 506 may utilize only a subset or all of the components shown.
- the host application programs 2914 may be implemented in a container-based architecture and may provide support for video codecs (e.g., Versatile Video Coding (VVC), High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), MPEG, VP9) and audio codecs (e.g., FLAC, Advanced Audio Coding (AAC), MPEG, G.711, EVS, IVAS), including transcoding for multiple different classes, types, or implementations of UEs (e.g., handsets, desktop computers, wearable display systems, heads-up display systems).
- VVC Versatile Video Coding
- HEVC High Efficiency Video Coding
- AVC Advanced Video Coding
- MPEG MPEG
- VP9 video codecs
- audio codecs e.g., FLAC, Advanced Audio Coding (AAC), MPEG, G.711, EVS, IVAS
- UEs e.g., handset
- the host application programs 2914 may also provide for user authentication and licensing checks and may periodically report health, routes, and content availability to a central node, such as a device in or on the edge of a core network. Accordingly, the host 506 may select and/or indicate a different host for over-the-top services for a UE.
- the host application programs 2914 may support various protocols, such as the HTTP Live Streaming (HLS) protocol, Real-Time Messaging Protocol (RTMP), Real-Time Streaming Protocol (RTSP), Dynamic Adaptive Streaming over HTTP (MPEG-DASH), etc.
- FIG. 30 is a block diagram illustrating a virtualization environment 3000 in which functions implemented by some embodiments of the audio encoder 502 or components of the audio encoder 502 may be virtualized.
- virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources.
- virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components.
- VMs virtual machines
- hardware nodes such as a hardware computing device that operates as a decoder, encoder, network node, UE, core network node, or host.
- the virtual node does not require radio connectivity (e.g., a core network node or host)
- the node may be entirely virtualized.
- Applications 3002 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment 3000 to implement some of the features, functions, and/or benefits of some of the embodiments disclosed herein.
- Hardware 3004 includes processing circuitry, memory that stores software and/or instructions executable by hardware processing circuitry, and/or other hardware devices as described herein, such as a network interface, input/output interface, and so forth.
- Software may be executed by the processing circuitry to instantiate one or more virtualization layers 3006 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs 3008A and 3008B (one or more of which may be generally referred to as VMs 3008), and/or perform any of the functions, features and/or benefits described in relation with some embodiments described herein.
- the virtualization layer 3006 may present a virtual operating platform that appears like networking hardware to the VMs 3008.
- the VMs 3008 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer 3006.
- a virtualization layer 3006 may be implemented on one or more of VMs 3008, and the implementations may be made in different ways.
- Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV).
- NFV network function virtualization
- NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment.
- a VM 3008 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine.
- Each of the VMs 3008, and that part of hardware 3004 that executes that VM forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs 3008 on top of the hardware 3004 and corresponds to the application 3002.
- Hardware 3004 may be implemented in a standalone network node with generic or specific components. Hardware 3004 may implement some functions via virtualization.
- hardware 3004 may be part of a larger cluster of hardware (e.g., such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration 3010, which, among others, oversees lifecycle management of applications 3002.
- hardware 3004 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station.
- some signaling can be provided with the use of a control system 3012 which may alternatively be used for communication between hardware nodes and radio units.
- a control system 3012 which may alternatively be used for communication between hardware nodes and radio units.
- the computing devices described herein e.g., decoders, audio object renderers, encoders, hosts
- other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and/or software needed to perform the tasks, features, functions and methods disclosed herein.
- Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and/or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination.
- processing circuitry may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and/or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination.
- computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components.
- a communication interface may be configured to include any of the components described herein, and/or the functionality of the components may be partitioned between the processing circuitry and the communication interface.
- non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.
- some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer- readable storage medium.
- some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner.
- the processing circuitry can be configured to perform the described functionality.
- Example embodiments 1. A method in an encoder to estimate an inter-channel time difference, ITD, between two waveforms, the method comprising: obtaining (901) an estimate of a threshold based on a shape of a cross-correlation estimation between the two waveforms; determining (903) a second threshold based on a maximum value of the first threshold and a non-zero constant; determining (905) at least one lag for which of cross-correlation estimate magnitude value is above the second threshold to form a list of potential peak positions; determining (907) an indication of how many true peaks there are and a list of lags of the true peaks; responsive to there being three or more true peaks, keeping three lags (909) having a higher cross-correlation estimate magnitude value than other true peaks and setting lags and cross
- determining an indication of how many true peaks there are comprises: determining (1001) that a possible peak is a true peak when a cross-correlation estimate magnitude value of the possible peak is larger than neighbor positions of the possible peak; and removing (1003) possible peaks that are too close to each other by determining when lags between the possible peaks are less than a constant apart, keeping only a lag corresponding to a largest cross-correlation estimate magnitude value.
- 3. The method of any of Embodiments 1-2, further comprising: determining (1301) whether or not to utilize level difference information for ITD estimation based on a consistency measure between estimated ITD values and level difference parameters. 4.
- Embodiment 3 further comprising determining (1401) the consistency measure.
- determining the consistency measure comprises determining the consistency measure in accordance with where ⁇ _ ⁇ _ ⁇ is an updated consistency measure, ⁇ _ ⁇ _ ⁇ ⁇ is the consistency measurement being updated, ⁇ ⁇ ( ⁇ ) is a side gain parameter, ⁇ [0] is a ITD candidate corresponding to the largest correlation among the ITD candidates, and ⁇ ⁇ is a filtering parameter which may be set to any value between 0 and 1 or be adaptively set based on the cross correlation value corresponding to ⁇ [0]. 6.
- the method of any of Embodiments 4-5 further comprising applying (1403) a forgetting factor to the consistency measure such that the consistency measure slowly approaches zero when there is no information to reliably update the consistency measure. 7.
- the method of any of Embodiments 4-5 further comprising updating the consistency measure when there is only one ITD candidate identified.
- the method of any of Embodiments 3-7 further comprising determining (1601) if the consistency measure is above a consistency threshold.
- the method of Embodiment 8 further comprising: responsive to the consistency measure being above the consistency threshold, using 1603) the level difference parameter to select among positive and negative ITD candidates. 10.
- ⁇ [0] is a ITD candidate corresponding to a largest cross-correlation estimate magnitude value among the ITD candidates
- ⁇ [1] is a ITD candidate corresponding to a second largest cross-correlation estimate magnitude value among the ITD candidates
- ⁇ [2] is a ITD candidate corresponding to a third largest correlation among the ITD candidates
- ⁇ [0] is a cross-correlation estimate magnitude value for ⁇ [0]
- ⁇ [1] is a cross-correlation estimate magnitude value for ⁇ [1]
- ⁇ [2] is a cross-correlation estimate magnitude value for ⁇ [2]
- ⁇ ⁇ is a side gain parameter
- ⁇ [0] is a ITD candidate corresponding to a largest cross-correlation estimate magnitude value among the ITD candidates
- ⁇ [1] is a ITD candidate corresponding to a second largest cross-correlation estimate magnitude value among the ITD candidates
- ⁇ [2] is a ITD candidate corresponding to a third largest correlation among the ITD candidates
- ⁇ [0] is a cross-correlation estimate magnitude value for ⁇ [0]
- ⁇ [1] is a cross-correlation estimate magnitude value for ⁇ [1]
- ⁇ [2] is a cross-correlation estimate magnitude value for ⁇ [2]
- ⁇ ⁇ is a side gain parameter
- SG_THR and SG_THR2 are thresholds
- Embodiments 8-13 further comprising: responsive to the consistency measure being below the consistency threshold, not using (1605) the level difference parameter to select among positive and negative ITD candidates.
- ⁇ [0] is a ITD candidate corresponding to a largest correlation among the ITD candidates, the method further comprising: selecting ⁇ [0] to be the ITD if there are only two sources identified, and ⁇ [0] is different from an ITD of the previous frame, a peak importance magnitude is greater than an importance magnitude threshold, and peak relative importance is less than a relative magnitude threshold.
- An encoder (502, 3002) comprising: processing circuitry (2802); and memory (2810) coupled with the processing circuitry, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations comprising: obtaining (901) an estimate of a threshold based on a shape of a cross-correlation estimation between the two waveforms; determining (903) a second threshold based on a maximum value of the first threshold and a non-zero constant; determining (905) at least one lag for which of cross-correlation estimate magnitude value is above the second threshold to form a list of potential peak positions; determining (907) an indication of how many true peaks there are and a list of lags of the true peaks; responsive to there being three or more true peaks, keeping three lags (909) having a higher cross-correlation estimate magnitude value than other true peaks and setting lags and cross-correlation estimate magnitude values of the other true peaks to zero to form a list of three candidate ITD
- determining an indication of how many true peaks there are comprises: determining (1001) that a possible peak is a true peak when a cross-correlation estimate magnitude value of the possible peak is larger than neighbor positions of the possible peak; and removing (1003) possible peaks that are too close to each other by determining when lags between the possible peaks are less than a constant apart, keeping only a lag corresponding to a largest cross-correlation estimate magnitude value. 19.
- the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising determining (1401) the consistency measure. 21.
- determining the consistency measure comprises determining the consistency measure in accordance with where ⁇ _ ⁇ _ ⁇ is an updated consistency measure, ⁇ _ ⁇ _ ⁇ ⁇ is the consistency measure being updated, ⁇ ⁇ ( ⁇ ) is a side gain parameter, ⁇ [0] is a ITD candidate corresponding to the largest correlation among the ITD candidates, and ⁇ ⁇ is a filtering parameter which may be set to any value between 0 and 1 or be adaptively set based on the cross correlation value corresponding to ⁇ [0]. 22.
- the encoder (502, 3002) of any of Embodiments 20-21 wherein the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising applying (1403) a forgetting factor to the consistency measure such that the consistency measure slowly approaches zero when there is no information to reliably update the consistency measure.
- the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising updating the consistency measure when there is only one ITD candidate identified.
- the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising: responsive to the consistency measure being above the consistency threshold, using 1603) the level difference parameter to select among positive and negative ITD candidates.
- the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising: responsive to the consistency measure being above the consistency threshold and a current frame down mix energy estimate is an energy threshold above a previous frames long term energy estimate, using 1603) the level difference parameter to select among positive and negative ITD candidates.
- ⁇ [0] is a ITD candidate corresponding to a largest cross-correlation estimate magnitude value among the ITD candidates
- ⁇ [1] is a ITD candidate corresponding to a second largest cross-correlation estimate magnitude value among the ITD candidates
- ⁇ [2] is a ITD candidate corresponding to a third largest cross-correlation estimate magnitude value among the ITD candidates
- ⁇ [0] is a cross-correlation estimate magnitude value for ⁇ [0]
- ⁇ [1] is a cross-correlation estimate magnitude value for ⁇ [1]
- ⁇ [2] is a cross-correlation estimate magnitude value for ⁇ [2]
- ⁇ ⁇ is a side gain parameter
- the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising: if there are more than one ITD candidate identified and none of them are considered close to zero based on an ITD candidate distance to zero being
- ⁇ [0] is a ITD candidate corresponding to a largest cross-correlation estimate magnitude value among the ITD candidates
- ⁇ [1] is a ITD candidate corresponding to a second largest cross-correlation estimate magnitude value among the ITD candidates
- ⁇ [2] is a ITD candidate corresponding to a third largest correlation among the ITD candidates
- ⁇ [0] is a cross-correlation estimate magnitude value for ⁇ [0]
- ⁇ [1] is a cross-correlation estimate magnitude value for ⁇ [1]
- ⁇ [2] is a cross-correlation estimate magnitude value for ⁇ [2]
- ⁇ ⁇ is a side gain parameter
- SG_THR and SG_THR2 are thresholds
- the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising: if there are more than one ITD candidate identified and none of them are considered close to zero
- a computer program comprising program code to be executed by processing circuitry (2802) of an encoder (502, 3002), whereby execution of the program code causes the encoder (502, 3002) to perform operations according to any of Embodiments 1-16.
- a computer program product comprising a non-transitory storage medium including program code to be executed by processing circuitry (2802) of an encoder (502, 3002), whereby execution of the program code causes the encoder (502, 3002) to perform operations according to any of Embodiments 1-16.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
A method in an encoder to estimate an inter-channel time difference, ITD, between two waveforms. In the method, an estimate of a threshold based on a shape of a cross-correlation function between the two waveforms is obtained. A second threshold is determined based on a maximum value of the first threshold and a non-zero constant. At least one lag for which of cross-correlation estimates magnitude is above the second threshold is determined to form a list of potential peak positions. An indication of how many true peaks there are and a list of lags of the true peaks is determined and the ITD is estimated based on the list of three candidate ITD estimates.
Description
REFINED INTER-CHANNEL TIME DIFFERENCE (ITD) SELECTION FOR MULTI-SOURCE STEREO SIGNALS TECHNICAL FIELD [0001] The present disclosure relates generally to communications, and more particularly to encoder/decoder methods and related encoders and decoders supporting encoding and/or decoding. BACKGROUND [0002] In communication networks one strives to achieve as good quality as possible while still using as few resources as possible. The main resource for transfer of speech or audio is in most cases the bit rate as it affects the capacity of the network. The quality relates to the similarity of the decoded output compared to the source at the input and typically refers to how humans perceive this similarity which is affected by human auditory system. Some other resources that may limit the solution are complexity, which affect the energy consumption and therefore the battery life for mobile devices, and the amount of program and data memory used by the implementation. [0003] In general, the encoding process is made by segmenting the input signal into frames of 10-20ms which are processed to extract parameters in the encoder and the parameters extracted are then encoded. These encoded parameters are then transferred to the decoder where the signal is decoded and reconstructed. It is also common to have some overlap between consecutive frames and then use an analysis/synthesis window combination that enables perfect (or at last nearly perfect) reconstruction of the input signal from the parameters in the decoder when the synthesis window is used in an overlap add reconstruction. The system typically includes quantization and encoding processes which introduce impairments in the reconstruction of the signals at the decoder (sometimes also including or being connected to a renderer). [0004] While stereo signals could be treated as two separate mono channels, this would not be as efficient as coding the two channels together, as one unit. The reason is that in a typical stereo signal, the channel pair shows a high degree of similarity, or correlation. State-of-the-art stereo coding schemes exploit this correlation by employing, e.g., parametric coding, where a single channel is encoded with high quality and complemented with a parametric description that allows to reconstruct the full stereo image.
[0005] The type of parameters used as stereo parameters are, e.g., inter-channel level difference (or corresponding parameters such as side-gain), inter-channel phase difference, inter- channel time difference, and inter-channel coherence, which are typically derived through analysis in the encoder. The encoder may use these (potentially quantized) parameters for a downmix of the input stereo signals to a mono signal being encoded and transmitted to a decoder. In the decoder these quantized parameters received from the encoder may be used for a stereo upmix synthesizing stereo signals based on the mono downmix signal and the stereo parameters. [0006] Parametric coding techniques are typically beneficial at lower bitrates. At high bitrates, encoding of the audio signals of more audio channels may give a better quality, e.g., utilizing M/S stereo coding techniques that transform the left and right channel signals into a mid channel and a side channel signals, or directly encoding of the left and right audio channel signals. [0007] However, for intermediate bitrates hybrid coding techniques, combining parametric models with encoding of residual signals, may obtain a better coding efficiency, i.e., quality given the constraints in resources. The residual signals aim to capture the error between the input audio signals and what is described by the parametric model, so that utilizing the residual signals at the decoder will give an output that better resembles the original audio signals input to the encoder. However, as the bitrate is limited, quantization and lossy coding would typically be applied for such residual signals also, which means there are still quantization errors in the decoded audio signals. [0008] The process of reducing the channel pair into a single channel is often called a down-mix and the resulting channel the down-mix channel. The down-mix procedure typically tries to maintain the energy by aligning the signals of the stereo channels based on estimated inter-channel time differences (ITD) and inter-channel phase differences (IPD) before mixing the channels. To capture the energy balance of the input signal, an inter-channel level difference (ILD) or a side gain parameter may also be estimated. These stereo parameters are then encoded and may be used in an up-mix procedure when reconstructing the stereo channel pair at a decoder. Figure 1 and Figure 2 show block diagrams of an example of a parametric stereo encoder and decoder. [0009] If a stereo signal is captured by two microphones the distance from the source to the left microphone may be different from the distance to the right microphone as illustrated in Figure 3. When Speaker 1 talks the different distances (L1 and L2) to the two microphones left and right will generate a small difference in delay and level for the two microphones. These are
called inter channel time difference (ITD) and inter channel level difference (ILD) and are important parameters in the parametric stereo model. [0010] The amount of inter channel level difference (ILD) depends on the source’s direction, distance from the microphones and the distance between the microphones. This is simply the effect that the signal strength is attenuated at a rate proportional to the square of the distance that the signal travels. Its main use is in the parametric stereo coder where the signal is up-mixed from the decoded mono to the two stereo channels again. [0011] In general, if there are several audio sources they may have different ITDs. The background noise will often be a sum of many sources and does not have one clear ITD. [0012] If the difference in delay is such that significant part of the source signals are out of phase with each other when reaching the microphones, it may cause destructive interference in a downmixing process. That is, signals with opposite signs in the left and right microphone may cancel out if just added. To avoid this, the signals are time aligned before they are added together. [0013] This is done by estimating the ITD and then time shifting one of the channel signal so that they are aligned (which can sometimes also be done in frequency domain by a phase shift of the transformed signal). [0014] After time shifting, the mono signal resulting from the downmix is fed to a speech/audio encoder where the bitstream from the coder and the stereo parameters are encoded and transmitted to the decoder. [0015] In the decoder, the process is reversed as illustrated in Figure 2. First the bitstream from the encoder is feed to the decoder to create the decoded mono signal. This is then feed to the stereo processing and up-mixing block, where stereo parameters are used to recreate the two stereo channels. For example, the ITD parameter may be used to restore the time difference between the left and right channel and the ILD or side gain to recreate the level difference. SUMMARY [0016] There currently exist certain challenge(s). [0017] For stationary signals one wants a stable ITD estimate – as there would be a degradation in the output signal if the ITD estimate is noisy or unstable. It is therefore common to use smoothing in the ITD estimate. Such smoothing can be done both in the parameter estimation as such but also in the form of decision logic with a preference of ITD estimates close to previous estimates or using a hangover logic where one keeps the ITD estimate fixed to the last ITD over short periods of time when the input signal does not contain/show any ITD. This
usually works for stationary signals where there is one source active at the time. Even multiple sources in different directions are handled if they are not active at the same time. [0018] The problem with this method is that if a second source becomes active while the first source is still present, the smoothing/hangover will keep the ITD estimate to the value ITD of the first signal. In cases where the second source has become the dominant signal the decoded signal, based on the ITD from the first, can appear to have diffuse or even faulty position compared to the original input signals. [0019] Certain aspects of the disclosure and their embodiments may provide solutions to these or other challenges. According to some embodiments, a method in an encoder to estimate an inter-channel time difference, ITD, between two waveforms includes obtaining an estimate of a threshold based on a shape of a cross-correlation function between the two waveforms. The method includes determining a second threshold based on a maximum value of the first threshold and a non-zero constant. The method includes determining at least one lag for which of cross- correlation estimates magnitude is above the second threshold to form a list of potential peak positions. The method includes determining an indication of how many true peaks there are and a list of lags of the true peaks. The method includes responsive to there being three or more true peaks, keeping three lags having a higher correlation magnitude values than other true peaks and setting lags and correlation magnitude values of the other true peaks to zero to form a list of three candidate ITD estimates. The method includes responsive to there being less than three true peaks, keeping the lags and correlation of the true peaks and setting lags and correlation values of remaining positions to zero to form the list of three candidate ITD estimates. The method includes estimating the ITD based on the list of three candidate ITD estimates. [0020] According to some embodiments, an encoder to estimate an inter-channel time difference, ITD, between two waveforms is configured to perform operations comprising obtaining an estimate of a threshold based on a shape of a cross-correlation function between the two waveforms. Determining a second threshold based on a maximum value of the first threshold and a non-zero constant, and determining at least one lag for which of cross-correlation estimates magnitude is above the second threshold to form a list of potential peak positions. The encoder being configured to perform determining an indication of how many true peaks there are and a list of lags of the true peaks. Responsive to there being three or more true peaks, keeping three lags having a higher correlation magnitude values than other true peaks and setting lags and correlation magnitude values of remaining positions to zero to form a list of three candidate ITD estimates. Responsive to there being less than three true peaks, keeping the lags and correlation of the true peaks and setting lags and correlation values of the other true peaks to zero to form
the list of three candidate ITD estimates, and estimating the ITD based on the list of three candidate ITD estimates. [0021] Certain embodiments may provide one or more of the following technical advantage(s). With the embodiments, it is possible to identify when a source in one direction is interrupted by a second source in another direction and to enable a quick adjustment of the stereo parameters, more specifically the ITD, allowing the decoder to quickly adapt to the new dominant direction and thereby improves the audio quality of the decoded signal. BRIEF DESCRIPTION OF THE DRAWINGS [0022] The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this application, illustrate certain non-limiting embodiments of inventive concepts. In the drawings: [0023] Figure 1 is a block diagram of a parametric stereo encoder; [0024] Figure 2 is a block diagram of a parametric stereo decoder; [0025] Figure 3 is an illustration of inter channel time difference (ITD) and inter-channel level difference (ILD); [0026] Figure 4 is an illustration of ITD estimates of separate speakers and combined with overlap; [0027] Figure 5 is an illustration of an example of an operating environment of an encoder according to some embodiments; [0028] Figure 6 is a flow chart illustrating frame processing steps for a parametric stereo encoder; [0029] Figure 7 is a flow chart illustrating ITD estimation with hangover logic; [0030] Figure 8 is a flow chart illustrating ITD estimation with detection of multiple sources according to some embodiments; [0031] Figure 9-10 are flow charts illustrating operations an encoder performs according to some embodiments; [0032] Figure 11 is a flow chart illustrating peak pruning according to some embodiments; [0033] Figure 12 is an illustration of how the various embodiments handle multiple sources compared to conventional approaches; [0034] Figures 13-14 are flow charts illustrating operations an encoder performs according to some embodiments; [0035] Figure 15 is a flow chart illustrating updating itd_con_st according to some embodiments;
[0036] Figures 16-18 are flow charts illustrating further operations an encoder performs according to some embodiments; [0037] Figure 19 is a flow chart illustrating operations an encoder performs in ITD selection according to some embodiments; [0038] Figures 20-21 are flow charts illustrating operations an encoder performs in ITD selection according to some other embodiments; [0039] Figure 22 is a flow chart illustrating operations an encoder performs in ITD selection according to some embodiments; [0040] Figures 23-25 are flow charts illustrating operations an encoder performs in ITD selection according to some other embodiments; [0041] Figure 26 is a flow chart illustrating multi source detection is performed before ITD hangover according to some embodiments; [0042] Figure 27 is a flow chart illustrating multi source detection replaces ITD hangover according to some embodiments; [0043] Figure 28 is a block diagram of an encoder in accordance with some embodiments; [0044] Figure 29 is a block diagram of a host device in accordance with some embodiments; and [0045] Figure 30 is a block diagram of a virtualization environment in accordance with some embodiments. DETAILED DESCRIPTION [0046] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art, in which examples of embodiments of inventive concepts are shown. Inventive concepts may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of present inventive concepts to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present/used in another embodiment. [0047] A more detailed description of the ITD estimation shall now be described as the ITD estimation creates the cross correlation estimate that is an important feature. [0048] The conventional parametric approach to estimate the ITD relies on the cross- correlation function (CCF) ^^^ which is a measure of similarity between two waveforms ^[^]
and ^[^], and is generally defined in the time domain as ^^^ [^] = ^{^[^]^[^ + ^]} where ^ is the time-lag parameter and ^{⋅} the expectation operator. For a signal frame of length ^, the cross-correlation is typically estimated as
[0049] The inter-channel coherence or inter-channel correlation (ICC) is conventionally obtained as the maximum of the CCF which is normalized by the signal energies as follows:
[0050] The time lag ^ corresponding to the ICC is determined as the ITD between the channels ^ and ^. By assuming ^[^] and ^[^] are zero outside the signal frame, the cross- correlation function can equivalently be expressed as a function of the cross-spectrum of the frequency spectra ^[^] and ^[^], with discrete frequency index ^, as ^^^^^ [ ^ ] = ^ [ ^ ] ^ ∗[^] ^^^ [^] = ^^^^^(^^^^^[^]) where ^[^] is the discrete Fourier transform (DFT) of the time domain signal ^[^], i.e.
and the ^^^^^(⋅) or ^^^^(⋅) denotes the inverse discrete Fourier transform. [0051] For the case when ^[^] is purely a delayed version of ^[^], the cross-correlation function is given
where ∗ denotes convolution and ^(^ − ^^) is the Kronecker delta function, i.e., it is equal to one at ^^ and zero otherwise. This means, the cross-correlation function between ^ and ^ is the delta function spread by the convolution with the autocorrelation function for ^[^]. [0052] This will broaden the delta peak. For signal frames with several delay components, e.g., several talkers, there will be peaks at each delay present between the signals, and the cross correlation becomes
[0053] The delta functions might then be spread into each other and make it difficult to identify the several delays within the signal frame. There are however generalized cross-
correlation (GCC) functions that do not have this spreading. The GCC is generally defined as ^^ ^ ^ ^^ [^] = ^^^^^(ψ[k]^[^]^∗[^]) where ^[^] is a frequency weighting. Especially for spatial audio, the phase transform (PHAT) has been utilized due to its robustness for reverberation in low noise environments. The phase transform is basically the absolute value of each frequency coefficient, i.e.,
[0054] This weighting will thereby whiten the cross-spectrum such that the power of each component becomes equal. With pure delay and uncorrelated noise in the signals ^[^] and ^[^] the phase transformed GCC (GCC-PHAT) becomes just the Kronecker delta function ^(^ − ^^), i.e.,
[0055] The encoding process is done on time segments called frames. Common lengths of these segments are 10 or 20 ms. The coding parameters like the ITD are estimated at the encoding side on a per frame basis and are transmitted to the decoder. It is also common to not transmit a parameter if there is no clear gain in the encoding process with using the parameter. In the ITD case, this will be when the left and right signals are more or less uncorrelated. [0056] For the ITD it is desirable to have an ITD estimate that does not have a small random variation on a frame-by-frame basis. One way to stabilize the estimate is to low-pass filter the cross spectrum with a simple first order filter:
where ^ is a frequency bin index and ^ is a frame number. [0057] The filter coefficient α can be fixed but it may also be adaptive. One example is to use a spectral flatness measure, ^^^, calculated on the left or right input signal as the filter coefficient, as described in paragraphs 0079-0081 of U.S. Patent Application Publication No. 20200194013A1, and which can be determined in accordance with:
This measure will have the range 0.0 – 1.0 where a higher value would indicate a flatter spectrum. Using this coefficient may improve the robustness and accuracy of the ITD estimation. [0058] Ideally one would just have one peak in the cross correlation estimate ^^ ^ ^ ^^ [^] and it
would be close to 1.0 if there is correlation between the channels and 0.0 if it is not. Finding the peak would be just a matter of looking for the highest peak – and one would get a reading of the estimated time delay of the channels. This is not the case; by its nature ^^ ^ ^ ^^ [^] is highly dependent on the characteristics of the input signal – so one needs to have an adaptive threshold to judge if there is correlation or not. [0059] One way to handle the signal dependent characteristics of the ^^ ^ ^ ^^ [^] is to create an adaptive threshold, ^ℎ^^^ℎ, to decide if there is a sufficient cross correlation in the signals and an ITD hangover logic to keep a more stable estimate of such an delay. One way to determine the threshold is described in paragraphs 0097-0111 of U.S. Patent Application Publication No. 20200194013 and is based on measuring the variation of absolute levels in the ^^ ^ ^ ^^ [^] trying to find the range of variation between peaks and the variations due to noise and interference between sources and background noise. [0060] A feature that measures the energy balance between the channels will show if the source is to the right or to the left. One such feature is the side gain parameter which represents the projection of a side signal ^(^) on a downmix signal ^(^). Let the signals of the left and right channels (^(^) and ^(^)) for a frame ^ be represented as vectors ^^ and ^^, and the vectors of a downmix of the left and right channel signals ^^ be ^ + ^ ^ = ^ ^ ^ 2 and a side signal ^^ be
[0061] The scaling factor of ½ may not always be used. Thus, other scaling factors may be used. The side gain ^^^^^(^) can then be obtained by the inner product (also called dot product) as
[0062] If the signals are assumed to be phase aligned prior to downmix, the ^^ and ^^ vectors point in the same direction and the side gain will correspond to:
[0063] The energy of the signals corresponds to the squared length of the vectors, denoted by ‖⋅‖2. [0064] In some cases, an alternative side gain parameter may be used, e.g.,
where ‖^ ‖^ ^ = ^ ^ ‖^^‖^ [0065] To avoid division by zero, a small non-zero term ^^^ may be added to the energy measures, or to the denominator. Side gain parameters may also be estimated separately for certain frequencies (or frequency bands) of the signal, e.g., in the Discrete Fourier Transform (DFT) domain. [0066] As the ITD and ILD are both related to the position of a captured source, the level difference between channels may also be utilized in the ITD estimation. The method determines a set of ITD candidates corresponding to both positive and negative lags, and selects the sign of the ITD, i.e., whether to select a positive or a negative time lag based on the ILD. For example, if a source to the right corresponds to a positive time lag, and there is an energy-dominant right channel signal, e.g., as indicated by a negative side gain parameter, a positive time lag may be selected. This may e.g., be done if there are ITD candidates, peaks of the cross-correlation function, that are rather close (in respect of magnitude, not time lag). It may result in not selecting the time lag corresponding to the peak of highest magnitude (i.e., highest estimated correlation), instead another one is used which is more consistent with the estimated ILD. Further details of embodiments of the method is shown in U.S. Patent No.10,002,614 and U.S. Patent No.10,311,881. [0067] For stationary signals one wants a stable ITD estimate – as there would be a degradation in the output signal if the ITD estimate is noisy or unstable. It is therefore common to use smoothing in the ITD estimate. Such smoothing can be done both in the parameter estimation as such but also in the form of decision logic with a preference of ITD estimates close to previous estimates or using a hangover logic where one keeps the ITD estimate fixed to the last ITD over short periods of time when the input signal does not contain/show any ITD. This usually works for stationary signals where there is one source active at the time. Even multiple sources in different directions are handled if they are not active at the same time. [0068] The problem with this method is that if a second source becomes active while the first source is still present, the smoothing/hangover will keep the ITD estimate to the value ITD of the first signal. In cases where the second source has become the dominant signal the decoded signal, based on the ITD from the first, it can appear to have diffuse or even faulty position compared to the original input signals.
[0069] Figure 4 illustrates the problem with such existing solutions. Figure 4 is based on a synthesized example where two sources with different directions are processed both individually and in combination. In Figure 4, the top shows the energy of the separate speakers (labeled in Figure 4 as 1st and 2nd), the middle contains the ITD estimates for the separate files. The bottom figure shows the ITD track for the combined source file (where the 1st and 2nd signals are added before being sent to the codec). Conventional technology was used for the ITD estimations. [0070] Figure 4 illustrates how the codec misses the initial part of the 2nd speaker (start of overlap region) even though the 2nd speaker is clearly the louder of the two. There are also other regions where the 1st speaker is louder than the 2nd but the ITD used is the one for the 2nd speaker. These "misses" in ITD estimates used result in a degraded quality with diffuse of faulty position for the speaker or that the speaker appears in the faulty position and then "glides" to the correct one. [0071] Methods described in U.S. Patent Nos.10,002,614 and 10,311,881 may be used to improve the ITD estimation in certain cases. However, there are situations (e.g., certain signals) where a selection based on the ILD results in a suboptimal selection of the ITD among ITD candidates, i.e., peaks of a cross-correlation function. [0072] The cross-correlation function may typically give some information on the number of sources and their direction(s) even though some analysis may be necessary. To extract such information, one needs to further process the cross-correlation estimate to first decide how many source directions are present. If there are more than one source direction present, further processing is needed to find which of them is dominant and should be used for the ITD estimate. If the dominant source direction is different from the last used (e.g., for the previous frame) one can, if the new direction is strong enough (in terms of correlation), decide to immediately change the ITD estimate to use for the current frame. Some embodiments provide a number of identified sources and vectors of magnitudes in decreasing order for up to the three strongest ITD candidates and a corresponding vector with their lags to determine which source direction to use. [0073] Other embodiments improve the robustness of determining the sign of the ITD based on the level difference between signals of the stereo channels by a decision on whether or not to utilize the level difference information, e.g., side gain parameters, in determining the ITD value. [0074] While there are ^^^^(^) functions used, the functions are used to only look at the sign of the value ^. In some embodiments, a dead zone is used which can simplify the decision logic. The function ^^^^_^0^(^, ^ℎ^) takes the additional argument of a threshold for the dead- zone, that is
−1, ^^^ ^ < −^ℎ^ ^^^^_^0^(^, ^ℎ^) = ^ +1 ^^^ ^ > ^ℎ^ 0 ^^ℎ^^^^^^ [0075] The advantage with this function is that one can check for different signs and also handle a dead zone around zero where the sign would be reported as 0. [0076] When the signal-to-noise ratio, SNR, is low in the input signal, the cross-correlation becomes noisier and therefore unreliable – at the same time the noise makes it more difficult to hear the errors in the source direction. It is therefore desirable to have a logic that only modifies the ITD estimate when the features are reliable. [0077] Various embodiments provide peak pruning to determine the number of source directions available as the cross-correlation function can show many peaks, showing that the one source assumption is not true. Thus, the use of a stationary ITD would not be optimal and it should be investigated whether one should change the ITD to handle the direction of the new source. [0078] Prior to discussing further details of peak pruning, an overview of speech coding that operates with a parametric stereo encoder principle, as illustrated in Figure 1 shall be described. The generated bit-stream is transported to the decoder illustrated in Figure 2 where it is decoded, and a stereo signal is recreated. [0079] Figure 5 illustrates an example of an operating environment in which the various embodiments of the present disclosure may be implemented. Turning to Figure 5, in the example operating environment 500, the encoder 502 receives data, such as an audio file, to be encoded from an entity through network 504, such as a host 506, and/or from storage 508. The encoder 502 in various embodiments is a parametric stereo encoder such as, e.g., the parametric stereo encoder of Figure 1. In some embodiments, the host 506 may communicate directly to the encoder 502. The encoder 502 encodes the audio file as described herein and either stores the encoded audio file in storage 508 or transmits the encoded audio file to a decoder 512 via network 510. The decoder 512 is various embodiments is a parametric stereo decoder such as, e.g., the parametric stereo decoder illustrated in Figure 2. The decoder 512 decodes the audio file and transmits the decoded audio file to an audio player 514 for playback. The audio player 514 may be or be comprised in a user equipment, a terminal, a mobile phone, and the like. In other embodiments, the host 506 may transmit encoded audio files to the decoder 512 via network 510. [0080] The overall processing of the parametric stereo involves segmenting the input in to frames of 20ms of audio samples. Note that this refers to the amount of new data consumed by each frame processing loop. The actual data analysis for a frame is usually longer, by also including some earlier frame(s). The basic principle is to process each frame by deriving the ITD
estimate, time align the stereo channels before down mixing it to a mono channel that is encoded using a state-of-the-art mono speech codec and then sent to a receiver. This processing scheme for a typical parametric stereo solution utilizing ITD estimates is shown in Figure 6. Note that additional processing blocks may be included if other stereo parameters are included in the parametric stereo model. [0081] After transmission of the bitstream of encoded mono signal and stereo parameters to the decoder, the process is reversed. The mono bitstream is decoded and the mono signal is then up-mixed using decoded stereo parameters and the resulting output can be used for audio output. [0082] As the peak pruning is related to the ITD estimation, the various embodiments described below shall focus on ITD estimation in the encoder. [0083] There are two partly competing requirements on the system about ITD estimation. It is important to use the correct ITD estimate for the down mix, as this would prevent any signal cancelation that would happen if the two channels in the input signal are out of phase. At the same time in the decoder, the ITD contributes to the positioning of the rendered output where rapid variations can be perceived as disturbing. It is therefore common to use smoothing of the ITD estimate and even hangover logic to keep the ITD steady even in short periods where the input signal does not contain any detectable correlation between the channels. This works under the assumption that, in most cases, you have one stationary source, e.g., a speaker where it is beneficial to maintain the speaker’s position even during the short pauses that occur in and between utterances. [0084] This normal processing for ITD estimation is shown in Figure 7, where the method gcc_phat() as described in paragraphs 0079-0081 of U.S. Patent Application Publication No. 20200194013 is used for the determination of the ITD estimate of the current frame. Other methods may be used. This is followed by the ITD Hangover logic, which keeps the ITD estimate even during short pauses in normal speech. Further details of the ITD Hangover logic according to an embodiment may be found in U.S. Patent No.10,832,689. [0085] While the processing shown in Figure 7 works well for cases where a single source is active at a time, there would be problems if multiple sources are active at the same time as shown in Figure 4. Here it becomes important to determine which of the speakers is the dominant part and use the ITD corresponding to that speaker position. [0086] The peak pruning described herein is used to determine the number of source directions available as the cross-correlation function can show many peaks – showing that the one source assumption is not true, the use of a stationary ITD would not be optimal and it should be investigated whether one should change the ITD to handle the direction of the new source.
[0087] One feature from the peak pruning is the number of detected peaks ^^^^^^^^^^^(^), for the current frame, ^. Some other features are used to judge the importance of the three largest peaks found. The three lags corresponding to the three largest cross correlation peaks, found through pruning, are placed in the vector ^^^^[^], ^^^ ^ = 0 … 2 and the corresponding cross correlation magnitude values are placed in ^^^[^], ^^^ ^ = 0 … 2. Thus, ^^^[^] will be greater than or equal to zero. [0088] To better judge the magnitude of correlation regardless of how many correlation peaks there are, it is better to use a combined peak magnitude in the following way: ^^^^^^^^^^^ = ^^^[0] + ^^^[1] which is basically the sum of the two largest peaks. Note that if no peaks are found, the corresponding ^^^[^] is set to zero. The importance of the largest peak is calculated as ^^^[0] ^^^^^^^^^^ = max (^^^^^, ^^^^^^^^^^^) where ^^^^^ is a small non-zero constant to avoid division by zero (e.g., 0.12). The second peak relative importance is calculated as:
[0089] With these features, the decision logic to determine which peaks magnitude is important becomes less dependent of the peaks’ absolute magnitudes. [0090] These features are then used in a further ITD handling step to allow for a change of ITD when there are sufficient indications that there is a new dominant source direction present in the input signal. This is illustrated in Figure 8. Thus, the various embodiments provide an additional operation after the ITD hangover processing, which is called multisource detection and ITD selection in Figure 8. [0091] The first step in the multisource detection and ITD selection is to determine if there are multiple sources in the current input. This information is present in the cross correlation estimate, but additional processing is needed to make the determination. Figure 9 illustrates operations the encoder 502 performs in the multisource detection and ITD selection. Turning to Figure 9, in block 901, the encoder 502 obtains an estimate of a threshold based on a shape of a cross-correlation function between two waveforms. For example, using the gcc_phat(),a threshold based on the shape of the cross correlation can be estimated - see U.S. Patent Application Publication No.20200194013, paragraphs 0097-0111 for details of the threshold. [0092] In block 903, the encoder 502 determines a second threshold based on a maximum value of the first threshold and a non-zero constant. For example, a second threshold based on
the one from gcc_phat() is formed by taking the maximum value of the one from gcc_phat() and a non-zero constant. In some embodiments, a nonzero constant in the range of 0.1 to 0.2 can be used. In one embodiment, a value of 0.12 is used for the non-zero constant. [0093] In block 905, the encoder 502 determines at least one lag for which of cross- correlation estimate magnitude is above the second threshold to form a list of potential peak positions. Thus, the second threshold is used to find the lags for which of the cross-correlation estimate magnitude is above said threshold, these lags form a list of potential peak positions. [0094] In block 907, the encoder 502 determines an indication of how many true peaks there are and a list of lags of the true peaks. Then the list of potential peaks is evaluated to find the lags that are true peaks. Figure 10 illustrates a way to find lags that are true peaks. Turning to Figure 10, in block 1001, the encoder 502 determines that a possible peak is a true peak when a cross-correlation magnitude value is larger than its neighbors (the positions lag-1 and lag+1). Of these true peaks, one needs to remove peaks that are too close to each other. One way to do this is illustrated in block 1003 where the encoder 502 removes possible peaks that are too close to each other by determining when lags between the possible peaks are less than a constant apart (e.g., 10), keeping only a lag corresponding to a largest cross correlation estimate magnitude value. Thus, only the lags having the larger of the cross correlation estimate magnitude values are kept. At the end of this process, an indication of how many peaks there are, and a list of their lags is obtained. Returning to Figure 9, in block 909, the encoder 502, responsive to there being three or more true peaks, keeps three lags having a higher cross correlation estimate magnitude value than other true peaks and sets lags and cross correlation estimate magnitude values of the other true peaks to zero to form a list of three candidate ITD estimates. Thus, the three lags with the largest cross correlation estimate magnitude values are kept. In block 911, the encoder 502, responsive to there being less than three true peaks, keeps the lags and cross correlation estimate magnitude value of the true peaks and sets lags and cross correlation estimate magnitude values of what would correspond to the other true peaks (if there were at least three true peaks) to zero to form the list of three candidate ITD estimates. Thus, if there are fewer than three lags with non-zero cross correlation estimate magnitude vales, the remaining positions lags are set to 0 and their cross-correlation estimate magnitude value is set to 0. These up to three lags kept can be seen as ITD candidates and are put in ^^^^[^] and ^^^[^] in decreasing order of ^^^[^] values (where the ^^^[^] values are greater than or equal to zero). In block 913, the encoder 502 estimates the ITD based on the list of three candidate ITD estimates and a consistency measure between estimated ITD values and level different parameters. The process is also illustrated in the flow chart in Figure 11, which illustrates operations to determine a number of peaks and lag
and cross correlation estimate magnitude values for the three largest. How the encoder 502 estimates the ITD in various embodiments is described below. [0095] Note that there may be lag in the ITD estimation, e.g., due to smoothing of the cross- correlation function over time, ILD or side gain parameters may earlier indicate the change of a dominant source. Thereby, the level difference may indicate a preferred direction even when the cross-correlation function does not yet show this as a dominant source. The various embodiments provide a method selecting whether to utilize the level difference information for ITD estimation or not based on a consistency measure between estimated ITD values and level difference parameters. In some other embodiments, the method further utilizes the position of the ITD candidates, i.e., their position in the cross-correlation function, and/or information on whether the signals encoded contains speech or music. [0096] When there are multiple sources present, they will interfere with each other, and it is therefore likely that the cross-correlation peak of the first source will get lower when another source appears. To reliably detect if there are sources available one needs to at least look at the combined correlation of the two largest peaks. This feature is captured in the ^^^^^^^^^^^ feature defined above. [0097] The ^^^^^ changes sign with the appearance of an new dominant direction in the input signal. Using the ^^^^_^0^(^, ^ℎ^) makes it possible to form more compact logic that handles sign changes and minimum magnitudes of new ^^^^^ values. By combining the dead zone test with the sign, one also reduces the complexity of the logic. [0098] It can also be noted the commonly available ^^^^(^) function is to some degree implementation dependent, in some implementations it only returns -1 or +1, that is 0 is here interpreted as a positive value. While in other it would return -1, 0 and +1 – that is the same as ^^^^_^0^(^, ^ℎ^) would if ^ℎ^ = 0. [0099] For example with the condition −^^^^_^0^(^, ^ℎ^1) ^^^^_^0^(^, ^ℎ^2) ^^^(^) > ^ℎ^3, it is possible formulate a condition that ^ and ^ needs to have different signs – with possible dead zones for ^ (^ℎ^1) and ^ (^ℎ^2) and that the magnitude of ^ should be larger than ^ℎ^3. [0100] To avoid switching to a new ITD in the middle of a high energy segment of e.g., the 1st speaker, one needs to keep track of the energy profile of the input signal. One way of doing this is to use an energy estimation of the downmix channel to generate an energy track of the input signal. The energy estimate is first converted into a logarithmic domain in accordance with: ^^^^^^^^ = 10 ^^^10(^^^^^^), where ^^^^^^ is the energy estimate of the current frame of the mono down mix.
[0101] The converted energy estimate is then further filtered using an autoregressive (AR) process at the end of the processing of the current frame: ^^^^^^^^^^ = (1.0 − ^^^^^^ℎ^) ^^^^^^^^^^ + ^^^^^^ℎ^ ^^^^^^^^ where ^^^^^^ℎ^ is the AR coefficient and controls the smoothing process. This allows one to compare the current frame's energy level to the smoothed history of previous frames energy, so by comparing the feature ^^^^^^^^ − ^^^^^^^^^^ to a threshold – e.g., ^^^^^^^^ − ^^^^^^^^^^ > ^_^^^_^^^ one gets an indication of when there is sudden increase in the energy after a sufficient drop in energy from the previous utterance – controlled by the smoothing of ^^^^^^^^. This feature is combined with other features as described in the decision logic for the ITD selection. The filtered energy measure ^^^^^^^^^^ is a state variable being maintained between frames. For the processing of the next frame ^ + 1, ^^^^^^^^^^ would prior to the filtering process comprise the filtered energy measure obtained in frame ^. The state parameter ^^^^^^^^^^ may be initialized to zero. [0102] With the peak pruning, it is possible to identify when a source in one direction is interrupted by a second source in another direction and to enable a quick adjustment of the stereo parameters, more specifically the ITD, allowing the decoder to quickly adapt to the new dominant direction and thereby improving the audio quality of the decoded signal. Figure 12 illustrates how the various embodiments can handle overlapping sources as compared the techniques illustrated in Figure 4. In Figure 12, the ITD ORG is the techniques illustrated in Figure 4 and ITD CUT illustrates how the various embodiments detect multiple sources. The first and second in the Energy (dB) indicate the two sources. [0103] As a result of the peak pruning, one or more ITD candidates may have been identified. Next follows an ITD selection logic used to determine whether the ITD value determined using gcc_phat() and ITD Hangover logic shall be used or whether one of the ITD candidates shall be selected, and if so, which one of them. Note that one of the ITD candidates may be the same lag as was already obtained using gcc_phat() and the ITD Hangover logic. [0104] The ITD selection includes a decision on whether to utilize the level difference information for ITD estimation or not based on a consistency measure between estimated ITD values and level difference parameters, in this case the side gain parameter. This is shown in Figure 13, where in block 1301, the encoder 502 determines whether or not to utilize level difference information for ITD estimation based on a consistency measure between estimated ITD values and level difference parameters. [0105] Turning to Figure 14, in block 1401, the encoder 502 determines the consistency
measure. The updated consistency measure ^^^_^^^_^^ may be obtained in accordance with
where ^^^_^^^_^^^^^^^^^ is the current consistency measure that is being updated. ^^^^^ (^) is the side gain parameter, ^^^^[0] is the ITD candidate corresponding to the largest correlation among the ITD candidates, and ^^^^ is a filtering parameter which may be set to any value between 0 and 1, or be adaptively set based on the cross correlation value corresponding to ^^^^[0], i.e., ^^^^ = ^^^[0]. The consistency measure ^^^_^^^_^^^^^^^^^ is a state variable being maintained between frames, which corresponds to the consistency measure determined in the previous frame ^ − 1. For the processing of the next frame ^ + 1, ^^^_^^^_^^^^^^^^^ would comprise ^^^_^^^_^^ obtained in frame ^. The state parameter ^^^_^^^_^^^^^^^^^ may be initialized to zero. [0106] Note that in this case different signs of the side gain parameter and the ITD lag indicates consistent parameters. This means, the larger ^^^_^^^_^^, the more consistent the parameters are. A value approaching -1 means the parameters are fully inconsistent. However, depending on the definition of the level difference parameter and the ITD, it may rather be equal signs that indicate that the parameters are consistency describing the position of a source. [0107] Preferably, the consistency measure is only updated this way when there is just a single source identified, i.e., there is only a single ITD candidate. Further it may be determined whether the ITD candidate is close to a lag of zero, and the level difference parameter is also close to zero, indicating the signals of the audio channels are similar in level. If so, the parameters can be seen consistent, and the consistency measure may be increased. However, if there is not just a single ITD candidate or if only one of the ITD candidate and the level difference parameter is close to zero, the consistency measure may not reliably be updated. To not rely on old measures for too long time, the encoder 502 in block 1403 applies a forgetting factor to the consistency measures such that the consistency measure slowly approaches zero when there is no information to reliably update the consistency measure. [0108] The update of the consistency measure is illustrated by the following example pseudo code: if ( N_lag_pruned == 1 ) /* check whether there is a single ITD candidate */ { if ( abs( lags[0] ) >= ITD_NON_ZERO && abs( g_side ) >= ILD_THR ) { itd_con_st = ( 1.0f - xcs[0] ) * itd_con_st - xcs[0] * ( sign( g_side ) * sign( lags[0] ) );
} else if ( abs( lags[0] ) < ITD_NON_ZERO && abs( g_side ) < ILD_THR ) { /* parameters are consistent */ itd_con_st = ( 1.0f - xcs[0] ) * itd_con_st + xcs[0]; } else { /* consistency measure may not be reliably updated */ itd_con_st = ITD_CON_FORGET_FAC * itd_con_st; } } else { /* consistency measure may not be reliably updated */ itd_con_st = ITD_CON_FORGET_FAC * itd_con_st; } where N_lag_pruned is the number of sources identified, i.e., ITD candidates, ^^^_^^^_^^^^ (e.g. being 5) is used as a threshold for when the lags is considered being close to zero, ^^^_^^^ (e.g. being 0.1) is used as a threshold for when the side gain is considered being close to zero, and ^^^_^^^_^^^^^^_^^^ corresponds to the forgetting factor (e.g. being set to 0.98). Figure 15 shows a flow diagram for the update of ^^^_^^^_^^. [0109] Figure 16 illustrates an embodiment of using a consistency threshold. Turning to Figure 16, in block 1601, the encoder 502 compares the consistency measure to a consistency threshold. In block 1603, the encoder 502 responsive to the consistency measure being above the consistency threshold, uses the level difference parameter to select among positive and negative ITD candidates. In block 1605, the encoder 502, responsive to the consistency measure being below the consistency threshold, does not use the level difference parameter to select among positive and negative ITD candidates. For example, if the consistency measure is above a certain threshold ^^^_^^^_^^^, e.g., being above 0.6, the level difference parameter (in this case the side gain) is considered to be relevant for the ITD estimation and will be used to select among positive and negative ITD candidates as outlined below. [0110] However, in certain embodiments as illustrated in block 1701 of Figure 17, a further condition may be added, where the level difference parameter is considered to be relevant for the
ITD estimation only when speech signals are identified, or similarly when not music is identified. Blocks 1703 and 1705 are the same as blocks 1603 and 1605 of Figure 16. There are various existing classification methods (e.g., speech/music classifiers) that may be used for this purpose, see e.g., the 3GPP EVS codec. Further, as illustrated in block 1801 of Figure 18, in addition to the consistency measure (of block 1601 of Figure 16) being above the consistency threshold or the consistency measure (of block 1601 of Figure 16) being above the consistency threshold and when speech signals are identified or similarly when not music is identified (of block 1701 of Figure 17) the current frame's down mix energy estimate nrgDMXdB should be sufficiently larger than the previous frames smoothed long term energy estimate that is ^^^^^^^^ – ^^^^^^^^^^ > ^_^^^_^^^ for the level difference parameter to be considered to be relevant. The D_DMX_THR threshold depends on the used AR-coefficient (DMXalpha) and in some embodiments, may be in the range 10 - 20, for example 15. In the description hereinbelow, the value of 15 will be used to describe the various embodiments. Blocks 1803 and 1805 are the same as blocks 1603 and 1605 of Figure 16. [0111] If the analysis of the cross-correlation peaks has revealed that there is more than one detected peak and that there is sufficient cross-correlation detected in the input, the logic proceeds to decide if the ITD estimate should be updated and how. [0112] In case the level difference parameter is considered to be relevant for the ITD estimation, and if there are more than one ITD candidate identified and none of them are considered close to zero (e.g., using the same criteria as for the update of the consistency measure): 1. Check whether the first and second ITD candidate have different signs (i.e. ^^^^(^^^^[0]) ≠ ^^^^(^^^^[1]), where ^^^^() is a function returning the sign of the input), and the corresponding cross correlation values (i.e. ^^^[0] and ^^^[1]) are not differing more than a certain threshold ^^^_^^^_^^^ (e.g. being 0.4) and any of the following conditions are present: is there a certain level difference between the signals of the channels, i.e. |^^^^^ | > ^^_^^^ (e.g., being 0.1), or the current ^^^^^ magnitude is different from zero (e.g., >0.025) and its sign is different from the ^^^^^ of the previous frame and there is a sufficient change in ^^^^^ from last frame to current is larger than the threshold, ^^^(^^^^^ – ^^^^^^^^^) > ^^_^^^2, or the current ^^^^^ magnitude is different from zero (e.g., >0.025) and its sign is different from the ^^^^^ of the second previous frame and there is a sufficient change in ^^^^^ from the frame before the last frame to current is larger than the threshold, ^^^(^^^^^ – ^^^^2^^^^^) > ^^_^^^2.
a. If so, select an ITD being the first ITD candidate ^^^^[0] if (^^^^^ < 0 and ^^^^[0] > 0) or (^^^^^ > 0 and ^^^^[0] < 0), otherwise select an ITD being ^^^^[1] if ^^^^^ < 0 and ^^^^[1] > 0 ^^ ^^^^^ > 0 and ^^^^[1] < 0. This is illustrated as ITD Update 1 in Figures 19 and 22-25. 2. If the conditions of 1 are false, check whether there are more than two ITD candidates, and if so check if the first and the third ITD candidate have different signs (i.e. ^^^^(^^^^[0]) ≠ ^^^^(^^^^[2])), and the corresponding cross correlation values (i.e. ^^^[0] and ^^^[2]) are not differing more than a certain threshold ^^^_^^^_^^^ and there is a certain level difference between the signals of the channels, i.e. |^^^^^ | > ^^_^^^. b. If so, select an ITD being the first ITD candidate ^^^^[0] if (^^^^^ < 0 and ^^^^[0] > 0) or (^^^^^ > 0 and ^^^^[0] < 0), otherwise select an ITD being ^^^^[2] if ^^^^^ < 0 and ^^^^[2] > 0 ^^ ^^^^^ > 0 and ^^^^[2] < 0. This is illustrated as ITD Update 2 in Figures 20 and 22-25. [0113] In case the level difference parameter is not considered to be relevant for the ITD estimation or there has not been any update of the ITD from the above two conditions, the first ITD candidate ^^^^[0] may still be selected as ITD if there are only two sources identified, i.e. there are only two ITD candidates, and the first ITD candidate is different from the ITD of the previous frame, and ^^^^^^^^^^ > ^^^_^^^_^^^ (where the threshold may e.g. be ^^^_^^^_^^^ = 0.55) and ^^^^^^^^^^ < ^^^_^^^_^^^ (where the threshold may e.g. be ^^^_^^^_^^^ = 0.75) c. If so, select an ITD being the first ITD candidate ^^^^[0] if ^^^^^ < 0 and ^^^^[0] > 0 ^^ ^^^^^ > 0 and ^^^^[0] < 0 or if ^^^^^ and lags[0] are sufficiently close to 0, otherwise select and ITD being ^^^^[1] if ^^^^^^^^^^ > 0.4 and any of ^^^^^ < 0 and ^^^^[1] > 0 ^^ ^^^^^ > 0 and ^^^^[1] < 0. This is illustrated as ITD Update 3 in Figures 21-25. [0114] In other cases, the ITD estimation is not affected, i.e., the ITD value identified using the first two blocks of Figure 8 (gcc_phat() and ITD Hangover logic) is used. [0115] A flow chart with the decision logic for the multi-source detection and ITD selection logic is shown in Figure 22. [0116] In an embodiment of Figure 22, ITD_XCS_SUM_THR=0.525, ITD_NON_ZERO = 5, ITD_XCS_THR = 0.4, SG_THR = 0.1, SG THR2 = 0.09, IMP_MAG_THR = 0.55, NZ_EPS = 0.12, REL_MAG_THR = 0.75, ZERO=0.0, and APPROX_ZERO = 0.025f,
Cond_A = -sign_m0p(Gside, APPROX_ZERO)⋅sign_m0p(prev_Gside, ZERO)⋅abs(Gside- prev_Gside) > SG_THR2, Cond_B = -sign_m0p(Gside, APPROX_ZERO)⋅sign_m0p(prev2_Gside,ZERO)⋅abs(Gside- prev2_Gside) > SG_THR2, Cond_C=Not music && itd_con_st > ITD_CON_THR Cond_0 = More than one peak && PeakMagComb > ITD_XCS_SUM_THR, Cond_1 = abs(lags[0]) < ITD_NON_ZERO || abs(lags[1]) < ITD_NON_ZERO, Cond_2 = Cond_1 == 0 && sign(lags[0]) != sign(lags[1]) && abs(xcs[1]-xcs[0]) < ITD_XCS_THR && (abs(Gside) > SG_THR || Cond_A || Cond_B), Cond_3 = Cond_1 == 0 && N_lag_pruned > 2 && sign(lags[0]) != sign(lags[2]) && abs(xcs[2]-xcs[0]) < ITD_XCS_THR && abs(Gside) > SG_THR, and Cond_4 = N_lag_pruned == 2 && prev_itd != itd_lags_pruned[0] && xcs[0] / (xcs[0] + xcs[1]) > IMP_MAG_THR && xcs[1] / max( NZ_EPS, xcs[0] ) < REL_MAG_THR. [0117] Figure 23 is a flow chart of an alternative embodiment of the decision logic for the multi-source detection and ITD selection logic where the conditions Cond_A and Cond_B are handled separately without restrictions for abs(xcs[1] – xcs[0]). [0118] In an embodiment of Figure 23, ITD_XCS_SUM_THR=0.525, ITD_NON_ZERO = 5, ITD_XCS_THR = 0.4, SG_THR = 0.1, SG THR2 = 0.09, IMP_MAG_THR = 0.55, NZ_EPS = 0.12, REL_MAG_THR = 0.75, ZERO=0.0, and APPROX_ZERO = 0.025f, Cond_A = -sign_m0p(Gside, APPROX_ZERO)⋅sign_m0p(prev_Gside, ZERO)⋅abs(Gside- prev_Gside) > SG_THR2, Cond_B = -sign_m0p(Gside, APPROX_ZERO)⋅sign_m0p(prev2_Gside,ZERO)⋅abs(Gside- prev2_Gside) > SG_THR2, Cond_C=Not music && itd_con_st > ITD_CON_THR Cond_0 = More than one peak && PeakMagComb > ITD_XCS_SUM_THR, Cond_1 = abs(lags[0]) < ITD_NON_ZERO || abs(lags[1]) < ITD_NON_ZERO, Cond_2 = Cond_1 == 0 && sign(lags[0]) != sign(lags[1]) && abs(xcs[1]-xcs[0]) < ITD_XCS_THR && (abs(Gside) > SG_THR Cond_3 = Cond_1 == 0 && N_lag_pruned > 2 && sign(lags[0]) != sign(lags[2]) && abs(xcs[2]-xcs[0]) < ITD_XCS_THR && abs(Gside) > SG_THR, Cond_4 = N_lag_pruned == 2 && prev_itd != itd_lags_pruned[0] && xcs[0] / (xcs[0] + xcs[1]) > IMP_MAG_THR && xcs[1] / max( NZ_EPS, xcs[0] ) < REL_MAG_THR, and
Cond_5 = Cond_1 == 0 && sign(lags[0]) != sign(lags[1]) && ( Cond_A || Cond_B ) [0119] Figure 24 is a flow chart with the decision logic for the multi-source detection and ITD selection logic similar to Figure 21 with the addition of the energy estimation of the downmix channel for the energy tracking of the input signal. [0120] In an embodiment of Figure 24, ITD_XCS_SUM_THR=0.525, ITD_NON_ZERO = 5, ITD_XCS_THR = 0.4, SG_THR = 0.1, SG THR2 = 0.09, IMP_MAG_THR = 0.55, NZ_EPS = 0.12, REL_MAG_THR = 0.75, D_DMX_THR=15, ZERO=0.0, and APPROX_ZERO = 0.025f, Cond_A = -sign_m0p(Gside, APPROX_ZERO)⋅sign_m0p(prev_Gside, ZERO)⋅abs(Gside- prev_Gside) > SG_THR2, Cond_B = -sign_m0p(Gside, APPROX_ZERO)⋅sign_m0p(prev2_Gside,ZERO)⋅abs(Gside- prev2_Gside) > SG_THR2, Cond_C=Not music && itd_con_st>ITD_CON_THR && (nrg_DMX_dB – nrg_DMX_dB_stlp) > D_DMX_THR Cond_0 = More than one peak && PeakMagComb > ITD_XCS_SUM_THR, Cond_1 = abs(lags[0]) < ITD_NON_ZERO || abs(lags[1]) < ITD_NON_ZERO, Cond_2 = Cond_1 == 0 && sign(lags[0]) != sign(lags[1]) && abs(xcs[1]-xcs[0]) < ITD_XCS_THR && (abs(Gside) > SG_THR || Cond_A || Cond_B), Cond_3 = Cond_1 == 0 && N_lag_pruned > 2 && sign(lags[0]) != sign(lags[2]) && abs(xcs[2]-xcs[0]) < ITD_XCS_THR && abs(Gside) > SG_THR, and Cond_4 = N_lag_pruned == 2 && prev_itd != itd_lags_pruned[0] && xcs[0] / (xcs[0] + xcs[1]) > IMP_MAG_THR && xcs[1] / max( NZ_EPS, xcs[0] ) < REL_MAG_THR. [0121] Figure 25 is a flow chart of an alternative embodiment of the decision logic for the multi-source detection and ITD selection logic where the conditions Cond_A and Cond_B are handled separately without restrictions for abs(xcs[1] – xcs[0]) similar to Figure 23 with the addition of the energy estimation of the downmix channel for the energy tracking of the input signal. [0122] In an embodiment of Figure 25, ITD_XCS_SUM_THR=0.525, ITD_NON_ZERO = 5, ITD_XCS_THR = 0.4, SG_THR = 0.1, SG THR2 = 0.09, IMP_MAG_THR = 0.55, NZ_EPS = 0.12, REL_MAG_THR = 0.75, D_DMX_THR=15, ZERO=0.0, and APPROX_ZERO = 0.025f, Cond_A = -sign_m0p(Gside, APPROX_ZERO)⋅sign_m0p(prev_Gside, ZERO)⋅abs(Gside-
prev_Gside) > SG_THR2, Cond_B = -sign_m0p(Gside, APPROX_ZERO)⋅sign_m0p(prev2_Gside,ZERO)⋅abs(Gside- prev2_Gside) > SG_THR2, Cond_C=Not music && itd_con_st>ITD_CON_THR && (nrg_DMX_dB – nrg_DMX_dB_stlp) > D_DMX_THR Cond_0 = More than one peak && PeakMagComb > ITD_XCS_SUM_THR, Cond_1 = abs(lags[0]) < ITD_NON_ZERO || abs(lags[1]) < ITD_NON_ZERO, Cond_2 = Cond_1 == 0 && sign(lags[0]) != sign(lags[1]) && abs(xcs[1]-xcs[0]) < ITD_XCS_THR && (abs(Gside) > SG_THR Cond_3 = Cond_1 == 0 && N_lag_pruned > 2 && sign(lags[0]) != sign(lags[2]) && abs(xcs[2]-xcs[0]) < ITD_XCS_THR && abs(Gside) > SG_THR, Cond_4 = N_lag_pruned == 2 && prev_itd != itd_lags_pruned[0] && xcs[0] / (xcs[0] + xcs[1]) > IMP_MAG_THR && xcs[1] / max( NZ_EPS, xcs[0] ) < REL_MAG_THR, and Cond_5 = Cond_1 == 0 && sign(lags[0]) != sign(lags[1]) && ( Cond_A || Cond_B ) [0123] There are some alternative positions where the new multi-source detection step can be performed than shown in Figure 8. For example, in some embodiments, it could be performed before the ITD hangover, as shown in Figure 26. If done before the ITD hangover logic one may need to also update the ITD hangover logic so that it considers information gained in the multi- source detection step. [0124] Another option would be to allow the multi-source detection to replace ITD hangover logic in cases where multiple sources are detected, as illustrated in Figure 27. [0125] It would be possible to use an alternative way to calculate the filtered energy estimate. One option would be to use different AR-coefficients if the current frame value nrgDMXdB is large than nrgDMXdBst and another if it is lower. That is if nrgDMXdB>nrgDMXdBst then use ^^^^^^^^^^ = (1.0 − ^^^^^^ℎ^^^) ^^^^^^^^^^ + ^^^^^^ℎ^^^ ^^^^^^^^ otherwise use ^^^^^^^^^^ = (1.0 − ^^^^^^ℎ^^^^^) ^^^^^^^^^^ + ^^^^^^ℎ^^^^^ ^^^^^^^^ where the coefficient ^^^^^^ℎ^^^ is in the range 0.10-0.25 and the coefficient ^^^^^^ℎ^^^^^ would be larger and in the range 0.30 – 0.50. This would allow the smoothed long term estimate to be more responsive when the energy of the input is dropping. This could also affect the threshold D_DMX_THR – however, it would most likely still be in the range 10 – 30.
[0126] Figure 28 shows an audio encoder 502 (e.g., a parametric stereo encoder) in accordance with some embodiments where the audio encoder 502 is implemented as a stand- alone device. As used herein, an audio encoder refers to a device capable, configured, arranged and/or operable to encode objects and communicate with network nodes, encoders, and/or decoders. Examples of an audio encoder include, but are not limited to, a smart phone, mobile phone, cell phone, voice over IP (VoIP) phone, wireless local loop phone, desktop computer, personal digital assistant (PDA), wireless cameras, gaming console or device, storage device, playback appliance, wearable terminal device, wireless endpoint, mobile station, tablet, laptop, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), smart device, wireless customer-premise equipment (CPE), vehicle-mounted or vehicle embedded/integrated wireless device, etc. [0127] An audio encoder may support device-to-device (D2D) communication, for example by implementing a 3GPP standard for sidelink communication, Dedicated Short-Range Communication (DSRC), vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), or vehicle- to-everything (V2X). In other examples, an encoder may not necessarily have a user in the sense of a human user who owns and/or operates the relevant device. [0128] The audio encoder 502 includes processing circuitry 2802 that is operatively coupled via a bus 2804 to an input/output interface 2806, a power source 2808, a memory 2810, a communication interface 2812, and/or any other component, or any combination thereof. Certain encoders may utilize all or a subset of the components shown in Figure 28. The level of integration between the components may vary from one encoder to another encoder. Further, certain encoders may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc. [0129] The processing circuitry 2802 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory 2810. The processing circuitry 2802 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry 2802 may include multiple central processing units (CPUs). [0130] In the example, the input/output interface 2806 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and/or output
devices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into the audio encoder 502. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presence- sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device. [0131] In some embodiments, the power source 2808 is structured as a battery or battery pack. Other types of power sources, such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used. The power source 2808 may further include power circuitry for delivering power from the power source 2808 itself, and/or an external power source, to the various parts of the audio encoder 502 via input circuitry or an interface such as an electrical power cable. Delivering power may be, for example, for charging of the power source 2808. Power circuitry may perform any formatting, converting, or other modification to the power from the power source 2808 to make the power suitable for the respective components of the audio encoder 502 to which power is supplied. [0132] The memory 2810 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read- only memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory 2810 includes one or more application programs 2814, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data 2816. The memory 2810 may store, for use by the audio encoder 502, any of a variety of various operating systems or combinations of operating systems. [0133] The memory 2810 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data
storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a USIM and/or ISIM, other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUICC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory 2810 may allow the audio encoder 502 to access instructions, application programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 2810, which may be or comprise a device-readable storage medium. [0134] The processing circuitry 2802 may be configured to communicate with an access network or other network using the communication interface 2812. The communication interface 2812 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna 2822. The communication interface 2812 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., a UE or a network node in an access network). Each transceiver may include a transmitter 2818 and/or a receiver 2820 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth). Moreover, the transmitter 2818 and receiver 2820 may be coupled to one or more antennas (e.g., antenna 2822) and may share circuit components, software or firmware, or alternatively be implemented separately. [0135] In the illustrated embodiment, communication functions of the communication interface 2812 may include cellular communication, Wi-Fi communication, LPWAN communication, data communication, voice communication, multimedia communication, short- range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented in according to one or more communication protocols and/or standards, such as IEEE 802.11, Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol/internet protocol (TCP/IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), QUIC, Hypertext Transfer Protocol (HTTP), and so forth. [0136] Regardless of the type of sensor, an audio object renderer may provide an output of
decoded data, through its communication interface 2812, via a wireless connection to a network node. [0137] An audio encoder when in the form of an Internet of Things (IoT) device, may be a device for use in one or more application domains, these domains comprising, but not limited to, city wearable technology, extended industrial application and healthcare. Non-limiting examples of such an IoT device are a device which is or which is embedded in: a connected refrigerator or freezer, a TV, a connected lighting device, an electricity meter, a robot vacuum cleaner, a voice controlled smart speaker, a home security camera, a thermostat, an electrical door lock, a connected doorbell, an autonomous vehicle, a surveillance system, a weather monitoring device, a vehicle parking monitoring device, an electric vehicle charging station, a smart watch, a fitness tracker, a head-mounted display for Augmented Reality (AR) or Virtual Reality (VR), a wearable for tactile augmentation or sensory enhancement. A decoder in the form of an IoT device comprises circuitry and/or software in dependence of the intended application of the IoT device in addition to other components as described in relation to the audio encoder 502 shown in Figure 28. [0138] Figure 29 is a block diagram of a host 506 in accordance with various aspects described herein. As used herein, the host 506 may be or comprise various combinations hardware and/or software, including a standalone server, a blade server, a cloud-implemented server, a distributed server, a virtual machine, container, or processing resources in a server farm. The host 506 may provide one or more services to one or more UEs. [0139] The host 506 includes processing circuitry 2902 that is operatively coupled via a bus 2904 to an input/output interface 2906, a network interface 2908, a power source 2910, and a memory 2912. Other components may be included in other embodiments. Features of these components may be substantially similar to those described with respect to the devices of previous figures, such as Figure 28, such that the descriptions thereof are generally applicable to the corresponding components of host 506. [0140] The memory 2912 may include one or more computer programs including one or more host application programs 2914 and data 2916, which may include user data, e.g., data generated by a UE for the host 506 or data generated by the host 506 for a UE. Embodiments of the host 506 may utilize only a subset or all of the components shown. The host application programs 2914 may be implemented in a container-based architecture and may provide support for video codecs (e.g., Versatile Video Coding (VVC), High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), MPEG, VP9) and audio codecs (e.g., FLAC, Advanced Audio Coding (AAC), MPEG, G.711, EVS, IVAS), including transcoding for multiple different classes,
types, or implementations of UEs (e.g., handsets, desktop computers, wearable display systems, heads-up display systems). The host application programs 2914 may also provide for user authentication and licensing checks and may periodically report health, routes, and content availability to a central node, such as a device in or on the edge of a core network. Accordingly, the host 506 may select and/or indicate a different host for over-the-top services for a UE. The host application programs 2914 may support various protocols, such as the HTTP Live Streaming (HLS) protocol, Real-Time Messaging Protocol (RTMP), Real-Time Streaming Protocol (RTSP), Dynamic Adaptive Streaming over HTTP (MPEG-DASH), etc. [0141] Figure 30 is a block diagram illustrating a virtualization environment 3000 in which functions implemented by some embodiments of the audio encoder 502 or components of the audio encoder 502 may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments 3000 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a decoder, encoder, network node, UE, core network node, or host. Further, in embodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized. [0142] Applications 3002 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment 3000 to implement some of the features, functions, and/or benefits of some of the embodiments disclosed herein. [0143] Hardware 3004 includes processing circuitry, memory that stores software and/or instructions executable by hardware processing circuitry, and/or other hardware devices as described herein, such as a network interface, input/output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers 3006 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs 3008A and 3008B (one or more of which may be generally referred to as VMs 3008), and/or perform any of the functions, features and/or benefits described in relation with some embodiments described herein. The virtualization layer 3006 may present a virtual operating platform that appears like networking hardware to the VMs 3008.
[0144] The VMs 3008 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer 3006. Different embodiments of the instance of a virtual appliance 3002 may be implemented on one or more of VMs 3008, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment. [0145] In the context of NFV, a VM 3008 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of the VMs 3008, and that part of hardware 3004 that executes that VM, be it hardware dedicated to that VM and/or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs 3008 on top of the hardware 3004 and corresponds to the application 3002. [0146] Hardware 3004 may be implemented in a standalone network node with generic or specific components. Hardware 3004 may implement some functions via virtualization. Alternatively, hardware 3004 may be part of a larger cluster of hardware (e.g., such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration 3010, which, among others, oversees lifecycle management of applications 3002. In some embodiments, hardware 3004 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. In some embodiments, some signaling can be provided with the use of a control system 3012 which may alternatively be used for communication between hardware nodes and radio units. [0147] Although the computing devices described herein (e.g., decoders, audio object renderers, encoders, hosts) may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and/or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for
example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and/or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and/or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware. [0148] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer- readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer- readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device but are enjoyed by the computing device as a whole, and/or by end users and a wireless network generally. [0149] Example embodiments 1. A method in an encoder to estimate an inter-channel time difference, ITD, between two waveforms, the method comprising: obtaining (901) an estimate of a threshold based on a shape of a cross-correlation estimation between the two waveforms; determining (903) a second threshold based on a maximum value of the first threshold and a non-zero constant; determining (905) at least one lag for which of cross-correlation estimate magnitude value is above the second threshold to form a list of potential peak positions; determining (907) an indication of how many true peaks there are and a list of lags of the
true peaks; responsive to there being three or more true peaks, keeping three lags (909) having a higher cross-correlation estimate magnitude value than other true peaks and setting lags and cross-correlation estimate magnitude values of the other true peaks to zero to form a list of three candidate ITD estimates; responsive to there being less than three true peaks, keeping (911) the lags and cross- correlation estimate magnitude values of the true peaks and setting lags and cross-correlation estimate magnitude values of the other true peaks to zero to form the list of three candidate ITD estimates; and estimating (913) the ITD based on the list of three candidate ITD estimates and a consistency measure between estimated ITD values and level difference parameters. 2. The method of Embodiment 1, wherein determining an indication of how many true peaks there are comprises: determining (1001) that a possible peak is a true peak when a cross-correlation estimate magnitude value of the possible peak is larger than neighbor positions of the possible peak; and removing (1003) possible peaks that are too close to each other by determining when lags between the possible peaks are less than a constant apart, keeping only a lag corresponding to a largest cross-correlation estimate magnitude value. 3. The method of any of Embodiments 1-2, further comprising: determining (1301) whether or not to utilize level difference information for ITD estimation based on a consistency measure between estimated ITD values and level difference parameters. 4. The method of Embodiment 3, further comprising determining (1401) the consistency measure. 5. The method of Embodiment 4, wherein determining the consistency measure comprises determining the consistency measure in accordance with
where ^^^_^^^_^^ is an updated consistency measure, ^^^_^^^_^^^^^^^^^ is the consistency measurement being updated, ^^^^^ (^) is a side gain parameter, ^^^^[0] is a ITD candidate corresponding to the largest correlation among the ITD candidates, and ^^^^ is a filtering parameter which may be set to any value between 0 and 1 or be adaptively set based on the cross correlation value corresponding to ^^^^[0].
6. The method of any of Embodiments 4-5, further comprising applying (1403) a forgetting factor to the consistency measure such that the consistency measure slowly approaches zero when there is no information to reliably update the consistency measure. 7. The method of any of Embodiments 4-5, further comprising updating the consistency measure when there is only one ITD candidate identified. 8. The method of any of Embodiments 3-7, further comprising determining (1601) if the consistency measure is above a consistency threshold. 9. The method of Embodiment 8, further comprising: responsive to the consistency measure being above the consistency threshold, using 1603) the level difference parameter to select among positive and negative ITD candidates. 10. The method of Embodiment 8, further comprising: responsive to the consistency measure being above the consistency threshold and speech signals are identified or no music is identified, using the level difference parameter to select among positive and negative ITD candidates. 11. The method of Embodiment 8, further comprising: responsive to the consistency measure being above the consistency threshold and a current frame down mix energy estimate is an energy threshold above a previous frames long term energy estimate, using 1603) the level difference parameter to select among positive and negative ITD candidates. 12. The method of Embodiment 8, further comprising: responsive to the consistency measure being above the consistency threshold, speech signals are identified or no music is identified, and a current frame down mix energy estimate is an energy threshold above a previous frames long term energy estimate, using the level difference parameter to select among positive and negative ITD candidates. 13. The method of any of Embodiments 3-12, wherein ^^^^[0] is a ITD candidate corresponding to a largest cross-correlation estimate magnitude value among the ITD candidates, ^^^^[1] is a ITD candidate corresponding to a second largest cross-correlation estimate magnitude value among the ITD candidates, ^^^^[2] is a ITD candidate corresponding to a third largest correlation among the ITD candidates, ^^^[0] is a cross-correlation estimate magnitude value for ^^^^[0], ^^^[1] is a cross-correlation estimate magnitude value for ^^^^[1], ^^^[2] is
a cross-correlation estimate magnitude value for ^^^^[2], and ^^^^^ is a side gain parameter, the method further comprising: if there are more than one ITD candidate identified and none of them are considered close to zero based on an ITD candidate distance to zero being above an ITD non-zero threshold and the level difference parameter is being used: responsive to ^^^^(^^^^[0]) ≠ ^^^^(^^^^[1]) and ^^^[0] and ^^^[1] do not differ more than a cross correlation threshold, select an ITD to be ^^^^[0] if (^^^^^ < 0 and ^^^^[0] > 0) or (^^^^^ > 0 and ^^^^[0] < 0), otherwise select the ITD to be ^^^^[1]; and responsive to ^^^^(^^^^[0]) = ^^^^(^^^^[1]) or ^^^[0] and ^^^[1] differ more than the cross correlation threshold, determining if ^^^^(^^^^[0]) ≠ ^^^^(^^^^[2]) and ^^^[0] and ^^^[2] do not differ more than a cross correlation threshold and a magnitude of ^^^^^ is greater than a side gain threshold, select an ITD to be ^^^^[0] if (^^^^^ < 0 and ^^^^[0] > 0) or (^^^^^ > 0 and ^^^^[0] < 0), otherwise select an ITD to be ^^^^[2]. 14. The method of any of Embodiments 3-12, wherein ^^^^[0] is a ITD candidate corresponding to a largest cross-correlation estimate magnitude value among the ITD candidates, ^^^^[1] is a ITD candidate corresponding to a second largest cross-correlation estimate magnitude value among the ITD candidates, ^^^^[2] is a ITD candidate corresponding to a third largest correlation among the ITD candidates, ^^^[0] is a cross-correlation estimate magnitude value for ^^^^[0], ^^^[1] is a cross-correlation estimate magnitude value for ^^^^[1], ^^^[2] is a cross-correlation estimate magnitude value for ^^^^[2], ^^^^^ is a side gain parameter, SG_THR and SG_THR2 are thresholds, the method further comprising: if there are more than one ITD candidate identified and none of them are considered close to zero based on an ITD candidate distance to zero being above an ITD non-zero threshold and the level difference parameter is being used: responsive to ^^^^(^^^^[0]) ≠ ^^^^(^^^^[1]) and ^^^[0] and ^^^[1] do not differ more than a cross correlation threshold and at least one of: |^^^^^ | > ^^_^^^ or a current ^^^^^ magnitude is different from zero and a sign of the current ^^^^^ is different than the sign of a previous frame ^^^^^ and ^^^(^^^^^ − ^^^^^^^^^ ) > ^^_^^^2 or the current ^^^^^ magnitude is different from zero and the sign of the current ^^^^^ is different from the ^^^^^ of a second previous frame and ^^^(^^^^^ − ^^^^2^^^^^) > ^^_^^^2, select an ITD to be ^^^^[0] if (^^^^^ < 0 and ^^^^[0] > 0) or (^^^^^ > 0 and
^^^^[0] < 0), otherwise select the ITD to be ^^^^[1]; and responsive to ^^^^(^^^^[0]) = ^^^^(^^^^[1]) and ^^^[0] and ^^^[1] differ more than the cross correlation threshold and |^^^^^ | < ^^_^^^, the current ^^^^^ magnitude is < 0.025, and a sign of the current ^^^^^ is not different than the sign of a previous frame ^^^^^ and ^^^(^^^^^ − ^^^^^^^^^ ) < ^^_^^^2 and the sign of the current ^^^^^ is not different from the ^^^^^ of a second previous frame and ^^^(^^^^^ − ^^^^2^^^^^ ) < ^^_^^^2, determining if ^^^^(^^^^[0]) ≠ ^^^^(^^^^[2]) and ^^^[0] and ^^^[2] do not differ more than a cross correlation threshold and a magnitude of ^^^^^ is greater than a side gain threshold, select an ITD to be ^^^^[0] if (^^^^^ < 0 and ^^^^[0] > 0) or (^^^^^ > 0 and ^^^^[0] < 0), otherwise select an ITD to be ^^^^[2]. 15. The method of any of Embodiments 8-13, further comprising: responsive to the consistency measure being below the consistency threshold, not using (1605) the level difference parameter to select among positive and negative ITD candidates. 16. The method of Embodiment 15, wherein ^^^^[0] is a ITD candidate corresponding to a largest correlation among the ITD candidates, the method further comprising: selecting ^^^^[0] to be the ITD if there are only two sources identified, and ^^^^[0] is different from an ITD of the previous frame, a peak importance magnitude is greater than an importance magnitude threshold, and peak relative importance is less than a relative magnitude threshold. 17. An encoder (502, 3002) comprising: processing circuitry (2802); and memory (2810) coupled with the processing circuitry, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations comprising: obtaining (901) an estimate of a threshold based on a shape of a cross-correlation estimation between the two waveforms; determining (903) a second threshold based on a maximum value of the first threshold and a non-zero constant; determining (905) at least one lag for which of cross-correlation estimate magnitude value is above the second threshold to form a list of potential peak positions; determining (907) an indication of how many true peaks there are and a list of lags
of the true peaks; responsive to there being three or more true peaks, keeping three lags (909) having a higher cross-correlation estimate magnitude value than other true peaks and setting lags and cross-correlation estimate magnitude values of the other true peaks to zero to form a list of three candidate ITD estimates; responsive to there being less than three true peaks, keeping (911) the lags and correlation of the true peaks and setting lags and cross-correlation estimate magnitude values of the other true peaks to zero to form the list of three candidate ITD estimates; and estimating (913) the ITD based on the list of three candidate ITD estimates. 18. The encoder (502, 3002) of Embodiment 17, wherein determining an indication of how many true peaks there are comprises: determining (1001) that a possible peak is a true peak when a cross-correlation estimate magnitude value of the possible peak is larger than neighbor positions of the possible peak; and removing (1003) possible peaks that are too close to each other by determining when lags between the possible peaks are less than a constant apart, keeping only a lag corresponding to a largest cross-correlation estimate magnitude value. 19. The encoder (502, 3002) of any of Embodiments 18-17, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising: determining (1301) whether or not to utilize level difference information for ITD estimation based on a consistency measure between estimated ITD values and level difference parameters. 20. The encoder (502, 3002) of Embodiment 19, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising determining (1401) the consistency measure. 21. The encoder (502, 3002) of Embodiment 20, wherein determining the consistency measure comprises determining the consistency measure in accordance with
where ^^^_^^^_^^ is an updated consistency measure, ^^^_^^^_^^^^^^^^^ is the consistency measure being updated, ^^^^^ (^) is a side gain parameter, ^^^^[0] is a ITD candidate corresponding to the largest correlation among the ITD candidates, and ^^^^ is a filtering parameter which may be set to any value between 0 and 1 or be adaptively set based on the cross
correlation value corresponding to ^^^^[0]. 22. The encoder (502, 3002) of any of Embodiments 20-21, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising applying (1403) a forgetting factor to the consistency measure such that the consistency measure slowly approaches zero when there is no information to reliably update the consistency measure. 23. The encoder (502, 3002) of any of Embodiments 20-21, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising updating the consistency measure when there is only one ITD candidate identified. 24. The encoder (502, 3002) of any of Embodiments 19-23, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising determining (1601) if the consistency measure is above a consistency threshold. 25. The encoder (502, 3002) of Embodiment 24, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising: responsive to the consistency measure being above the consistency threshold, using 1603) the level difference parameter to select among positive and negative ITD candidates. 26. The encoder (502, 3002) of Embodiment 24, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising: responsive to the consistency measure being above the consistency threshold and speech signals are identified or no music is identified, using the level difference parameter to select among positive and negative ITD candidates. 27. The encoder (502, 3002) of Embodiment 24, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising: responsive to the consistency measure being above the consistency threshold and a current frame down mix energy estimate is an energy threshold above a previous frames long term energy estimate, using 1603) the level difference parameter to select among positive and
negative ITD candidates. 28. The encoder (502, 3002) of Embodiment 24, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising: responsive to the consistency measure being above the consistency threshold, speech signals are identified or no music is identified, and a current frame down mix energy estimate is an energy threshold above a previous frames long term energy estimate, using the level difference parameter to select among positive and negative ITD candidates. 29. The encoder (502, 3002) of any of Embodiments 19-29, wherein ^^^^[0] is a ITD candidate corresponding to a largest cross-correlation estimate magnitude value among the ITD candidates, ^^^^[1] is a ITD candidate corresponding to a second largest cross-correlation estimate magnitude value among the ITD candidates, ^^^^[2] is a ITD candidate corresponding to a third largest cross-correlation estimate magnitude value among the ITD candidates, ^^^[0] is a cross-correlation estimate magnitude value for ^^^^[0], ^^^[1] is a cross-correlation estimate magnitude value for ^^^^[1], ^^^[2] is a cross-correlation estimate magnitude value for ^^^^[2], and ^^^^^ is a side gain parameter, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising: if there are more than one ITD candidate identified and none of them are considered close to zero based on an ITD candidate distance to zero being above an ITD non-zero threshold and the level difference parameter is being used: responsive to ^^^^(^^^^[0]) ≠ ^^^^(^^^^[1]) and ^^^[0] and ^^^[1] do not differ more than a cross correlation threshold, select an ITD to be ^^^^[0] if (^^^^^ < 0 and ^^^^[0] > 0) or (^^^^^ > 0 and ^^^^[0] < 0), otherwise select the ITD to be ^^^^[1]; and responsive to ^^^^(^^^^[0]) = ^^^^(^^^^[1]) or ^^^[0] and ^^^[1] differ more than the cross correlation threshold, determining if ^^^^(^^^^[0]) ≠ ^^^^(^^^^[2]) and ^^^[0] and ^^^[2] do not differ more than a cross correlation threshold and a magnitude of ^^^^^ is greater than a side gain threshold, select an ITD to be ^^^^[0] if (^^^^^ < 0 and ^^^^[0] > 0) or (^^^^^ > 0 and ^^^^[0] < 0), otherwise select an ITD to be ^^^^[2]. 30. The encoder (502, 3002) of any of Embodiments 19-29, wherein ^^^^[0] is a ITD
candidate corresponding to a largest cross-correlation estimate magnitude value among the ITD candidates, ^^^^[1] is a ITD candidate corresponding to a second largest cross-correlation estimate magnitude value among the ITD candidates, ^^^^[2] is a ITD candidate corresponding to a third largest correlation among the ITD candidates, ^^^[0] is a cross-correlation estimate magnitude value for ^^^^[0], ^^^[1] is a cross-correlation estimate magnitude value for ^^^^[1], ^^^[2] is a cross-correlation estimate magnitude value for ^^^^[2], ^^^^^ is a side gain parameter, SG_THR and SG_THR2 are thresholds, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising: if there are more than one ITD candidate identified and none of them are considered close to zero based on an ITD candidate distance to zero being above an ITD non-zero threshold and the level difference parameter is being used: responsive to ^^^^(^^^^[0]) ≠ ^^^^(^^^^[1]) and ^^^[0] and ^^^[1] do not differ more than a cross correlation threshold and at least one of: |^^^^^| > ^^_^^^ or a current ^^^^^ magnitude is different from zero and a sign of the current ^^^^^ is different than the sign of a previous frame ^^^^^ and ^^^(^^^^^ − ^^^^^^^^^) > ^^_^^^2 or the current ^^^^^ magnitude is different from zero and the sign of the current ^^^^^ is different from the ^^^^^ of a second previous frame and ^^^(^^^^^ − ^^^^2^^^^^ ) > ^^_^^^2, select an ITD to be ^^^^[0] if (^^^^^ < 0 and ^^^^[0] > 0) or (^^^^^ > 0 and ^^^^[0] < 0), otherwise select the ITD to be ^^^^[1]; and responsive to ^^^^(^^^^[0]) = ^^^^(^^^^[1]) and ^^^[0] and ^^^[1] differ more than the cross correlation threshold and |^^^^^| < ^^_^^^, the current ^^^^^ magnitude is < 0.025, and a sign of the current ^^^^^ is not different than the sign of a previous frame ^^^^^ and ^^^(^^^^^ − ^^^^^^^^^ ) < ^^_^^^2 and the sign of the current ^^^^^ is not different from the ^^^^^ of a second previous frame and ^^^(^^^^^ − ^^^^2^^^^^ ) < ^^^^^^, determining if ^^^^(^^^^[0]) ≠ ^^^^(^^^^[2]) and ^^^[0] and ^^^[2] do not differ more than a cross correlation threshold and a magnitude of ^^^^^ is greater than a side gain threshold, select an ITD to be ^^^^[0] if (^^^^^ < 0 and ^^^^[0] > 0) or (^^^^^ > 0 and ^^^^[0] < 0), otherwise select an ITD to be ^^^^[2]. 31. The encoder (502, 3002) of any of Embodiments 24-30, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising: responsive to the consistency measure being below the consistency threshold, not using
(1605) the level difference parameter to select among positive and negative ITD candidates. 32. The encoder (502, 3002) of Embodiment 31, wherein ^^^^[0] is a ITD candidate corresponding to a largest cross-correlation estimate magnitude value among the ITD candidates, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder (502, 3002) to perform operations further comprising: selecting ^^^^[0] to be the ITD if there are only two sources identified, and ^^^^[0] is different from an ITD of the previous frame, a peak importance magnitude is greater than an importance magnitude threshold, and peak relative importance is less than a relative magnitude threshold 33. An encoder (502, 3002) adapted to perform according to any of Embodiments 1-16. 34. A computer program comprising program code to be executed by processing circuitry (2802) of an encoder (502, 3002), whereby execution of the program code causes the encoder (502, 3002) to perform operations according to any of Embodiments 1-16. 35. A computer program product comprising a non-transitory storage medium including program code to be executed by processing circuitry (2802) of an encoder (502, 3002), whereby execution of the program code causes the encoder (502, 3002) to perform operations according to any of Embodiments 1-16.
Claims
CLAIMS 1. A method in an encoder to estimate an inter-channel time difference, ITD, between two waveforms, the method comprising: obtaining (901) an estimate of a threshold based on a shape of a cross-correlation function between the two waveforms; determining (903) a second threshold based on a maximum value of the first threshold and a non-zero constant; determining (905) at least one lag for which of cross-correlation estimate magnitude value is above the second threshold to form a list of potential peak positions; determining (907) an indication of how many true peaks there are and a list of lags of the true peaks; responsive to there being three or more true peaks, keeping three lags (909) having a higher cross-correlation estimate magnitude value than other true peaks and setting lags and cross-correlation estimate magnitude values of the other true peaks to zero to form a list of three candidate ITD estimates; responsive to there being less than three true peaks, keeping (911) the lags and cross- correlation estimate magnitude values of the true peaks and setting lags and cross-correlation estimate magnitude values of remaining positions to zero to form the list of three candidate ITD estimates; and estimating (913) the ITD based on the list of three candidate ITD estimates and a consistency measure between estimated ITD values and a level difference parameter.
2. The method of Claim 1, wherein determining an indication of how many true peaks there are comprises: determining (1001) that a possible peak is a true peak when a cross-correlation estimate magnitude value of the possible peak is larger than neighbor positions of the possible peak; and removing (1003) possible peaks that are too close to each other by determining when lags between the possible peaks are less than a constant apart, keeping only a lag corresponding to a largest cross-correlation estimate magnitude value.
3. The method of any of Claims 1-2, further comprising: determining (1301) whether or not to utilize level difference information for ITD estimation based on a consistency measure between estimated ITD values and the level difference parameter.
4. The method of Claim 3, further comprising determining (1401) the consistency measure.
5. The method of Claim 4, wherein determining the consistency measure comprises determining the consistency measure in accordance with
where ^^^_^^^_^^ is an updated consistency measure, ^^^_^^^_^^^^^^^^^ is the consistency measurement being updated, ^^^^^ (^) is a side gain parameter, ^^^^[0] is a ITD candidate corresponding to the largest correlation among the ITD candidates, and ^^^^ is a filtering parameter which may be set to any value between 0 and 1 or be adaptively set based on the cross correlation value corresponding to ^^^^[0].
6. The method of any of Claims 4-5, further comprising applying (1403) a forgetting factor to the consistency measure such that the consistency measure slowly approaches zero when there is no information to reliably update the consistency measure.
7. The method of any of Claims 4-5, further comprising updating the consistency measure when there is only one true peak identified.
8. The method of any of Claims 3-7, further comprising determining (1601) if the consistency measure is above a consistency threshold.
9. The method of Claim 8, further comprising: responsive to the consistency measure being above the consistency threshold, using (1603) the level difference parameter to select among positive and negative ITD candidates.
10. The method of Claim 8, further comprising: responsive to the consistency measure being above the consistency threshold and speech signals are identified or no music is identified, using the level difference parameter to select among positive and negative ITD candidates.
11. The method of Claim 8, further comprising: responsive to the consistency measure being above the consistency threshold and a current frame down mix energy estimate is an energy threshold above a previous frames long term energy estimate, using (1603) the level difference parameter to select among positive and negative ITD candidates.
12. The method of Claim 8, further comprising:
responsive to the consistency measure being above the consistency threshold, speech signals are identified or no music is identified, and a current frame down mix energy estimate is an energy threshold above a previous frames long term energy estimate, using the level difference parameter to select among positive and negative ITD candidates.
13. The method of any of Claims 3-12, wherein ^^^^[0] is a ITD candidate corresponding to a largest cross-correlation estimate magnitude value among the ITD candidates, ^^^^[1] is a ITD candidate corresponding to a second largest cross-correlation estimate magnitude value among the ITD candidates, ^^^^[2] is a ITD candidate corresponding to a third largest correlation among the ITD candidates, ^^^[0] is a cross-correlation estimate magnitude value for ^^^^[0], ^^^[1] is a cross-correlation estimate magnitude value for ^^^^[1], ^^^[2] is a cross-correlation estimate magnitude value for ^^^^[2], and ^^^^^ is a side gain parameter, the method further comprising: if there are more than one ITD candidate identified and none of them are considered close to zero based on an ITD candidate distance to zero being above an ITD non-zero threshold and the level difference parameter is being used: responsive to ^^^^(^^^^[0]) ≠ ^^^^(^^^^[1]) and ^^^[0] and ^^^[1] do not differ more than a cross correlation threshold, select an ITD to be ^^^^[0] if (^^^^^ < 0 and ^^^^[0] > 0) or (^^^^^ > 0 and ^^^^[0] < 0), otherwise select the ITD to be ^^^^[1]; and responsive to ^^^^(^^^^[0]) = ^^^^(^^^^[1]) or ^^^[0] and ^^^[1] differ more than the cross correlation threshold, determining if ^^^^(^^^^[0]) ≠ ^^^^(^^^^[2]) and ^^^[0] and ^^^[2] do not differ more than a cross correlation threshold and a magnitude of ^^^^^ is greater than a side gain threshold, select an ITD to be ^^^^[0] if (^^^^^ < 0 and ^^^^[0] > 0) or (^^^^^ > 0 and ^^^^[0] < 0), otherwise select an ITD to be ^^^^[2].
14. The method of any of Claims 3-12, wherein ^^^^[0] is a ITD candidate corresponding to a largest cross-correlation estimate magnitude value among the ITD candidates, ^^^^[1] is a ITD candidate corresponding to a second largest cross-correlation estimate magnitude value among the ITD candidates, ^^^^[2] is a ITD candidate corresponding to a third largest correlation among the ITD candidates, ^^^[0] is a cross-correlation estimate magnitude value for ^^^^[0], ^^^[1] is a cross-correlation estimate magnitude value for ^^^^[1], ^^^[2] is a cross-correlation estimate magnitude value for ^^^^[2], ^^^^^ is a side gain parameter, SG_THR and SG_THR2
are thresholds, the method further comprising: if there are more than one ITD candidate identified and none of them are considered close to zero based on an ITD candidate distance to zero being above an ITD non-zero threshold and the level difference parameter is being used: responsive to ^^^^(^^^^[0]) ≠ ^^^^(^^^^[1]) and ^^^[0] and ^^^[1] do not differ more than a cross correlation threshold and at least one of: |^^^^^| > ^^_^^^ or a current ^^^^^ magnitude is different from zero and a sign of the current ^^^^^ is different than the sign of a previous frame ^^^^^ and ^^^(^^^^^ − ^^^^^^^^^ ) > ^^_^^^2 or the current ^^^^^ magnitude is different from zero and the sign of the current ^^^^^ is different from the ^^^^^ of a second previous frame and ^^^(^^^^^ − ^^^^2^^^^^ ) > ^^_^^^2, select an ITD to be ^^^^[0] if (^^^^^ < 0 and ^^^^[0] > 0) or (^^^^^ > 0 and ^^^^[0] < 0), otherwise select the ITD to be ^^^^[1]; and responsive to ^^^^(^^^^[0]) = ^^^^(^^^^[1]) and ^^^[0] and ^^^[1] differ more than the cross correlation threshold and |^^^^^| < ^^_^^^, the current ^^^^^ magnitude is < 0.025, and a sign of the current ^^^^^ is not different than the sign of a previous frame ^^^^^ and ^^^(^^^^^ − ^^^^^^^^^) < ^^_^^^2 and the sign of the current ^^^^^ is not different from the ^^^^^ of a second previous frame and ^^^(^^^^^ − ^^^^2^^^^^ ) < ^^_^^^2, determining if ^^^^(^^^^[0]) ≠ ^^^^(^^^^[2]) and ^^^[0] and ^^^[2] do not differ more than a cross correlation threshold and a magnitude of ^^^^^ is greater than a side gain threshold, select an ITD to be ^^^^[0] if (^^^^^ < 0 and ^^^^[0] > 0) or (^^^^^ > 0 and ^^^^[0] < 0), otherwise select an ITD to be ^^^^[2].
15. The method of any of Claims 8-14, further comprising: responsive to the consistency measure being below the consistency threshold, not using (1605) the level difference parameter to select among positive and negative ITD candidates.
16. The method of Claim 15, wherein ^^^^[0] is a ITD candidate corresponding to a largest correlation among the ITD candidates, the method further comprising: selecting ^^^^[0] to be the ITD if there are only two sources identified, and ^^^^[0] is different from an ITD of the previous frame, a peak importance magnitude is greater than an importance magnitude threshold, and peak relative importance is less than a relative magnitude threshold.
17. An encoder (502) comprising:
processing circuitry (2802); and memory (2810) coupled with the processing circuitry, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder (502) to perform operations comprising: obtaining an estimate of a threshold based on a shape of a cross-correlation function between the two waveforms; determining a second threshold based on a maximum value of the first threshold and a non-zero constant; determining at least one lag for which of cross-correlation estimate magnitude value is above the second threshold to form a list of potential peak positions; determining an indication of how many true peaks there are and a list of lags of the true peaks; responsive to there being three or more true peaks, keeping three lags having a higher cross-correlation estimate magnitude value than other true peaks and setting lags and cross-correlation estimate magnitude values of the other true peaks to zero to form a list of three candidate ITD estimates; responsive to there being less than three true peaks, keeping the lags and correlation of the true peaks and setting lags and cross-correlation estimate magnitude values of remaining positions to zero to form the list of three candidate ITD estimates; and estimating the ITD based on the list of three candidate ITD estimates and a consistency measure between estimated ITD values and a level difference parameter.
18. The encoder of Claim 17, wherein determining an indication of how many true peaks there are comprises: determining that a possible peak is a true peak when a cross-correlation estimate magnitude value of the possible peak is larger than neighbor positions of the possible peak; and removing possible peaks that are too close to each other by determining when lags between the possible peaks are less than a constant apart, keeping only a lag corresponding to a largest cross-correlation estimate magnitude value.
19. The encoder of any of Claims 18-17, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder to perform operations further comprising: determining whether or not to utilize level difference information for ITD estimation based on a consistency measure between estimated ITD values and the level difference
parameter.
20. The encoder of Claim 19, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder to perform operations further comprising determining the consistency measure.
21. The encoder of Claim 20, wherein determining the consistency measure comprises determining the consistency measure in accordance with
where ^^^_^^^_^^ is an updated consistency measure, ^^^_^^^_^^^^^^^^^ is the consistency measure being updated, ^^^^^ (^) is a side gain parameter, ^^^^[0] is a ITD candidate corresponding to the largest correlation among the ITD candidates, and ^^^^ is a filtering parameter which may be set to any value between 0 and 1 or be adaptively set based on the cross correlation value corresponding to ^^^^[0].
22. The encoder of any of Claims 20-21, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder to perform operations further comprising applying a forgetting factor to the consistency measure such that the consistency measure slowly approaches zero when there is no information to reliably update the consistency measure.
23. The encoder of any of Claims 20-21, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder to perform operations further comprising updating the consistency measure when there is only one true peak identified.
24. The encoder of any of Claims 19-23, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder to perform operations further comprising determining if the consistency measure is above a consistency threshold.
25. The encoder of Claim 24, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder to perform operations further comprising: responsive to the consistency measure being above the consistency threshold, using the level difference parameter to select among positive and negative ITD candidates.
26. The encoder of Claim 24, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder to perform operations further comprising: responsive to the consistency measure being above the consistency threshold and speech signals are identified or no music is identified, using the level difference parameter to select
among positive and negative ITD candidates.
27. The encoder of Claim 24, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder to perform operations further comprising: responsive to the consistency measure being above the consistency threshold and a current frame down mix energy estimate is an energy threshold above a previous frames long term energy estimate, using the level difference parameter to select among positive and negative ITD candidates.
28. The encoder of Claim 24, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder to perform operations further comprising: responsive to the consistency measure being above the consistency threshold, speech signals are identified or no music is identified, and a current frame down mix energy estimate is an energy threshold above a previous frames long term energy estimate, using the level difference parameter to select among positive and negative ITD candidates.
29. The encoder of any of Claims 19-29, wherein ^^^^[0] is a ITD candidate corresponding to a largest cross-correlation estimate magnitude value among the ITD candidates, ^^^^[1] is a ITD candidate corresponding to a second largest cross-correlation estimate magnitude value among the ITD candidates, ^^^^[2] is a ITD candidate corresponding to a third largest cross-correlation estimate magnitude value among the ITD candidates, ^^^[0] is a cross-correlation estimate magnitude value for ^^^^[0], ^^^[1] is a cross-correlation estimate magnitude value for ^^^^[1], ^^^[2] is a cross-correlation estimate magnitude value for ^^^^[2], and ^^^^^ is a side gain parameter, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder to perform operations further comprising: if there are more than one ITD candidate identified and none of them are considered close to zero based on an ITD candidate distance to zero being above an ITD non-zero threshold and the level difference parameter is being used: responsive to ^^^^(^^^^[0]) ≠ ^^^^(^^^^[1]) and ^^^[0] and ^^^[1] do not differ more than a cross correlation threshold, select an ITD to be ^^^^[0] if (^^^^^ < 0 and ^^^^[0] > 0) or (^^^^^ > 0 and ^^^^[0] < 0), otherwise select the ITD to be ^^^^[1]; and responsive to ^^^^(^^^^[0]) = ^^^^(^^^^[1]) or ^^^[0] and ^^^[1] differ more than the cross correlation threshold, determining if ^^^^(^^^^[0]) ≠ ^^^^(^^^^[2]) and ^^^[0] and ^^^[2] do not differ more than a cross correlation threshold and a magnitude
of ^^^^^ is greater than a side gain threshold, select an ITD to be ^^^^[0] if (^^^^^ < 0 and ^^^^[0] > 0) or (^^^^^ > 0 and ^^^^[0] < 0), otherwise select an ITD to be ^^^^[2].
30. The encoder of any of Claims 19-29, wherein ^^^^[0] is a ITD candidate corresponding to a largest cross-correlation estimate magnitude value among the ITD candidates, ^^^^[1] is a ITD candidate corresponding to a second largest cross-correlation estimate magnitude value among the ITD candidates, ^^^^[2] is a ITD candidate corresponding to a third largest correlation among the ITD candidates, ^^^[0] is a cross-correlation estimate magnitude value for ^^^^[0], ^^^[1] is a cross-correlation estimate magnitude value for ^^^^[1], ^^^[2] is a cross-correlation estimate magnitude value for ^^^^[2], ^^^^^ is a side gain parameter, SG_THR and SG_THR2 are thresholds, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder to perform operations further comprising: if there are more than one ITD candidate identified and none of them are considered close to zero based on an ITD candidate distance to zero being above an ITD non-zero threshold and the level difference parameter is being used: responsive to ^^^^(^^^^[0]) ≠ ^^^^(^^^^[1]) and ^^^[0] and ^^^[1] do not differ more than a cross correlation threshold and at least one of: |^^^^^ | > ^^_^^^ or a current ^^^^^ magnitude is different from zero and a sign of the current ^^^^^ is different than the sign of a previous frame ^^^^^ and ^^^(^^^^^ − ^^^^^^^^^ ) > ^^_^^^2 or the current ^^^^^ magnitude is different from zero and the sign of the current ^^^^^ is different from the ^^^^^ of a second previous frame and ^^^(^^^^^ − ^^^^2^^^^^ ) > ^^_^^^2, select an ITD to be ^^^^[0] if (^^^^^ < 0 and ^^^^[0] > 0) or (^^^^^ > 0 and ^^^^[0] < 0), otherwise select the ITD to be ^^^^[1]; and responsive to ^^^^(^^^^[0]) = ^^^^(^^^^[1]) and ^^^[0] and ^^^[1] differ more than the cross correlation threshold and |^^^^^ | < ^^_^^^, the current ^^^^^ magnitude is < 0.025, and a sign of the current ^^^^^ is not different than the sign of a previous frame ^^^^^ and ^^^(^^^^^ − ^^^^^^^^^ ) < ^^_^^^2 and the sign of the current ^^^^^ is not different from the ^^^^^ of a second previous frame and ^^^(^^^^^ − ^^^^2^^^^^ ) < ^^^^^^, determining if ^^^^(^^^^[0]) ≠ ^^^^(^^^^[2]) and ^^^[0] and ^^^[2] do not differ more than a cross correlation threshold and a magnitude of ^^^^^ is greater than a side gain threshold, select an ITD to be ^^^^[0] if (^^^^^ < 0 and ^^^^[0] > 0) or (^^^^^ > 0 and ^^^^[0] < 0), otherwise select an ITD to be ^^^^[2].
31. The encoder of any of Claims 24-30, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder to perform operations further comprising: responsive to the consistency measure being below the consistency threshold, not using the level difference parameter to select among positive and negative ITD candidates.
32. The encoder of Claim 31, wherein ^^^^[0] is a ITD candidate corresponding to a largest cross-correlation estimate magnitude value among the ITD candidates, wherein the memory includes instructions that when executed by the processing circuitry causes the encoder to perform operations further comprising: selecting ^^^^[0] to be the ITD if there are only two sources identified, and ^^^^[0] is different from an ITD of the previous frame, a peak importance magnitude is greater than an importance magnitude threshold, and peak relative importance is less than a relative magnitude threshold.
33. An encoder (502) adapted to perform the method according to at least one of Claims 1-16.
34. A computer program comprising program code to be executed by processing circuitry (2802) of an encoder (502), whereby execution of the program code causes the encoder (502) to perform operations according to any one of Claims 1-16.
35. A computer program product comprising a non-transitory storage medium including program code to be executed by processing circuitry (2802) of an encoder (502), whereby execution of the program code causes the encoder (502) to perform operations according to any one of Claims 1-16.
Applications Claiming Priority (4)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US202363442163P | 2023-01-31 | 2023-01-31 | |
| US202363447661P | 2023-02-23 | 2023-02-23 | |
| US202363455687P | 2023-03-30 | 2023-03-30 | |
| PCT/EP2024/052293 WO2024160859A1 (en) | 2023-01-31 | 2024-01-31 | Refined inter-channel time difference (itd) selection for multi-source stereo signals |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| EP4659244A1 true EP4659244A1 (en) | 2025-12-10 |
Family
ID=89843383
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| EP24703301.2A Pending EP4659244A1 (en) | 2023-01-31 | 2024-01-31 | Refined inter-channel time difference (itd) selection for multi-source stereo signals |
Country Status (3)
| Country | Link |
|---|---|
| EP (1) | EP4659244A1 (en) |
| CN (1) | CN121002568A (en) |
| WO (1) | WO2024160859A1 (en) |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2671221B1 (en) | 2011-02-03 | 2017-02-01 | Telefonaktiebolaget LM Ericsson (publ) | Determining the inter-channel time difference of a multi-channel audio signal |
| RU2704733C1 (en) | 2016-01-22 | 2019-10-30 | Фраунхофер-Гезелльшафт Цур Фердерунг Дер Ангевандтен Форшунг Е.Ф. | Device and method of encoding or decoding a multichannel signal using a broadband alignment parameter and a plurality of narrowband alignment parameters |
| JP6641027B2 (en) | 2016-03-09 | 2020-02-05 | テレフオンアクチーボラゲット エルエム エリクソン(パブル) | Method and apparatus for increasing the stability of an inter-channel time difference parameter |
-
2024
- 2024-01-31 CN CN202480021669.0A patent/CN121002568A/en active Pending
- 2024-01-31 EP EP24703301.2A patent/EP4659244A1/en active Pending
- 2024-01-31 WO PCT/EP2024/052293 patent/WO2024160859A1/en not_active Ceased
Also Published As
| Publication number | Publication date |
|---|---|
| CN121002568A (en) | 2025-11-21 |
| WO2024160859A1 (en) | 2024-08-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA3011741C (en) | Encoding of multiple audio signals | |
| JP6253776B2 (en) | Multi-channel audio decoder, multi-channel audio encoder, method and computer program using residual signal-based adjustment of the decorrelated signal contribution | |
| AU2018256414B2 (en) | Non-harmonic speech detection and bandwidth extension in a multi-source environment | |
| KR102709512B1 (en) | Decoding of multiple audio signals | |
| KR102724147B1 (en) | Inter-channel phase difference parameter modification | |
| US12266371B2 (en) | Multi-channel audio encoder, decoder, methods and computer program for switching between a parametric multi-channel operation and an individual channel operation | |
| JP2026016503A (en) | Method and apparatus for low-cost error recovery in predictive coding | |
| US20260088035A1 (en) | Adaptive inter-channel time difference estimation | |
| WO2024160859A1 (en) | Refined inter-channel time difference (itd) selection for multi-source stereo signals | |
| WO2024126467A1 (en) | Improved transitions in a multi-mode audio decoder | |
| EP4356373B1 (en) | Improved stability of inter-channel time difference (itd) estimator for coincident stereo capture | |
| US20260128050A1 (en) | Coherence calculation for stereo discontinuous transmission (dtx) | |
| AU2023385242A1 (en) | Adaptive encoding of transient audio signals | |
| AU2023355540A1 (en) | Coherence calculation for stereo discontinuous transmission (dtx) | |
| KR20250103678A (en) | Efficient time delay synthesis | |
| CN118414662A (en) | Adaptive Predictive Coding |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: UNKNOWN |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
| PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
| 17P | Request for examination filed |
Effective date: 20251001 |
|
| AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR |