Classifications

• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
  • G10L19/04—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using predictive techniques
  • G10L19/16—Vocoder architecture
  • G10L19/18—Vocoders using multiple modes
  • G10L19/20—Vocoders using multiple modes using sound class specific coding, hybrid encoders or object based coding
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
  • G10L19/008—Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04S—STEREOPHONIC SYSTEMS 
  • H04S3/00—Systems employing more than two channels, e.g. quadraphonic
  • H04S3/008—Systems employing more than two channels, e.g. quadraphonic in which the audio signals are in digital form, i.e. employing more than two discrete digital channels
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04S—STEREOPHONIC SYSTEMS 
  • H04S2420/00—Techniques used stereophonic systems covered by H04S but not provided for in its groups
  • H04S2420/11—Application of ambisonics in stereophonic audio systems

Definitions

• the invention relates to a method and to an apparatus for improving the coding of side information required for coding a Higher Order Ambisonics representation of a sound field.
• HOA Higher Order Ambisonics
• WFS wave field synthesis
• channel based approaches like the 22.2 multichannel audio format.
• the HOA representation offers the advantage of being independent of a specific loudspeaker set-up. This flexibility, however, is at the expense of a decoding pro ⁇ cess which is required for the playback of the HOA representation on a particular loudspeaker set-up.
• HOA signals may also be rendered to set- ups consisting of only few loudspeakers.
• a further advantage of HOA is that the same representation can also be employed without any modification for binaural rendering to headphones .
• HOA is based on the representation of the spatial density of complex harmonic plane wave amplitudes by a truncated Spher ⁇ ical Harmonics (SH) expansion.
• SH Spher ⁇ ical Harmonics
• the spatial resolution of the HOA representation improves with a growing maximum order N of the expansion.
• the total bit rate for the transmission of HOA representation given a desired single- channel sampling rate f $ and the number of bits per sam- pie, is determined by 0 ⁇ f s ⁇ .
• HOA sound field representations are proposed in WO 2013/171083 Al, EP 13305558.2 and PCT/EP2013/075559. These processings have in common that they perform a sound field analysis and decompose the given HOA representation into a directional component and a residual ambient compo ⁇ nent.
• the final compressed representation is as ⁇ sumed to consist of a number of quantised signals, resulting from the perceptual coding of the directional signals and relevant coefficient sequences of the ambient HOA component.
• a problem to be solved by the invention is to provide a more efficient way of coding side information related to that spatial prediction.
• the inventive method is suited for improving the coding of side information required for coding a Higher Order Ambisonics representation of a sound field, denoted HOA, with input time frames of HOA coefficient sequences, wherein dominant directional signals as well as a residual ambient HOA component are determined and a prediction is used for said dominant directional signals, thereby provid ⁇ ing, for a coded frame of HOA coefficients, side information data describing said prediction, and wherein said side information data can include:
• said method including the step:
• the inventive apparatus is suited for improving the coding of side information required for coding a Higher Order Ambisonics representation of a sound field, denoted HOA, with input time frames of HOA coefficient sequences, wherein dominant directional signals as well as a residual ambient HOA component are determined and a prediction is used for said dominant directional signals, thereby provid- ing, for a coded frame of HOA coefficients, side information data describing said prediction, and wherein said side information data can include:
• said apparatus including means which:
• Fig. 1 Exemplary coding of side information related to spa- tial prediction in the HOA compression processing described in EP 13305558.2;
• FIG. 2 Exemplary decoding of side information related to spatial prediction in the HOA decompression processing described in patent application EP 13305558.2; Fig. 3 HOA decomposition as described in patent application
• Fig. 4 Illustration of directions (depicted as crosses) of general plane waves representing the residual signal and the directions (depicted as circles) of dominant sound sources.
• the directions are presented in a three-dimensional coordinate system as sampling po ⁇ sitions on the unit sphere;
• Fig. 6 Inventive coding of spatial prediction side information
• Fig. 7 Inventive decoding of coded spatial prediction side information
• Fig. 1 it is illustrated how the coding of side information related to spatial prediction can be embedded into the HOA compression processing described patent application EP 13305558.2.
• a frame-wise processing with non-overlapping input frames C(/c) of HOA coeffi ⁇ cient sequences of length L is assumed, where k denotes the frame index.
• the first step or stage 11/12 in Fig. 1 is op ⁇ tional and consists of concatenating the non-overlapping k- th and ( k— 1) -th frames of HOA coefficient sequences C(/c) in ⁇ to a long frame C(/c) as
• C(fc): [C(fc-l) C ⁇ k) ⁇ , (1) which long frame is 50% overlapped with an adjacent long frame and which long frame is successively used for the es ⁇ timation of dominant sound source directions. Similar to the notation for C(/c), the tilde symbol is used in the following description for indicating that the respective quantity re ⁇ fers to long overlapping frames. If step/stage 11/12 is not present, the tilde symbol has no specific meaning.
• a parameter in bold means a set of values, e.g. a matrix or a vector.
• the long frame C(/c) is successively used in step or stage 13 for the estimation of dominant sound source directions as described in EP 13305558.2.
• This estimation provides a data set JDIR , ACT(k) ⁇ ⁇ 1, ... , D ⁇ of indices of the related directional signals that have been detected, as well as a data set
• step or stage 14 the current (long) frame C(/c) of HOA co- efficient sequences is decomposed (as proposed in EP 13305156.5) into a number of directional signals X mR (k— 2) belonging to the directions contained in the set Qa,Acr(k , and a residual ambient HOA component C AMB (k— 2).
• the delay of two frames is introduced as a result of overlap-add processing in order to obtain smooth signals. It is assumed that X mR (k— 2) is con ⁇ taining a total of D channels, of which however only those corresponding to the active directional signals are non ⁇ zero. The indices specifying these channels are assumed to be output in the data set mRACT (k— 2).
• the de- composition in step/stage 14 provides some parameters (/c— 2) which can be used at decompression side for predicting portions of the original HOA representation from the directional signals (see EP 13305156.5 for more details) .
• the HOA decomposition is described in more detail in the below section HOA decomposition .
• step or stage 15 the number of coefficients of the ambi ⁇ ent HOA component C AMB (k— 2) is reduced to contain only
• N mRACT (k— 2) indicates the cardinality of the data set mRACT (k— 2), i.e. the number of active di ⁇ rectional signals in frame k— 2. Since the ambient HOA com ⁇ ponent is assumed to be always represented by a minimum num ⁇ ber ORED of HOA coefficient sequences, this problem can be actually reduced to the selection of the remaining D— N mRACT (k— 2) HOA coefficient sequences out of the possible 0— 0 REO ones. In order to obtain a smooth reduced ambient HOA representa ⁇ tion, this choice is accomplished such that, compared to the choice taken at the previous frame k— 3, as few changes as possible will occur.
• step/stage 16 The final ambient HOA representation with the reduced number of 0 RED + N DIRiACT (/c— 2) non-zero coefficient sequences is de- noted by C AMB REO (k— 2) .
• the indices of the chosen ambient HOA coefficient sequences are output in the data set 2) .
• step/stage 16 the active directional signals contained in X mR (k— 2) and the HOA coefficient sequences contained in C AMBjRED (/c— 2) are assigned to the frame Y(k— 2) of / channels for individual perceptual encoding as described in EP 13305558.2.
• Perceptual coding step/stage 17 encodes the / channels of frame Y(k— 2) and outputs an encoded frame Y(k— 2) .
• the spa ⁇ tial prediction parameters or side information data (/c— 2) resulting from the decomposition of the HOA representation are losslessly coded in step or stage 19 in order to provide a coded data representation ⁇ 2), using the index set
• Fig. 2 it is exemplary shown how to embed in step or stage 25 the decoding of the received encoded side infor- mation data ⁇ 2) related to spatial prediction into the HOA decompression processing described in Fig. 3 of patent application EP 13305558.2.
• the decoding of the encoded side information data ⁇ 2) is carried out before entering its decoded version (/c— 2) into the composition of the HOA representation in step or stage 23, using the received index set mRACT (k) delayed by two frames in delay 24.
• step or stage 21 a perceptual decoding of the / signals contained in Y(k— 2) is performed in order to obtain the / decoded signals in Y(k— 2) .
• the perceptually decoded signals in Y(k— 2) are re-distributed in order to recreate the frame X mR (k— 2) of directional signals and the frame C AMB RED (k— 2) of the ambient HOA component.
• the infor ⁇ mation about how to re-distribute the signals is obtained by reproducing the assigning operation performed for the HOA compression, using the index data sets mR ACT (k) an d ⁇ AMB.ACT C ⁇ — 2) .
• composition step or stage 23 a current frame C(k— 3) of the desired total HOA representation is re-composed (accord ⁇ ing to the processing described in connection with Fig. 2b and Fig.
• C AMBJRED (/c— 2) corresponds to component D A (k— 2) in PCT/EP2013/ 075559
• Gn,A j (k A N CL ⁇ DIRACT C ⁇ ) correspond to A ⁇ (k) in PCT/ EP2013/075559, wherein active directional signal indices can be obtained by taking those indices of rows of A ⁇ k which contain valid elements.
• directional signals with re ⁇ spect to uniformly distributed directions are predicted from the directional signals X mR (k— 2) using the received parame ⁇ ters (/c— 2) for such prediction, and thereafter the current decompressed frame C(k— 3) is re-composed from the frame of directional signals X mR (k— 2) , from mR ACT (k) and OO r and from the predicted portions and the reduced ambient HOA compo- nent CAMB.RED (fc - 2) .
• the smoothed dominant directional signals X mR (k— 1) and their HOA representation C DIR (/c— 1) are computed in step or stage 31, using the long frame C(/c) of the input HOA rep- resentation, the set of directions and the set 3 ⁇ 4IR,ACT of corresponding indices of directional signals. It is as ⁇ sumed that X mR (k— 1) contains a total of D channels, of which however only those corresponding to the active directional signals are non-zero. The indices specifying these channels are assumed to be output in the set mR ACT (k— 1) .
• step or stage 33 the residual between the original HOA representation C(k— 1) and the HOA representation C DIR (/c— 1) of the dominant directional signals is represented by a num ⁇ ber of 0 directional signals -X RES (/C— 1), which can be consid- ered as being general plane waves from uniformly distributed directions, which are referred to a uniform grid.
• step or stage 34 these directional signals are predicted from the dominant directional signals X mR (k— 1) in order to provide the predicted signals -X RES (/C— 1) together with the respective prediction parameters (/c— 1) .
• the dominant directional signals Xum ,d (k— 1) with indices d which are contained in the set ⁇ DIRACT C ⁇ — Or are consid ⁇ ered. The prediction is described in more detail in the be ⁇ low section Spatial prediction.
• step or stage 35 the smoothed HOA representation C RES (/c— 2) of the predicted directional signals X RE$ (k— 1) is computed.
• step or stage 37 the residual C AMB (/c— 2) between the orig- inal HOA representation C(k— 2) and the HOA representation C DIR (/c— 2) of the dominant directional signals together with the HOA representation C RES (/c— 2) of the predicted directional signals from uniformly distributed directions is computed and is output.
• the required signal delays in the Fig. 3 processing are per ⁇ formed by corresponding delays 381 to 387.
• the goal of the spatial prediction is to predict the 0 re ⁇ sidual signals
• X mR (k - 1): [X OlR (k - 3) X OlR (k - 2) X OlR (k - 1)] (3)
• Dp ED t 9 1, denote the indices from which di ⁇ rectional signals the prediction for the direction q has to be performed. If no prediction is to be performed for a direction q , the corresponding column of the matrix ⁇ IND C ⁇ - 1) consists of zeros. Further, if less than D PRED directional signals are used for the prediction for a di ⁇ rection q , the non-required elements in the 9-th column of P IND (/c— 1) are also zero.
• P TYPE O -1) [1 0 0 0 0 0 2 0 0 0 0 0 0 0 0] , (7) r j , _ ⁇ ⁇ - ⁇ 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 01
• the general plane wave sig ⁇ nal 3 ⁇ 4ES,GRID,I — 1) from direction ⁇ is predicted from the di- rectional signal 3 ⁇ 4IR,I( ⁇ — ⁇ from direction -QACT.I by a pure multiplication (i.e. full band) with a factor that results from de-quantising the value 40.
• the general plane wave signal 3 ⁇ 4ES,GRID,7 — 1) from direction ⁇ 7 is predicted from the directional signals x DIR1 (/c— 1) and 3 ⁇ 4iR,4(k — 1) by a lowpass filtering and multiplication with factors that result from de-quantising the values 15 and —13.
• B sc denotes a predefined number of bits to be used for the quantisation of the prediction factors.
• PF ,d, q.k— 1) is assumed to be set to zero, if iND,d,q(k — 1) is equal to zero.
• a bit array ActivePred consisting of 0 bits is creat- ed, in which the bit ActivePred [q>] indicates whether or not for the direction q a prediction is performed.
• the number of 'ones' in this array is denoted by NumActivePred .
• the bit array PredType of length NumActivePred is creat- ed where each bit indicates, for the directions where a pre ⁇ diction is to be performed, the kind of the prediction, i.e. full band or low pass.
• the unsigned inte ⁇ ger array PredDirSiglds of length NumActivePred ⁇ D PRED is created, whose elements denote for each active prediction the
• OpRED indices of the directional signals to be used If less than DpRED directional signals are to be used for the predic ⁇ tion, the indices are assumed to be set to zero.
• Each ele ⁇ ment of the array PredDirSiglds is assumed to be represented by [log 2 (D + 1)1 bits. The number of non-zero elements in the array PredDirSiglds is denoted by NumNonZerolds .
• the integer array QuantPredGains of length NumNonZerolds is created, whose elements are assumed to represent the quantised scaling factors f Q ,F ,d,q (k — 1) to be used in equation (17) .
• the dequantisation to obtain the corresponding dequan- tised scaling factors P Fdq (k— l) is given in equation (10).
• Each element of the array QuantPredGains is assumed to be represented by B sc bits.
• the coded repre ⁇ sentation of equations (7) to (9) is used:
• QuantPredGains [40 15 -13] . (23)
• the state- of-the-art processing is advantageously modified.
• PSPredictionActive is zero (or '1' as an alternative), the array ActivePred and further data related to the prediction are not to be included into the coded side information ⁇ COD ⁇ I R practise, this operation reduces over time the average bit rate for the transmission of ⁇ COD ⁇
• NumActivePred of active prediction is often very low. In such situation, instead of using the bit array ActivePred for indicating for each direction q whether or not the prediction is performed, it can be more efficient to transmit or transfer instead the number of active predic ⁇ tions and the respective indices. In particular, this modified kind of coding the activity is more efficient in case that NumActivePred ⁇ M M , (24) where M M is the greatest integer number that satisfies
• Equation (25) [log 2 (M M )l denotes the number of bits re ⁇ quired for coding the actual number NumActivePred of active predictions, and M M ⁇ [log 2 (0)l is the number of bits re ⁇ quired for coding the respective direction indices.
• the right hand side of equation (25) corresponds to the num ⁇ ber of bits of the array ActivePred , which would be re ⁇ quired for coding the same information in the known way.
• a single bit KindOfCodedPredlds can be used for indicating in which way the indices of those directions, where a prediction is supposed to be performed, are coded. If the bit
• KindOfCodedPredlds has the value '1' (or '0' in the alterna- tive) , the number NumActivePred and the array Predlds containing the indices of directions, where a prediction is supposed to be performed, are added to the coded side in ⁇ formation ⁇ COD ⁇ Otherwise, if the bit KindOfCodedPredlds has the value '0' (or '1' in the alternative), the array
• ActivePred is used to code the same information.
• bits can be used for coding each element of the index ar ⁇ ray PredDirSiglds , which kind of coding is more efficient.
• the data set 3 ⁇ 4IR,ACT is assumed to be known, and thus the decoder also knows how many bits have to be read for decoding an index of a directional signal. Note that the frame indices of ⁇ COD to be computed and the used index data set 3 ⁇ 4IR,ACT have to be identical.
• PredGains which however contains quantised values.
• this representation coded according to the invention requires 8 bits less.
• the decoding of the modified side information related to spatial prediction is summarised in the example decoding processing depicted in Fig. 7 and Fig. 8 (the processing depicted in Fig. 8 is the continuation of the processing depicted in Fig. 7 ) and is explained in the following.
• NumActivePred elements is read, where each element is assumed to be coded by [log 2 (0)l bits.
• the elements of this array are the indices of directions, where a prediction has to be per ⁇ formed.
• the bit array PredType of length is read, where each element is assumed to be coded by [log 2 (0)l bits.
• the elements of this array are the indices of directions, where a prediction has to be per ⁇ formed.
• NumActivePred is read, of which the elements indicate the kind of prediction to be performed for each one of the relevant directions.
• the elements of the vector PTYPE are computed.
• the array PredDirSiglds is read, which con ⁇ sists of NumActivePred ⁇ D PRED elements. Each element is assumed to be coded by log 2 (OACT)l bits. Using the information con ⁇ tained in PTYPE 3 ⁇ 4IR , ACT an d PredDirSiglds , the elements of ma ⁇ trix PJ D are set and the number NumNonZeroIds of non-zero el- ements in P IND is computed.
• the array QuantPredGains is read, which consists of NumNonZeroIds elements, each coded by B SC bits. Using the information contained in and QuantPredGains , the elements of the matrix P QF are set.
• inventive processing can be carried out by a single pro ⁇ cessor or electronic circuit, or by several processors or electronic circuits operating in parallel and/or operating on different parts of the inventive processing.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Multimedia (AREA)
• Acoustics & Sound (AREA)
• Signal Processing (AREA)
• Computational Linguistics (AREA)
• Health & Medical Sciences (AREA)
• Audiology, Speech & Language Pathology (AREA)
• Human Computer Interaction (AREA)
• Mathematical Physics (AREA)
• Compression, Expansion, Code Conversion, And Decoders (AREA)
• Stereophonic System (AREA)
• Circuit For Audible Band Transducer (AREA)

Priority Applications (25)

Applications Claiming Priority (4)

Related Child Applications (2)

Publications (1)

Family

ID=52134201

Family Applications (1)

Country Status (6)

Families Citing this family (2)

Citations (3)

Family Cites Families (15)

• 2014
  • 2014-12-19 WO PCT/EP2014/078641 patent/WO2015104166A1/en not_active Ceased
  • 2014-12-19 EP EP19208682.5A patent/EP3648102B1/en active Active
  • 2014-12-19 CN CN201480072725.XA patent/CN105981100B/zh active Active
  • 2014-12-19 EP EP22176389.9A patent/EP4089675B1/en active Active
  • 2014-12-19 CN CN202010019977.3A patent/CN111179955B/zh active Active
  • 2014-12-19 CN CN202010020047.XA patent/CN111028849B/zh active Active
  • 2014-12-19 KR KR1020167021560A patent/KR102338374B1/ko active Active
  • 2014-12-19 CN CN202410171734.XA patent/CN118016077A/zh active Pending
  • 2014-12-19 CN CN202010019997.0A patent/CN111182443B/zh active Active
  • 2014-12-19 KR KR1020227019915A patent/KR102686291B1/ko active Active
  • 2014-12-19 CN CN202410341175.2A patent/CN118248156A/zh active Pending
  • 2014-12-19 KR KR1020217040165A patent/KR102409796B1/ko active Active
  • 2014-12-19 US US15/110,354 patent/US9990934B2/en active Active
  • 2014-12-19 KR KR1020247023646A patent/KR20240116835A/ko active Pending
  • 2014-12-19 CN CN202010025266.7A patent/CN111179951B/zh active Active
  • 2014-12-19 EP EP25158678.0A patent/EP4554255A3/en active Pending
  • 2014-12-19 EP EP14815731.6A patent/EP3092641B1/en active Active
  • 2014-12-19 JP JP2016544628A patent/JP6530412B2/ja active Active
• 2018
  • 2018-04-18 US US15/956,295 patent/US10147437B2/en active Active
  • 2018-11-13 US US16/189,797 patent/US10424312B2/en active Active
• 2019
  • 2019-05-16 JP JP2019092768A patent/JP6848004B2/ja active Active
  • 2019-08-05 US US16/532,302 patent/US10553233B2/en active Active
  • 2019-12-18 US US16/719,806 patent/US10714112B2/en active Active
• 2020
  • 2020-07-10 US US16/925,334 patent/US11211078B2/en active Active
• 2021
  • 2021-03-03 JP JP2021033172A patent/JP7258063B2/ja active Active
  • 2021-12-21 US US17/558,550 patent/US11488614B2/en active Active
• 2022
  • 2022-10-20 US US17/970,118 patent/US11869523B2/en active Active
• 2023
  • 2023-04-04 JP JP2023061042A patent/JP7589876B2/ja active Active
  • 2023-12-20 US US18/390,546 patent/US12277948B2/en active Active
• 2024
  • 2024-10-29 JP JP2024189442A patent/JP2025003689A/ja active Pending
• 2025
  • 2025-04-10 US US19/176,090 patent/US20250329338A1/en active Pending

Patent Citations (3)

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events