EP3329485B1 - System and method for spatial processing of soundfield signals - Google Patents
System and method for spatial processing of soundfield signals Download PDFInfo
- Publication number
- EP3329485B1 EP3329485B1 EP16747709.0A EP16747709A EP3329485B1 EP 3329485 B1 EP3329485 B1 EP 3329485B1 EP 16747709 A EP16747709 A EP 16747709A EP 3329485 B1 EP3329485 B1 EP 3329485B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- soundfield
- signal
- arrival
- input
- delayed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims description 52
- 238000012545 processing Methods 0.000 title claims description 7
- 230000009466 transformation Effects 0.000 claims description 37
- 239000011159 matrix material Substances 0.000 claims description 31
- 230000003111 delayed effect Effects 0.000 claims description 23
- 238000001914 filtration Methods 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims 1
- 230000001934 delay Effects 0.000 claims 1
- 239000000203 mixture Substances 0.000 description 10
- 238000004091 panning Methods 0.000 description 9
- 238000002592 echocardiography Methods 0.000 description 8
- 230000005236 sound signal Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000004088 simulation Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000002238 attenuated effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S3/00—Systems employing more than two channels, e.g. quadraphonic
- H04S3/02—Systems employing more than two channels, e.g. quadraphonic of the matrix type, i.e. in which input signals are combined algebraically, e.g. after having been phase shifted with respect to each other
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10K—SOUND-PRODUCING DEVICES; METHODS OR DEVICES FOR PROTECTING AGAINST, OR FOR DAMPING, NOISE OR OTHER ACOUSTIC WAVES IN GENERAL; ACOUSTICS NOT OTHERWISE PROVIDED FOR
- G10K15/00—Acoustics not otherwise provided for
- G10K15/08—Arrangements for producing a reverberation or echo sound
- G10K15/12—Arrangements for producing a reverberation or echo sound using electronic time-delay networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S2420/00—Techniques used stereophonic systems covered by H04S but not provided for in its groups
- H04S2420/11—Application of ambisonics in stereophonic audio systems
Definitions
- the present invention provides for systems and methods for the input of an audio soundfield signal and the creation of a reverberant acoustic equivalent soundfield signal.
- Multi-channel audio signals are used to store or transport a listening experience, for an end listener, that may include the impression of a very complex acoustic scene.
- the multi-channel signals may carry the information that describes the acoustic scene using a number of common conventions including, but not limited to, the following:
- D1 describes a reverberation architecture implemented within the signal chain of a periphonic HOA audio stream.
- a HOA signal representing the dry source signal is decoded into an array of virtual sources uniformly distributed within the reverberant space being simulated. These virtual sources are convolved with independent, decorrelated impulse responses. The output of each convolver is then encoded back into HOA and mixed with the dry HOA signal.
- By driving a set of statistically independent directional impulse responses with signals derived from the HOA signal the directionality of the dry signal is preserved in the reverberated soundfield.
- the acoustic transformation process utilises a multi-channel matrix mixer.
- the multi-channel matrix mixer can be formed by combining one or more spatial operations, including a spatial rotation operation.
- the multi-channel matrix mixer can be formed by combining one or more spatial operations, including a spatial mirror operation.
- the multi-channel matrix mixer can be formed by combining one or more spatial operations, including a directional gain operation.
- the multi-channel matrix mixer can be formed by combining one or more spatial operations, including a directional permutation operation.
- the acoustic transformation process preferably can include frequency-dependant filtering.
- each simulated echo can comprise a delayed and rotated copy of the input sound field signal. In some embodiments, each simulated echo preferably can include substantially the same delay. In some embodiments, the alternative direction of arrival can comprise a geometric transformation of the first direction of arrival.
- the acoustic transformation units can include: a multi channel matrix multiplier for applying a geometric transformation to an output tap to produce a geometric transformed output; and a series of linear audio filters applied to each channel of the geometric transformed output.
- the preferred embodiments provide for a system and method which, given that an input soundfield signal contains audio components that are encoded with different directions of arrival, produces an output soundfield signal that will contain simulated echoes, such that each simulated echo will have a direction of arrival that is a function of the direction of arrival of the original audio component as it appeared in the input signal.
- the output soundfield signal thereby provides for reverberance and other simulated audio effects.
- N -channel Soundfield Format is often defined by it's panning function, P N ( ⁇ ).
- G P N ( ⁇ )
- G is an [ N ⁇ 1] column vector of gain values
- a set of M objects (represented by the M audio signals o 1 ( t ), o 2 ( t ) , ... , o M ( t ) ) can be encoded into a N -channel Spatial Format signal X N ( t ) as per Equation 2 below (where object m is "located" at the position defined by ⁇ m ):
- X N t x 1 t x 2 t ⁇ x N t
- the signal X N ( t ) can be referred to as an Anechoic Mixture of the audio objects.
- any sound emitted by the audio object will reach the listener via multiple paths.
- This phenomenon is well known in the art, and the resulting sound, received at the listening position, is said to be reverberant.
- the number of acoustic paths, formed by the propagation of sound from the object and reflected off acoustic surfaces to reach the listener, may be infinite, but a reasonably close estimate of the sound received at the listening position may be formed by considering a finite number ( E ) of echoes.
- Fig. 2 illustrates an example of reverberance, where the sound from audio object m , 20, is received at the listening position from direction ⁇ m , along with one echo (echo e ) being received at the listening position from direction ⁇ ' m , e .
- Equation 2 shows how an N -channel soundfield signal, X N ( t ) , may be created by combining M audio objects, based on the assumption that each audio object has a location ( ⁇ m ) and an audio signal ( o m ( t )).
- R N ( t ) X N ( t ) + Y N ( t ) , intended to contain all of the M audio objects, combined together in a way that includes a simulation of an acoustic space (by including E echoes for each object).
- the signal Y N ( t ) can be referred to as the Reverberant Mixture of the audio objects.
- the complete acoustic-simulation is created by summing together the Anechoic Mixture, X N ( t ), and the Reverberant Mixture, Y N ( t ).
- Equation 10 the terminology [ o m ⁇ h m,e ]( t ) is used to indicate the convolution of the object audio signal o m ( t ) with the impulse response h m,e ( t ) , and hence o m ⁇ h m , e t ⁇ d m , e F s indicates the convolved signal with an additional delay of d m,e samples (where F s is the sample frequency).
- the N -channel soundfield signal format is defined by the panning function, P ( ⁇ ).
- Equation 14 tells us that, if we wish to apply a 3 ⁇ 3 matrix transformation, A , to the ( x , y , z ) coordinates of an object location, prior to the computation of the panning function, we can instead achieve this transformation as a 4 ⁇ 4 matrix operation, applied to the panning-gain vector, after the computation of the panning function.
- Equation 14 can be applied to Equation 2, in order to manipulate the location of all objects in audio scene, as per Equation 17 below.
- a transformed soundfield signal, X N ′ t is created from X N ( t ), achieving the same result that would have occured if all of the objects had their (x, y, z) locations modified by the 3 ⁇ 3 matrix A .
- the locations of all objects within a soundfield can be rotated around the listening position.
- the manipulation of the ( x, y, z ) coordinates of each object may be defined in terms of a 3 ⁇ 3 matrix, A , and the manipulation of the 4-channel soundfield signal may be carried out according to Equation 17.
- the locations of all objects within a soundfield may be mirrored about a plane that passes through the listening position.
- the manipulation of the ( x , y, z ) coordinates of each object may be defined in terms of a 3 ⁇ 3 matrix, A , and the manipulation of the 4-channel soundfield signal may be carried out according to Equation 17.
- Dominance A transformation of the 4-channel soundfield signal (known as the Lorentz transformation) may be applied by multiplying the 4 channels of the signal by the following 4 ⁇ 4 matrix:
- Dominance X ⁇ 1 2 ⁇ + ⁇ ⁇ 1 1 2 2 ⁇ ⁇ ⁇ ⁇ 1 0 0 1 2 ⁇ ⁇ ⁇ ⁇ 1 1 2 ⁇ + ⁇ ⁇ 1 0 0 0 0 1 0 0 0 0 1
- a unique Shared Echo Model is utilised, whereby all objects share the same time-delay pattern of echoes.
- Equation 10 In order to use the Anechoic Mixture, X N ( t ), as the starting point for creating the Reverberant Mixture, Y N ( t ) , it is desirable to apply some modified rules for the behaviour of the reverberation function as shown in Equation 10. In one embodiment of the invention, the following simplifications may be made:
- Echo Time Simplification It will be recalled that the original reverberation calculation (as per Equation 10) treats the reverberation for each object as a series of echoes, wherein for object m , echo e, has a time delay (relative to the direct-path) equal to d m , e (so, the echo times are different for each object).
- a delay d' k is defined to be the arrival time (relative to the direct sound) of echo k, and this delay is the same for every object (and hence, the echo delay, d' k , is no longer dependant on the object identifier, m ).
- Echo Direction Simplification The original reverberation calculation (as per Equation 10) treats the reverberation for each object as a series of echoes, wherein for object m , echo e has a direction of arrival, ⁇ m , e ′ (so, the echo arrival directions are different for each object).
- the processing chain 100 includes a Delay Line, 3, with K taps (and, in the following explanation, the variable k can be used to refer to a specific tap number, so that k ⁇ ⁇ 1,2,..., K ⁇ ).
- the input, 2, to the Delay Line 3 is the N -channel input signal, X N ( t ) .
- an N -channel delayed signal e.g. 5 is taken from the Delay Line, and processed via an acoustic transformation process, 200, to produce an acoustically transformed delayed signal, 6.
- the set of K acoustically transformed delayed signals are added together 7 to produce the output soundfield signal, 8.
- Fig. 3 illustrates one example form of implementation of an Echo Processor 200 which applies an acoustic transformation process.
- the input N -channel delayed signal 5 is processed, to produce the N -channel acoustically transformed delayed signal 6.
- two operations are performed by the acoustic transformation process, a multi-channel matrix mixer (represented by the N ⁇ N matrix R k ) 11, and a linear time-invariant filter, H k ( z ) e.g. 12, applied to each of the N channels of the soundfield signal.
- the intention of the acoustic transformation process is to create a simulation of the k th acoustic echo according to the following operating principles:
- Echo Delay The time delay of echo k is defined by use of the Delay Line so that input to the Delay Line 2 (of Fig. 2 ), is delayed by d' k samples to give the input 5, to the k th acoustic transformation process (referring to Figure 2 ).
- Echo Amplitude and Frequency Response The amplitude and frequency response of echo k are provided by the filter, H k (z) e.g. 12, applied to each of the N channels as per Fig. 3 .
- Equation 20 defines the 4 ⁇ 4 matrix, BtoA that is the inverse of AtoB.
- a processing train for implementing the method of Equation 25 is also shown in Fig. 4 , with the matrix processing Bk and Ck being separately implemented 21, 23.
- an acoustic transformation process can be implemented as a 4x4 matrix of arbitrary filter operations 200.
- the methods described above may also be combined with alternative reverberation processes, which may be known in the art, to produce a reverberant mixture that contains some echoes generated according to the above described methods, along with additional echoes and reverberation that are generated by the alternative methods.
- any one of the terms comprising, comprised of or which comprises is an open term that means including at least the elements/features that follow, but not excluding others.
- the term comprising, when used in the claims should not be interpreted as being limitative to the means or elements or steps listed thereafter.
- the scope of the expression a device comprising A and B should not be limited to devices consisting only of elements A and B.
- Any one of the terms including or which includes or that includes as used herein is also an open term that also means including at least the elements/features that follow the term, but not excluding others. Thus, including is synonymous with and means comprising.
- exemplary is used in the sense of providing examples, as opposed to indicating quality. That is, an "exemplary embodiment” is an embodiment provided as an example, as opposed to necessarily being an embodiment of exemplary quality.
- an element described herein of an apparatus embodiment is an example of a means for carrying out the function performed by the element for the purpose of carrying out the invention.
- Coupled when used in the claims, should not be interpreted as being limited to direct connections only.
- the terms “coupled” and “connected,” along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other.
- the scope of the expression a device A coupled to a device B should not be limited to devices or systems wherein an output of device A is directly connected to an input of device B. It means that there exists a path between an output of A and an input of B which may be a path including other devices or means.
- Coupled may mean that two or more elements are either in direct physical or electrical contact, or that two or more elements are not in direct contact with each other but yet still co-operate or interact with each other.
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Acoustics & Sound (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Mathematical Physics (AREA)
- Pure & Applied Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Algebra (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Stereophonic System (AREA)
Description
- This application claims priority to
US Provisional Patent Application No. 62/198,440, filed July 29, 2015 15185913.9, filed September 18, 2015 - The present invention provides for systems and methods for the input of an audio soundfield signal and the creation of a reverberant acoustic equivalent soundfield signal.
- Any discussion of the background art throughout the specification should in no way be considered as an admission that such art is widely known or forms part of common general knowledge in the field.
- Multi-channel audio signals are used to store or transport a listening experience, for an end listener, that may include the impression of a very complex acoustic scene. The multi-channel signals may carry the information that describes the acoustic scene using a number of common conventions including, but not limited to, the following:
- Discrete Speaker Channels: The audio scene may have been rendered in some way, to form speaker channels which, when played back on the appropriate arrangement of loudspeakers, create the illusion of the desired acoustic scene. Examples of Discrete Speaker Formats include stereo, 5.1 or 7.1 speaker signals, as used in many sound formats today.
- Audio Objects: The audio scene may be represented as one or more object audio channels which, when rendered by the listener's playback equipment, can re-create the acoustic scene. In some cases, each audio object will be accompanied by metadata (implicit or explicit) that is used by the renderer to pan the obj ect to the appropriate "location" in the listener's playback environment. Examples of Audio Object Formats include Dolby Atmos (Trade Mark), which is used in the carriage of rich sound-tracks on Blu-Ray Disc and other motion picture delivery formats.
- Soundfield Channels: The audio scene may be represented by a Soundfield Format - a set of two or more audio signals that collectively contain one or more audio objects with the spatial location of each object "encoded" in the Spatial Format in the form of panning gains. Examples of Soundfield Formats include Ambisonics, and Higher Order Ambisonics (both of which are well known in the art). Example systems are described in Gerzon, M.A., Periphony: With-Height Sound Reproduction. J. Audio Eng. Soc., 1973. 21(1): p. 2-10, and 3D Sound Field Recording with Higher Order Ambisonics-Objective Measurements and Validation of Spherical Microphone S Bertet, J Daniel, S Moreau - Audio Engineering Society Convention 120, 2006
- The international search report cites the article "An Architecture for Reverberation in High Order Ambisonics", LOPEZ-LEZCANO ET AL, hereinafter "D1". D1 describes a reverberation architecture implemented within the signal chain of a periphonic HOA audio stream. A HOA signal representing the dry source signal is decoded into an array of virtual sources uniformly distributed within the reverberant space being simulated. These virtual sources are convolved with independent, decorrelated impulse responses. The output of each convolver is then encoded back into HOA and mixed with the dry HOA signal. By driving a set of statistically independent directional impulse responses with signals derived from the HOA signal, the directionality of the dry signal is preserved in the reverberated soundfield.
- It is an object of the invention, in its preferred form to provide for the modification of multi channel audio signals that adhere to various Soundfield formats for the creation of reverberant soundfield signals.
- In accordance with a first aspect of the present invention, there is provided a method for creating an output soundfield signal from an input soundfield signal, as according to
claim 1. - Preferably, the acoustic transformation process utilises a multi-channel matrix mixer. The multi-channel matrix mixer can be formed by combining one or more spatial operations, including a spatial rotation operation. The multi-channel matrix mixer can be formed by combining one or more spatial operations, including a spatial mirror operation. The multi-channel matrix mixer can be formed by combining one or more spatial operations, including a directional gain operation. In some embodiments, the multi-channel matrix mixer can be formed by combining one or more spatial operations, including a directional permutation operation. The acoustic transformation process preferably can include frequency-dependant filtering.
- In accordance with a further aspect of the present invention, there is provided a method for adding simulated reverberance to an input sound field signal, as according to
claim 8. - In some embodiments, each simulated echo can comprise a delayed and rotated copy of the input sound field signal. In some embodiments, each simulated echo preferably can include substantially the same delay. In some embodiments, the alternative direction of arrival can comprise a geometric transformation of the first direction of arrival.
- In accordance with a further aspect of the present invention, there is provided a system for processing of soundfield signals to simulate the presence of reverberance, as according to
claim 11. - In some embodiments, the acoustic transformation units can include: a multi channel matrix multiplier for applying a geometric transformation to an output tap to produce a geometric transformed output; and a series of linear audio filters applied to each channel of the geometric transformed output.
- Embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings in which:
-
Fig. 1 illustrates schematically an audio object, at direction φ m, and an echo at direction -
Fig. 2 is a schematic block diagram of a tapped delay line. -
Fig. 3 is a schematic block diagram of an echo processor. -
Fig. 4 is a schematic block diagram of an echo processor with direction-dependant filtering; and -
Fig. 5 illustrates an alternative form of an echo processor. - The preferred embodiments provide for a system and method which, given that an input soundfield signal contains audio components that are encoded with different directions of arrival, produces an output soundfield signal that will contain simulated echoes, such that each simulated echo will have a direction of arrival that is a function of the direction of arrival of the original audio component as it appeared in the input signal. The output soundfield signal thereby provides for reverberance and other simulated audio effects.
-
-
- The signal XN (t) can be referred to as an Anechoic Mixture of the audio objects. The symbol φm is used to denote the abstract concept of "the location of object m ". In some cases, this symbol may be used to indicate the 3-vector: φm = (xm, ym, zm ), indicating that the object is located at a specific point in 3D space. In other cases, a restriction can be added that φm corresponds to a unit-vector, so that
- When an audio object and a listener are both located within the boundaries of an acoustic space (defined by a set of acoustically reflective surfaces), any sound emitted by the audio object will reach the listener via multiple paths. This phenomenon is well known in the art, and the resulting sound, received at the listening position, is said to be reverberant. The number of acoustic paths, formed by the propagation of sound from the object and reflected off acoustic surfaces to reach the listener, may be infinite, but a reasonably close estimate of the sound received at the listening position may be formed by considering a finite number (E) of echoes.
-
Fig. 2 illustrates an example of reverberance, where the sound from audio object m, 20, is received at the listening position from direction φm , along with one echo (echo e) being received at the listening position from direction φ' m,e . - In order to express this mathematically, the following variables can be defined:
- e : echonumber 1 ≤ e ≤ E (4)
- φm : the direction of arrival of sound from object m (5)
-
- dm,e: the delay (in samples) of echo e from object m (7)
- h m,e (t): the impulse response of echo e from object m (8)
-
Equation 2 shows how an N -channel soundfield signal, XN (t), may be created by combining M audio objects, based on the assumption that each audio object has a location (φm ) and an audio signal (om (t)). - It is possible to devise a more complex acoustic soundfield signal, RN (t) = XN (t) + YN (t), intended to contain all of the M audio objects, combined together in a way that includes a simulation of an acoustic space (by including E echoes for each object). This is shown in Equation 10 below:
- The signal YN (t) can be referred to as the Reverberant Mixture of the audio objects. The complete acoustic-simulation is created by summing together the Anechoic Mixture, XN (t), and the Reverberant Mixture, YN (t).
-
-
-
-
- Equation 14 tells us that, if we wish to apply a 3 × 3 matrix transformation, A, to the (x, y, z) coordinates of an object location, prior to the computation of the panning function, we can instead achieve this transformation as a 4 × 4 matrix operation, applied to the panning-gain vector, after the computation of the panning function.
- The result shown in in Equation 14 can be applied to
Equation 2, in order to manipulate the location of all objects in audio scene, as per Equation 17 below. In this case, a transformed soundfield signal, - It is known in the art, that certain manipulations of the objects within an N -channel soundfield can be achieved by applying a N × N matrix to the N channels of the soundfield signal. In the example given here, whereby the soundfield panning-function is the known Ambisonic panning function, the available manipulations of the soundfield include:
- Rotation: The locations of all objects within a soundfield can be rotated around the listening position. The manipulation of the (x, y, z) coordinates of each object may be defined in terms of a 3 × 3 matrix, A, and the manipulation of the 4-channel soundfield signal may be carried out according to Equation 17.
- Mirroring: The locations of all objects within a soundfield may be mirrored about a plane that passes through the listening position. The manipulation of the (x, y, z) coordinates of each object may be defined in terms of a 3 × 3 matrix, A, and the manipulation of the 4-channel soundfield signal may be carried out according to Equation 17.
-
- The result of this transformation is to boost the gain of the audio objects located at φ = (1,0,0) by λ. Audio objects located at φ = (-1,0,0) will be attenuated by λ -1.
- All Rotation and Mirroring operations are defined in terms of 3 × 3 unitary matrices (so that A × AT = I 3×3). If det(A) = 1, the matrix A corresponds to a rotation in 3D space, and if det(A) = -1, the matrix A corresponds to a mirroring operation in 3D space. In many of the embodiments described below, it will be convenient to assume that A is unitary.
- The above manipulations of Ambisonic soundfield signal are known in the Art.
- It is one intention of the preferred embodiments to create a Reverberant Mixture, YN (t), of the audio objects from the Anechoic Mixture, XN (t). In the preferred embodiments, a unique Shared Echo Model is utilised, whereby all objects share the same time-delay pattern of echoes.
- In order to use the Anechoic Mixture, XN (t), as the starting point for creating the Reverberant Mixture, YN (t), it is desirable to apply some modified rules for the behaviour of the reverberation function as shown in Equation 10. In one embodiment of the invention, the following simplifications may be made:
- Echo Time Simplification: It will be recalled that the original reverberation calculation (as per Equation 10) treats the reverberation for each object as a series of echoes, wherein for object m, echo e, has a time delay (relative to the direct-path) equal to d m,e (so, the echo times are different for each object). For the new Shared Echo Model, a delay d'k is defined to be the arrival time (relative to the direct sound) of echo k, and this delay is the same for every object (and hence, the echo delay, d'k , is no longer dependant on the object identifier, m).
- Echo Direction Simplification: The original reverberation calculation (as per Equation 10) treats the reverberation for each object as a series of echoes, wherein for object m, echo e has a direction of arrival,
-
- In
Fig. 2 , theprocessing chain 100 includes a Delay Line, 3, with K taps (and, in the following explanation, the variable k can be used to refer to a specific tap number, so that k ∈ {1,2,...,K}). The input, 2, to theDelay Line 3 is the N -channel input signal, XN (t). At each of the taps (for example, k = 1), an N -channel delayed signal, e.g. 5, is taken from the Delay Line, and processed via an acoustic transformation process, 200, to produce an acoustically transformed delayed signal, 6. The set of K acoustically transformed delayed signals are added together 7 to produce the output soundfield signal, 8. -
-
Fig. 3 illustrates one example form of implementation of anEcho Processor 200 which applies an acoustic transformation process. InFig. 3 , the input N -channel delayedsignal 5, is processed, to produce the N -channel acoustically transformed delayedsignal 6. In the example shown inFig. 3 , two operations are performed by the acoustic transformation process, a multi-channel matrix mixer (represented by the N×N matrix Rk ) 11, and a linear time-invariant filter, Hk (z) e.g. 12, applied to each of the N channels of the soundfield signal. - The intention of the acoustic transformation process, in one embodiment, is to create a simulation of the kth acoustic echo according to the following operating principles:
- Echo Delay: The time delay of echo k is defined by use of the Delay Line so that input to the Delay Line 2 (of
Fig. 2 ), is delayed by d'k samples to give theinput 5, to the kth acoustic transformation process (referring toFigure 2 ). - Echo Direction: The direction of arrival of echo k, for object m, is determined by applying a matrix, Ak to the direction unit-vector of the object, φm = [xm ym zm ] T , resulting in:
- Echo Amplitude and Frequency Response: The amplitude and frequency response of echo k are provided by the filter, Hk (z) e.g. 12, applied to each of the N channels as per
Fig. 3 . - In the case where the soundfield is defined in terms of an Ambisonic panning function (as per Equation 13), a more general version of the acoustic transformation process may be built by converting the Ambisonic signals from B-Format to A-Format. This transformation is known in the art.
-
- Equation 19 defines a 4×4 matrix, AtoB that maps an A-format signal, represented by a 4xlcolumn vector, to a B-format signals, also represented by a 4x1 column vector : BF = AtoB×AF. Likewise,
Equation 20 defines the 4×4 matrix, BtoA that is the inverse of AtoB. -
-
- A processing train for implementing the method of Equation 25 is also shown in
Fig. 4 , with the matrix processing Bk and Ck being separately implemented 21, 23. - As shown in
Fig. 5 , in it's most general form, an acoustic transformation process can be implemented as a 4x4 matrix ofarbitrary filter operations 200. - The methods described above may also be combined with alternative reverberation processes, which may be known in the art, to produce a reverberant mixture that contains some echoes generated according to the above described methods, along with additional echoes and reverberation that are generated by the alternative methods.
- Reference throughout this specification to "one embodiment", "some embodiments" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, appearances of the phrases "in one embodiment", "in some embodiments" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment, but may. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more embodiments.
- As used herein, unless otherwise specified the use of the ordinal adjectives "first", "second", "third", etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.
- In the claims below and the description herein, any one of the terms comprising, comprised of or which comprises is an open term that means including at least the elements/features that follow, but not excluding others. Thus, the term comprising, when used in the claims, should not be interpreted as being limitative to the means or elements or steps listed thereafter. For example, the scope of the expression a device comprising A and B should not be limited to devices consisting only of elements A and B. Any one of the terms including or which includes or that includes as used herein is also an open term that also means including at least the elements/features that follow the term, but not excluding others. Thus, including is synonymous with and means comprising.
- As used herein, the term "exemplary" is used in the sense of providing examples, as opposed to indicating quality. That is, an "exemplary embodiment" is an embodiment provided as an example, as opposed to necessarily being an embodiment of exemplary quality.
- It should be appreciated that in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, FIG., or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.
- Furthermore, while some embodiments described herein include some but not other features included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention, and form different embodiments, as would be understood by those skilled in the art. For example, in the following claims, any of the claimed embodiments can be used in any combination.
- Furthermore, some of the embodiments are described herein as a method or combination of elements of a method that can be implemented by a processor of a computer system or by other means of carrying out the function. Thus, a processor with the necessary instructions for carrying out such a method or element of a method forms a means for carrying out the method or element of a method. Furthermore, an element described herein of an apparatus embodiment is an example of a means for carrying out the function performed by the element for the purpose of carrying out the invention.
- In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
- Similarly, it is to be noticed that the term coupled, when used in the claims, should not be interpreted as being limited to direct connections only. The terms "coupled" and "connected," along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. Thus, the scope of the expression a device A coupled to a device B should not be limited to devices or systems wherein an output of device A is directly connected to an input of device B. It means that there exists a path between an output of A and an input of B which may be a path including other devices or means. "Coupled" may mean that two or more elements are either in direct physical or electrical contact, or that two or more elements are not in direct contact with each other but yet still co-operate or interact with each other.
- Thus, while there has been described what are believed to be the preferred embodiments of the invention, those skilled in the art will recognize that other and further modifications may be made thereto, and it is intended to claim all such changes and modifications as falling within the scope of the invention. For example, any formulas given above are merely representative of procedures that may be used. Functionality may be added or deleted from the block diagrams and operations may be interchanged among functional blocks. Steps may be added or deleted to methods described within the scope of the present invention. The matter for which protection is sought is defined in the appended set of claims.
Claims (15)
- A method for creating an output soundfield signal (8) from an input soundfield signal (2), the method including the steps of:(a) forming at least one delayed soundfield signal (5) from said input soundfield signal,(b) for each of said delayed soundfield signals, creating an acoustically transformed delayed soundfield signal (6), by an acoustic transformation process (200), and(c) combining together (7) said acoustically transformed delayed soundfield signals and said input soundfield signal to produce said output soundfield signal.
- A method according to Claim 1, wherein the acoustic transformation process includes creating a direction of arrival of the respective delayed soundfield signal different from a direction of arrival of the input; soundfield signal, relative to a listening position, wherein optionally the direction of arrival of the respective delayed soundfield signal is created by applying a geometric transformation to the direction of arrival regarding the input soundfield signal.
- A method as claimed in claim 1 or 2, wherein the acoustic transformation process utilises a multi-channel matrix mixer (11),
wherein optionally the multi-channel matrix mixer is formed by combining one or more spatial operations, including a spatial rotation operation. - A method as claimed in claim 3, wherein the multi-channel matrix mixer is formed by combining one or more spatial operations, including a spatial mirror operation.
- A method as claimed in claim 3 or 4, wherein the multi-channel matrix mixer is formed by combining one or more spatial operations, including a directional gain operation.
- A method as claimed in any of claims 3 to 5, wherein the multi-channel matrix mixer is formed by combining one or more spatial operations, including a directional permutation operation.
- A method as claimed in any of claims 3 to 6 wherein the acoustic transformation process includes frequency-dependant filtering.
- A method for adding simulated reverberance to an input sound field signal, the method including the steps of:(a) receiving an input soundfield signal (2) including at least one audio component encoded with a first direction of arrival;(b) determining a further soundfield signal (6) including at least one simulated echo of the at least one audio component, the at least one simulated echo having an alternative direction of arrival;(c) combining (7) the input soundfield signal and the further soundfield signal to produce an output sound field signal (8),
wherein determining a further soundfield signal comprises applying a geometric transformation (200) to the input soundfield signal such that the alternative direction of arrival comprises a geometric transformation of the first direction of arrival. - A method as claimed in claim 8 wherein the method further comprises delaying and rotating a copy of the input soundfield signal and each simulated echo comprises a delayed and rotated copy of the input sound field signal,
wherein optionally each simulated echo includes substantially the same delay. - A method according to claim 8 or 9, wherein the direction of arrival and the alternative direction of arrival relate to a listening position.
- A system (100) for processing of soundfield signals (2) to simulate the presence of reverberance, the system including:an input unit for the input of a soundfield encoded signal;a tapped delay line (3) interconnected to the input unit and providing a series of tapped delays (5) of the soundfield encoded signal;a series of acoustic transformation units (200), each acoustic transformation unit interconnected to an output tap of the tapped delay line, each acoustic transformation unit adapted for applying an acoustic transformation to a respective delayed soundfield encoded signal to produce a transformed delayed soundfield encoded signal (6); anda combining unit (7) for combining the transformed delayed soundfield encoded signals into an output soundfield signal (8).
- A system as claimed in claim 11 wherein said acoustic transformation units include:a multi channel matrix multiplier (11) for applying a geometric transformation to a delayed soundfield encoded signal to produce a geometric transformed output; anda series of linear audio filters (12) applied to each channel of the geometric transformed output,wherein optionally said filters are linear time invariant filters.
- A system as claimed in claim 12 wherein said multi channel matrix multiplier implements one or more spatial operations on a delayed soundfield encoded signal,
wherein optionally said spatial operations include at least one of spatial rotation, spatial mirroring, directional gain operation, or a directional permutation operation. - A system as claimed in claim 12 or 13, wherein the acoustic transformation includes creating a direction of arrival of the respective delayed soundfield encoded signal different from a direction of arrival of the soundfield encoded signal, relative to a listening position,
wherein optionally the direction of arrival of the respective delayed soundfield encoded signal is created by applying a geometric transformation to the direction of arrival regarding the soundfield encoded signal. - A computer program products including program instructions which, when executed by a computing device or system, cause said computing device or system to perform the method according to any of claims 1 to 10.
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201562198440P | 2015-07-29 | 2015-07-29 | |
EP15185913 | 2015-09-18 | ||
PCT/US2016/044286 WO2017019781A1 (en) | 2015-07-29 | 2016-07-27 | System and method for spatial processing of soundfield signals |
Publications (2)
Publication Number | Publication Date |
---|---|
EP3329485A1 EP3329485A1 (en) | 2018-06-06 |
EP3329485B1 true EP3329485B1 (en) | 2020-08-26 |
Family
ID=54150339
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP16747709.0A Active EP3329485B1 (en) | 2015-07-29 | 2016-07-27 | System and method for spatial processing of soundfield signals |
Country Status (3)
Country | Link |
---|---|
EP (1) | EP3329485B1 (en) |
CN (2) | CN107851432B (en) |
WO (1) | WO2017019781A1 (en) |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
HUE056176T2 (en) | 2015-02-12 | 2022-02-28 | Dolby Laboratories Licensing Corp | Headphone virtualization |
EP3329485B1 (en) * | 2015-07-29 | 2020-08-26 | Dolby Laboratories Licensing Corporation | System and method for spatial processing of soundfield signals |
CN110800048B (en) * | 2017-05-09 | 2023-07-28 | 杜比实验室特许公司 | Processing of multichannel spatial audio format input signals |
US10390166B2 (en) | 2017-05-31 | 2019-08-20 | Qualcomm Incorporated | System and method for mixing and adjusting multi-input ambisonics |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
GB9107011D0 (en) * | 1991-04-04 | 1991-05-22 | Gerzon Michael A | Illusory sound distance control method |
WO2007010771A1 (en) * | 2005-07-15 | 2007-01-25 | Matsushita Electric Industrial Co., Ltd. | Signal processing device |
US8705757B1 (en) * | 2007-02-23 | 2014-04-22 | Sony Computer Entertainment America, Inc. | Computationally efficient multi-resonator reverberation |
WO2014159376A1 (en) * | 2013-03-12 | 2014-10-02 | Dolby Laboratories Licensing Corporation | Method of rendering one or more captured audio soundfields to a listener |
HUE056176T2 (en) * | 2015-02-12 | 2022-02-28 | Dolby Laboratories Licensing Corp | Headphone virtualization |
EP3329485B1 (en) * | 2015-07-29 | 2020-08-26 | Dolby Laboratories Licensing Corporation | System and method for spatial processing of soundfield signals |
-
2016
- 2016-07-27 EP EP16747709.0A patent/EP3329485B1/en active Active
- 2016-07-27 CN CN201680043670.9A patent/CN107851432B/en active Active
- 2016-07-27 CN CN202111507803.2A patent/CN114302315B/en active Active
- 2016-07-27 WO PCT/US2016/044286 patent/WO2017019781A1/en unknown
Non-Patent Citations (1)
Title |
---|
None * |
Also Published As
Publication number | Publication date |
---|---|
WO2017019781A1 (en) | 2017-02-02 |
CN107851432A (en) | 2018-03-27 |
CN114302315A (en) | 2022-04-08 |
EP3329485A1 (en) | 2018-06-06 |
CN107851432B (en) | 2022-01-28 |
CN114302315B (en) | 2023-10-31 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110809227B (en) | Reverberation generation for headphone virtualization | |
Noisternig et al. | A 3D ambisonic based binaural sound reproduction system | |
US5802180A (en) | Method and apparatus for efficient presentation of high-quality three-dimensional audio including ambient effects | |
Davis et al. | High order spatial audio capture and its binaural head-tracked playback over headphones with HRTF cues | |
EP2829082B1 (en) | Method and system for head-related transfer function generation by linear mixing of head-related transfer functions | |
EP3329485B1 (en) | System and method for spatial processing of soundfield signals | |
EP3569000B1 (en) | Dynamic equalization for cross-talk cancellation | |
WO2014076030A1 (en) | Segment-wise adjustment of spatial audio signal to different playback loudspeaker setup | |
Farina et al. | Ambiophonic principles for the recording and reproduction of surround sound for music | |
JP2013211906A (en) | Sound spatialization and environment simulation | |
CN104581610A (en) | Virtual stereo synthesis method and device | |
US11950078B2 (en) | Binaural dialogue enhancement | |
Koyama et al. | Wave field reconstruction filtering in cylindrical harmonic domain for with-height recording and reproduction | |
Pihlajamäki et al. | Projecting simulated or recorded spatial sound onto 3D-surfaces | |
US11381927B2 (en) | System and method for spatial processing of soundfield signals | |
EP3375208B1 (en) | Method and apparatus for generating from a multi-channel 2d audio input signal a 3d sound representation signal | |
Tsingos et al. | Surround sound with height in games using Dolby Pro Logic Iiz | |
Vorländer et al. | Virtual room acoustics | |
Pelzer et al. | 3D reproduction of room acoustics using a hybrid system of combined crosstalk cancellation and ambisonics playback | |
Zotter et al. | Signal flow and effects in ambisonic productions | |
Kirkeby | Reproduction of acoustic fields | |
McGrath et al. | Creation, manipulation and playback of sound field | |
Vorländer | Convolution and sound synthesis | |
Saari | Modulaarisen arkkitehtuurin toteuttaminen Directional Audio Coding-menetelmälle | |
Pulkki | Implementing a modular architecture for virtual-world Directional Audio Coding |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20180228 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20200330 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602016042768 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 1307120 Country of ref document: AT Kind code of ref document: T Effective date: 20200915 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20201228 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20201127 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20201126 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20201126 |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20200826 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1307120 Country of ref document: AT Kind code of ref document: T Effective date: 20200826 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20201226 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602016042768 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 |
|
26N | No opposition filed |
Effective date: 20210527 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20210731 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20210731 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20210731 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20210727 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20210727 Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20210731 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20160727 |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230513 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20230621 Year of fee payment: 8 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20230620 Year of fee payment: 8 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200826 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20240620 Year of fee payment: 9 |