EP3987822A1 - Sound pickup device with improved microphone network - Google Patents
Sound pickup device with improved microphone networkInfo
- Publication number
- EP3987822A1 EP3987822A1 EP20739743.1A EP20739743A EP3987822A1 EP 3987822 A1 EP3987822 A1 EP 3987822A1 EP 20739743 A EP20739743 A EP 20739743A EP 3987822 A1 EP3987822 A1 EP 3987822A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- sphere
- capsules
- planes
- spherical harmonics
- ambisonic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 239000002775 capsule Substances 0.000 claims abstract description 44
- 238000012545 processing Methods 0.000 claims abstract description 26
- 239000011159 matrix material Substances 0.000 claims abstract description 22
- 238000000034 method Methods 0.000 claims abstract description 14
- 230000005236 sound signal Effects 0.000 claims abstract description 9
- 230000008569 process Effects 0.000 claims abstract description 4
- 239000013598 vector Substances 0.000 claims description 19
- 230000000717 retained effect Effects 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 3
- 230000010354 integration Effects 0.000 claims description 3
- 230000001052 transient effect Effects 0.000 claims description 2
- 230000008901 benefit Effects 0.000 description 7
- 238000000354 decomposition reaction Methods 0.000 description 5
- 238000001514 detection method Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 2
- 238000009434 installation Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 210000000481 breast Anatomy 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000001747 exhibiting effect Effects 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 238000010438 heat treatment Methods 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000000523 sample Substances 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R1/00—Details of transducers, loudspeakers or microphones
- H04R1/20—Arrangements for obtaining desired frequency or directional characteristics
- H04R1/32—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only
- H04R1/40—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only by combining a number of identical transducers
- H04R1/403—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only by combining a number of identical transducers loud-speakers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S3/00—Systems employing more than two channels, e.g. quadraphonic
- H04S3/02—Systems employing more than two channels, e.g. quadraphonic of the matrix type, i.e. in which input signals are combined algebraically, e.g. after having been phase shifted with respect to each other
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R1/00—Details of transducers, loudspeakers or microphones
- H04R1/20—Arrangements for obtaining desired frequency or directional characteristics
- H04R1/32—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only
- H04R1/40—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only by combining a number of identical transducers
- H04R1/406—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only by combining a number of identical transducers microphones
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R3/00—Circuits for transducers, loudspeakers or microphones
- H04R3/005—Circuits for transducers, loudspeakers or microphones for combining the signals of two or more microphones
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R2201/00—Details of transducers, loudspeakers or microphones covered by H04R1/00 but not provided for in any of its subgroups
- H04R2201/40—Details of arrangements for obtaining desired directional characteristic by combining a number of identical transducers covered by H04R1/40 but not provided for in any of its subgroups
- H04R2201/401—2D or 3D arrays of transducers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S2400/00—Details of stereophonic systems covered by H04S but not provided for in its groups
- H04S2400/01—Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S2400/00—Details of stereophonic systems covered by H04S but not provided for in its groups
- H04S2400/15—Aspects of sound capture and related signal processing for recording or reproduction
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S2420/00—Techniques used stereophonic systems covered by H04S but not provided for in its groups
- H04S2420/11—Application of ambisonics in stereophonic audio systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
- H04S7/30—Control circuits for electronic adaptation of the sound field
Definitions
- the invention relates to acoustic capture equipment intended to be integrated into a building, for domestic use (home automation context - connected home) or professional (business context).
- this equipment aims to capture the sounds present in a room in order to feed an ambient intelligence system composed of a set of sensors and actuators making it possible to control the parameters (for example temperature, light, or others) and the corresponding building devices (connected objects in particular such as a connected heating installation, connected lamps, etc.).
- the parameters for example temperature, light, or others
- the corresponding building devices connected objects in particular such as a connected heating installation, connected lamps, etc.
- the sounds to be captured can be located anywhere in a room. It is not possible to know their position a priori and to position the sound recording equipment accordingly. It is therefore necessary to have capture equipment capable of covering the entire space homogeneously.
- the visual appearance of the part can also be a constraint parameter.
- the aesthetics of the room should not be degraded by a multitude of capture equipment. It is therefore necessary to favor discreet and compact capture equipment.
- Voice assistants are also known today exhibiting good voice recognition performance in order to improve the quality of interaction with the user. They are equipped with an array of microphones (often circular) in order to be able to focus the capture on the source of interest (i.e. the user) by applying an antenna processing (typically training methods). of tracks or "beamforming"). This improves the quality of the signals received, and eliminates interactions with surrounding noise and the room effect.
- the microphone arrays which can be designed for the context of audio ambient intelligence are conventionally of the linear or spherical type.
- Linear geometry is not optimal because it requires a large number of sensors for efficient capture.
- this type of geometry (linear or spherical) requires placing the antenna in the middle of the room to take advantage of its omnidirectional coverage, which is incompatible with the constraint of discretion of the equipment.
- the geometry is suboptimal in the sense that the microphones pointed towards the wall are useless, and can even be a source of disturbance (capture of unwanted reflections by example).
- a sound pickup device comprising at least:
- processing unit connected to the capsules to receive the signals picked up by the capsules, said processing unit being arranged to:
- such a device can be inserted, for example, in an upper corner of a room or between a wall and a ceiling, discreetly.
- an advantage of such an embodiment is that the number of capsules to be provided can be reduced, compared to what usually requires an embodiment based on a solid sphere.
- the reflections from the ceiling and from the wall (s) are used here to limit the number of spherical harmonics to be taken into account and thus retain a limited number of ambisonic components. Indeed, the supposedly rigid walls induce a large number of zero components. Only harmonics respecting symmetry can be used.
- the retained ambisonic components are associated with spherical harmonics symmetrical with respect to each of the three perpendicular planes and intersecting each other at the center of the sphere S. It is thus possible to select only the harmonics having such symmetries.
- the device may further comprise a fixing support suitable for fixing the device in an upper corner of a room defined by two perpendicular walls and a ceiling overhanging the walls, the walls and the ceiling coinciding with the three aforementioned perpendicular planes and acting as reflective walls of sound waves.
- the ambisonic components retained are associated with spherical harmonics having a degree I and an order m (couples ⁇ l, m ⁇ of Figure 3 described below), such as:
- the number of ambisonic components selected is equal to (A + 1) (A + 2) / 2 where A is the integer part of half of a maximum degree L of the spherical harmonics with which are associated the ambisonic components selected.
- the aforementioned maximum degree L is greater than 4 and preferably greater than 6.
- the ambisonic components retained are associated with spherical harmonics symmetrical with respect to two perpendicular planes and intersecting each other in a straight line passing through the center of the sphere S.
- the device may further comprise a fixing support suitable for fixing the device in a corner of a room defined by a wall and a ceiling, perpendicular to each other, the wall and the ceiling coinciding with said two perpendicular planes and acting as reflecting walls of sound waves.
- the capsules can be positioned on a Gauss-Legendre spherical mesh, and in this case, the device preferably comprises a number N of capsules given by :
- the processing unit can be configured to break down signals from the microphone capsules, on the spherical harmonics associated with the ambisonic components selected, using a matrix of the type:
- - b is a vector matrix containing the selected ambisonic components
- - E is a diagonal matrix containing radial equalization filters of each capsule
- - Y is a matrix containing the spherical harmonics with which the selected ambisonic components are associated
- - G is a diagonal matrix containing integration weights of a Gauss-Legendre mesh for each of the capsules
- the processing unit can be further configured to then weight the vector b by a steering vector given in azimuth and in elevation with respect to a mark defined by the center of the sphere S and the three intersections between the three planes. For example, it is possible to provide for a sweep of this angle of the steering vector to probe the different sources of a part.
- the invention also relates to a method implemented by a processing unit of a device of the above type, in which:
- the signals picked up by the capsules are matrixed according to an ambisonic representation in which only the ambisonic components associated with spherical harmonics, symmetrical with respect to at least two of the aforementioned planes, are retained, and
- the matrix thus obtained (typically a vector of ambisonic components for example) is processed to identify at least one sound source in a space surrounding the portion of the sphere, and to interpret a sound signal from this source. It is thus possible, for example, to focus the listening in a given direction.
- Such an embodiment can be illustrated by way of example by the flowchart of FIG. 6, on which, following the obtaining of the signals from the capsules in step SO, a matrixing of these is carried out.
- This vector b can be weighted in step S2 by a steering vector as presented above.
- Such an embodiment makes it possible to refine the detection of source (s) in step S4 for a better interpretation of the sound signal SIG coming from this (or these) source (s).
- the device is used as a voice assistant to distinctly recognize a COM command in step S5.
- the present invention is also aimed at a computer program comprising instructions for implementing the above method when this program is executed by a processor.
- This may typically be the PROC processor of a processing unit UT such as illustrated by way of example in FIG. 7 further comprising:
- an output interface OUT able to deliver, for example, the interpreted COM control signal (or alternatively the sound signal from the detected source, or alternatively also processed ambisonic signals making it possible to identify a sound source generating the SIG signal).
- the OUT output can deliver the interpretation of the sound event (s) (alarm, dog barking, person falling, etc., or any other situation characterized by the identified sounds), and any information associated with this event (temporal and / or spatial localization).
- the sound event s
- the OUT output can deliver the interpretation of the sound event (s) (alarm, dog barking, person falling, etc., or any other situation characterized by the identified sounds), and any information associated with this event (temporal and / or spatial localization).
- the present invention also relates to a non-transient recording medium readable by a computer on which a program is recorded for the implementation of the above method when this program is executed by a processor.
- FIG. 1 shows embodiments of portions of a sphere.
- FIG. 3 illustrates the principle of a source and an image microphone in the case of acoustic reflection (on a wall such as a wall, a ceiling).
- FIG. 4 illustrates an array of real microphones on a fraction of 1/8 sphere and image microphones (grayed out) generated by reflections on rigid walls.
- FIG. 5 shows an example of channel formation using spherical harmonics.
- FIG. 6 shows an example of a flowchart defining a succession of steps of a method according to one embodiment.
- FIG. 7 shows an example of the structure of a processing unit UT of a device according to one embodiment.
- FIG. 1 a device within the meaning of the invention DIS is in the form of a quarter sphere (upper part of Figure 1) or in the form of an eighth sphere (lower part of figure 1).
- the surface of these portions of sphere is meshed (in a chosen way which can correspond to the Gauss-Legendre spherical mesh as described later) and microphone capsules MIC are arranged on this mesh in a number which can also be determined by the aforementioned Gauss-Legendre mesh.
- These MIC capsules are connected to a processing unit UT (visible in the upper part of FIG. 1) to receive the picked up sound signals and process them by matrixing in ambisonic representation as described in detail below.
- the DIS device may further include a SUP fixing support to be fixed for example:
- the invention thus proposes a capture device consisting of one or more elementary networks of PCM capsules which can be distributed for example in a building room.
- the geometry of an elementary lattice is a fraction of a sphere (1/8 or 1/4) which naturally fits into the upper corners of a room so as to match its architecture, or even on an edge of a room. '' intersection between a ceiling and a wall, in order to take advantage of reflections on such walls.
- the set of capture systems obtained is thus very discreet, considerably reduces the number of microphones while maintaining high directivity, and offers wide coverage of ambient sounds in the room. Indeed, the microphones being located in height, they benefit from a privileged point of capture on the whole of the room without being obstructed by the furniture or the users nearby.
- the choice of a spherical geometry is advantageous in the sense that it makes it possible to obtain (by associating the microphones with an appropriate processing of antenna signals) a high directivity with a low number of sensors.
- the processing of the antenna signals uses spherical harmonic functions in a so-called "ambisonic" context.
- the conventional harmonic functions cannot be applied directly and they should be adapted to the geometry chosen for the array of microphones, according to one embodiment.
- the choice of the positions of the microphones on the fraction of a sphere is to be optimized.
- the optimal mesh must satisfy the best compromise between the number of sensors (to be minimized) and the quality of the information captured (which requires a minimum number of sensors). This is a problem of spatial sampling to adapt to a fraction of a sphere.
- each spherical harmonic is described by its degree I and its order m. At degree I, there is (2I +1) spherical harmonics. Up to the maximum degree L, there are (L + 1) 2 harmonics.
- a spherical array of microphones is usually used to decompose a sound pressure field on the basis of the spherical harmonics, a representation of which is then illustrated in figure 2.
- the pressure received by the image sensor is assumed to be the same as that received by the real sensor without the wall.
- this is an example of an embodiment where the device is fixed between a wall and the ceiling, for example the Oxy and Oyz plans. It can also be fixed between two walls Oyz and Oxz and it is advisable to add the condition of symmetry m greater than or equal to 0, which is specific to Oxz, to the previous condition relating to Oyz (m greater than or equal to 0 AND m is even, OR m ⁇ 0 AND m is odd),
- the signals of the microphones S1, S2, ..., SN are broken down (for example in the frequency domain) on the spherical harmonics, using an equation of the type :
- - b is a vector containing the ambisonic components associated with the spherical harmonics respecting the aforementioned symmetries
- - E is a diagonal (square) matrix containing radial equalization filters of each microphone
- - Y is a matrix (not square because processing more signals from the capsules than ambisonic components at the output) containing the spherical harmonics respecting the aforementioned symmetries evaluated at the different directions of the microphones, and
- - G is a diagonal (square) matrix containing integration weights of the Gauss-Legendre quadrature for each of the microphones of the eighth of a sphere,
- s being a vector containing the signals coming from the microphones.
- Such an embodiment amounts to applying a spherical Fourier transform (referenced SFT in FIG. 5).
- the Spherical harmonic components are first estimated using the matrix equation above.
- the vector obtained b is then weighted by a steering vector (or “steering vector”) which makes it possible to describe the listening in a steering direction.
- the weighted components are summed to obtain the output signal.
- Weightings wi m can be provided for a regular directivity function given by the following equation:
- wim Y im (tetaO, phiO)
- An example of a steering angle can be such that tetaO and phiO are 45 and 135 ° respectively (pointing in this example towards the interior of the room). These respective azimuth and elevation coordinates are given relative to the base formed by the intersections of the three planes Oxy, Oxz, Oyz.
- the directivity function obtained is the superposition of eight directivity functions of a complete sphere pointing in symmetrical directions with respect to the Oxy, Oxz, Oyz planes together.
- the invention finds many applications, in particular in:
- an audio ambient intelligence system allowing from the analysis and recognition of ambient sounds to infer actions and to offer services to the inhabitants of a house or to people of a business (potentially applicable to any place of life);
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Otolaryngology (AREA)
- Acoustics & Sound (AREA)
- Signal Processing (AREA)
- General Health & Medical Sciences (AREA)
- Mathematical Analysis (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Optimization (AREA)
- Mathematical Physics (AREA)
- Pure & Applied Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Algebra (AREA)
- Circuit For Audible Band Transducer (AREA)
- Obtaining Desirable Characteristics In Audible-Bandwidth Transducers (AREA)
Abstract
Description
Claims
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
FR1906840A FR3096550B1 (en) | 2019-06-24 | 2019-06-24 | Advanced microphone array sound pickup device |
PCT/FR2020/050852 WO2020260780A1 (en) | 2019-06-24 | 2020-05-20 | Sound pickup device with improved microphone network |
Publications (2)
Publication Number | Publication Date |
---|---|
EP3987822A1 true EP3987822A1 (en) | 2022-04-27 |
EP3987822B1 EP3987822B1 (en) | 2023-07-05 |
Family
ID=68425020
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP20739743.1A Active EP3987822B1 (en) | 2019-06-24 | 2020-05-20 | Sound pickup device with improved microphone network |
Country Status (4)
Country | Link |
---|---|
US (1) | US11895478B2 (en) |
EP (1) | EP3987822B1 (en) |
FR (1) | FR3096550B1 (en) |
WO (1) | WO2020260780A1 (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11728906B1 (en) * | 2022-04-20 | 2023-08-15 | The United States Of America As Represented By The Secretary Of The Navy | Constant beam width acoustic transducer design method |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6072878A (en) * | 1997-09-24 | 2000-06-06 | Sonic Solutions | Multi-channel surround sound mastering and reproduction techniques that preserve spatial harmonics |
US7782710B1 (en) * | 2005-08-09 | 2010-08-24 | Uzes Charles A | System for detecting, tracking, and reconstructing signals in spectrally competitive environments |
WO2015013058A1 (en) * | 2013-07-24 | 2015-01-29 | Mh Acoustics, Llc | Adaptive beamforming for eigenbeamforming microphone arrays |
US10770087B2 (en) * | 2014-05-16 | 2020-09-08 | Qualcomm Incorporated | Selecting codebooks for coding vectors decomposed from higher-order ambisonic audio signals |
FR3060830A1 (en) * | 2016-12-21 | 2018-06-22 | Orange | SUB-BAND PROCESSING OF REAL AMBASSIC CONTENT FOR PERFECTIONAL DECODING |
US10657974B2 (en) * | 2017-12-21 | 2020-05-19 | Qualcomm Incorporated | Priority information for higher order ambisonic audio data |
JP7072186B2 (en) * | 2018-02-08 | 2022-05-20 | 株式会社オーディオテクニカ | Microphone device and case for microphone device |
EP3525482B1 (en) * | 2018-02-09 | 2023-07-12 | Dolby Laboratories Licensing Corporation | Microphone array for capturing audio sound field |
-
2019
- 2019-06-24 FR FR1906840A patent/FR3096550B1/en active Active
-
2020
- 2020-05-20 EP EP20739743.1A patent/EP3987822B1/en active Active
- 2020-05-20 US US17/622,679 patent/US11895478B2/en active Active
- 2020-05-20 WO PCT/FR2020/050852 patent/WO2020260780A1/en unknown
Also Published As
Publication number | Publication date |
---|---|
EP3987822B1 (en) | 2023-07-05 |
WO2020260780A1 (en) | 2020-12-30 |
FR3096550B1 (en) | 2021-06-04 |
FR3096550A1 (en) | 2020-11-27 |
US20220256302A1 (en) | 2022-08-11 |
US11895478B2 (en) | 2024-02-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP3807669B1 (en) | Location of sound sources in a given acoustic environment | |
Cao et al. | Acoustic vector sensor: reviews and future perspectives | |
US9191738B2 (en) | Sound enhancement method, device, program and recording medium | |
Sun et al. | Localization of distinct reflections in rooms using spherical microphone array eigenbeam processing | |
US11190900B2 (en) | Spatial audio array processing system and method | |
US9100734B2 (en) | Systems, methods, apparatus, and computer-readable media for far-field multi-source tracking and separation | |
US9973848B2 (en) | Signal-enhancing beamforming in an augmented reality environment | |
JP2017102488A (en) | Systems and methods for source signal separation | |
US10708687B1 (en) | System and method for including soundscapes in online mapping utilities | |
MXPA05002370A (en) | System and method for beamforming using a microphone array. | |
Crocco et al. | Audio tracking in noisy environments by acoustic map and spectral signature | |
McCormack et al. | Parametric acoustic camera for real-time sound capture, analysis and tracking | |
Nair et al. | Audiovisual zooming: what you see is what you hear | |
EP3987822B1 (en) | Sound pickup device with improved microphone network | |
EP1230784B1 (en) | Echo attenuating method and device | |
Blanco Galindo et al. | Microphone array geometries for horizontal spatial audio object capture with beamforming | |
Bountourakis et al. | Parametric spatial post-filtering utilising high-order circular harmonics with applications to underwater sound-field visualisation | |
Bianchi et al. | High resolution imaging of acoustic reflections with spherical microphone arrays | |
WO2022106765A1 (en) | Improved location of an acoustic source | |
US11997474B2 (en) | Spatial audio array processing system and method | |
Blanco Galindo et al. | Robust hypercardioid synthesis for spatial audio capture: microphone geometry, directivity and regularization | |
Bountourakis et al. | Underwater soundfield visualisation using directionally constrained acoustic parameters | |
Riaz | Adaptive blind source separation based on intensity vector statistics | |
Fu | Visually-guided beamforming for a circular microphone array | |
Changxi et al. | Audiovisual Zooming: What You See Is What You Hear |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: UNKNOWN |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20211221 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20230126 |
|
GRAJ | Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted |
Free format text: ORIGINAL CODE: EPIDOSDIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTC | Intention to grant announced (deleted) | ||
INTG | Intention to grant announced |
Effective date: 20230414 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 1585914 Country of ref document: AT Kind code of ref document: T Effective date: 20230715 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602020013377 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D Free format text: LANGUAGE OF EP DOCUMENT: FRENCH |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG9D |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20230705 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1585914 Country of ref document: AT Kind code of ref document: T Effective date: 20230705 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230705 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231006 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230705 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231105 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230705 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230705 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231106 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231005 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230705 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230705 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231105 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230705 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231006 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230705 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230705 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230705 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230705 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602020013377 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230705 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230705 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230705 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230705 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230705 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230705 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |