EP3843419B1 - Method for controlling a microphone array and device for controlling a microphone array - Google Patents
Method for controlling a microphone array and device for controlling a microphone array Download PDFInfo
- Publication number
- EP3843419B1 EP3843419B1 EP20209487.6A EP20209487A EP3843419B1 EP 3843419 B1 EP3843419 B1 EP 3843419B1 EP 20209487 A EP20209487 A EP 20209487A EP 3843419 B1 EP3843419 B1 EP 3843419B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- directional characteristic
- opening angle
- sound
- width
- moving object
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims description 20
- 230000035945 sensitivity Effects 0.000 claims description 9
- 239000002775 capsule Substances 0.000 claims description 6
- 238000001514 detection method Methods 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000003079 width control Methods 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 230000007423 decrease Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000005236 sound signal Effects 0.000 description 2
- NCGICGYLBXGBGN-UHFFFAOYSA-N 3-morpholin-4-yl-1-oxa-3-azonia-2-azanidacyclopent-3-en-5-imine;hydrochloride Chemical compound Cl.[N-]1OC(=N)C=[N+]1N1CCOCC1 NCGICGYLBXGBGN-UHFFFAOYSA-N 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R3/00—Circuits for transducers, loudspeakers or microphones
- H04R3/005—Circuits for transducers, loudspeakers or microphones for combining the signals of two or more microphones
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R1/00—Details of transducers, loudspeakers or microphones
- H04R1/20—Arrangements for obtaining desired frequency or directional characteristics
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R1/00—Details of transducers, loudspeakers or microphones
- H04R1/20—Arrangements for obtaining desired frequency or directional characteristics
- H04R1/32—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only
- H04R1/326—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only for microphones
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R1/00—Details of transducers, loudspeakers or microphones
- H04R1/20—Arrangements for obtaining desired frequency or directional characteristics
- H04R1/32—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only
- H04R1/40—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only by combining a number of identical transducers
- H04R1/406—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only by combining a number of identical transducers microphones
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R2430/00—Signal processing covered by H04R, not provided for in its groups
- H04R2430/20—Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R2430/00—Signal processing covered by H04R, not provided for in its groups
- H04R2430/20—Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic
- H04R2430/23—Direction finding using a sum-delay beam-former
Definitions
- the invention relates to a method for controlling a microphone array.
- the invention also relates to a device for controlling a microphone array.
- WO2019/211487A1 proposed a microphone array consisting of a circular array of shotgun microphones pointing radially outwards. Because no time-variable control of the sound beam (beam) along the dimension perpendicular to the detection plane is required for level acoustic detection areas, the directivity of the microphones is used there directly as a fixed directivity in relation to this dimension. With regard to the dimension of the plane, however, such an array enables time-variant acoustic beam steering (beam steering) with an almost unchanging beam pattern (beam pattern) in all directions.
- a typical example of such a large planar detection area with a high background noise level at the same time is the sound recording of individual ball kick noises or the referee's whistle during a soccer game.
- the possible Detection area the soccer field.
- there is typically a high level of background noise in a football stadium during a game mainly from the spectator stands around the field.
- a special feature of ball sports in general is the fact that both the ball and the players usually move very quickly and therefore a high speed for the jet steering is required in order to be able to capture the ball kicking noises.
- the microphone array should not be positioned on the playing field, but can be e.g. B. are located on the edge of the field.
- acoustic target's position relative to the array's position can be tracked automatically (e.g., visually using video cameras), beam steering can be performed fully automatically, with no human operator necessary.
- An automatic tracking system tilt system, tracker
- position and speed data of various target objects which are referred to as tracking data.
- the most important target object in this context is the ball.
- the tracking data has a delay (latency) and an uncertainty of that delay.
- the tracking data for controlling the beam direction usually has a certain latency, which is caused, for example, by image processing algorithms used in the context of visual tracking or by the transmission of the tracking data itself from the tracking system to the microphone array. For the detection of moving sound objects with the microphone array, this means that by the time the information about the object position arrives at the microphone array, the object is usually already in a different position, which leads to mismatched beam steering.
- the latency of the tracking data is time-invariant and, more importantly, is usually not precisely known.
- tracking systems usually cannot indicate the exact position of the tracked objects, but they can only indicate the position with a certain positional accuracy, for example in the form of a confidence interval.
- WO2007037700 A1 discloses a system for digitally directional focusing and steering of sampled sound within a target area to produce a selective audio output, comprising one or more broadband arrays of microphones, an A/D signal conversion unit and a control unit, the control unit comprising: receiver means for receiving digital signals of captured sound from all microphones comprising the system; input means for receiving instructions comprising selective position data in the form of coordinates; signal processing means for selecting signals from a selection of relevant microphones in the array for further processing; signal processing means for performing signal processing on the signals from the selection of relevant microphones to focus and direct the sound according to the instructions received; Signal processing means for generating a selective audio output in accordance with the instructions received and the signal processing performed.
- Claim 8 relates to a computer-readable data carrier with instructions stored thereon which are suitable for programming a computer or processor in such a way that it executes the steps of the method. Further advantageous embodiments are described in claims 2-7 and 10-12.
- the latency (including the latency uncertainty) of the tracking data accounts for sound propagation by temporally altering the width of the steered audio beam (beam) so that the beam is still as narrow as possible, but as wide as necessary to safely completely capture the desired object sound capture.
- the beam width depends on the following parameters: the tracking data, ie the speed of the moving object and its distance from the microphone array, and the time it takes for the tracking data to arrive (tracking latency).
- 1 shows a sketched sequence of position measurement, sound event and arrival of the sound at the microphone array using the example of a soccer game.
- 1 a) is at a first time t TR the position of a ball 10 on a playing field and its speed along the trajectory Tro on which the ball is moving, determined by an automatic video tracking system. However, its tracking data is not yet available at this time.
- the video tracking system can also measure positions and speeds of players 20 or the referee.
- Fig. 1 b At a second, initially unknown point in time t E , player 20 hits ball 10 , with a sound event occurring whose sound waves 50 are to be picked up by microphone array 40 .
- the ball changes its movement and follows e.g. B. the new trajectory Tr 1 .
- the sound waves need some time to arrive at the microphone array.
- the microphone array 40 therefore aligns its directional characteristic accordingly in order to specifically record the sound waves 50 of the sound event.
- the beam can be steered with virtually no delay.
- the beamforming can be done in an external computer or processor. This then receives the tracking data from the tracker and the audio data from the microphone array.
- the tracking data relate to a position of the ball at time t TR , while the sound waves originate from the sound event at time t E . If the sound propagation time equals the tracking latency, as in the case described above, the tracking data and the sound match. Otherwise the sound event is generated at an earlier or later point in time and thus at a different position. Since the position, the trajectory Tr 0 (i.e. the direction of movement) and the speed of the ball at time t TR are known from the tracking data, and the tracking latency and tracking accuracy are also known or can at least be estimated, the ball position at Time t E are calculated or estimated. The beam can then be made so wide that the location of the sound event at time t E is also recorded.
- r MAX v S * d TRACK (with the speed of sound vs and the tracking latency d TRACK ).
- the array therefore already receives the information at the point in time to as to where it should sensibly align itself at the later point in time ts in order to possibly capture the sound of the sound event or of the acoustic object. This information is therefore buffered in a suitable manner between times to and ts.
- the acoustic beam for reported positions with a distance of r MAX or greater can be made as narrow as possible, and in particular narrower than with conventional microphone arrays.
- the width of the acoustic beam can also be controlled adaptively in order to take these uncertainties into account. In general, it then makes sense to increase the beam width the more the faster the object that generates the sound moves and the smaller the distance of the object from the array.
- the direction in which the ball is moving is not necessarily known. It is also known that the tracking system has a latency d TRACK that z. B. is 0.1s.
- the ball After the tracking time t TR , the ball is hit by a player at the event time t E , with the noise of the ball being kicked being produced as the sound event to be recorded, and deflected in the process.
- three different possible trajectories of the ball are represented by different trajectories Tr 1 , Tr 2 , Tr 3 , in which the sound of the ball being kicked occurs at different points and the sound of the ball being kicked, taking into account the sound propagation through the air, at the time to am array arrives. It is assumed here that the ball has the same speed v BALL on all three possible trajectories.
- the corresponding positions at which the ball can be kicked are denoted by p 1,K , p 2,K and p 3,K .
- a challenge for beam width control is therefore to fully capture the sound of the ball kick noise while maintaining the narrowest possible beam to avoid ambient noise, e.g. B. the diffuse noise of the spectator tiers, to suppress as much as possible.
- the range B Tr of the possible true ball position at the tracking time t TR is first constructed, which in 2 is represented by a dashed circle. Its radius r Tr of z. B. 3 m results from the ball movement, starting from the tracking position p TR , for the duration of the tracking latency d TRACK with a speed of v BALL .
- a simple choice of beamwidth could be as narrow as possible to fit the dashed circle capture. However, this method would overestimate the beam width actually required and would therefore be unnecessarily inaccurate.
- vs is the speed of sound
- r is the distance from the microphone array to the tracking position
- the range B real of the possible ball position becomes smaller when the distance from the tracking position p TR to the array increases, when the ball speed v BALL decreases , or when the maximum latency of the tracker decreases.
- tracking accuracy can also be factored into beamwidth control, with the less precise the tracking, the more the beamwidth needs to be increased.
- a basic idea of the proposed beam width control is that a certain amount of time has already elapsed between the occurrence of the sound event at the sound source and the time at which the sound reaches the microphone array, during which the sound source has already moved on.
- FIG. 12 shows, in an embodiment of the invention, a block diagram of a device according to the invention.
- the device 200 contains a first input interface 210 for position information, which contains at least the position p TR and the speed of a moving object 10 and can come from a tracking system, and a second input interface 220 with multiple inputs for microphone signals AS in,1 ,.., AS in,N coming from multiple microphone capsules.
- the device 200 includes a calculation unit 230 for calculating a directional characteristic from the plurality Microphone signals by means of beamforming (beamforming), the directional characteristic having at least one preferred direction of high sensitivity corresponding to the position information.
- beamforming beamforming
- the directional characteristic or the beam can be aligned with the position obtained from the tracking system in order to record the sound arriving from this direction.
- the calculation unit 230 recalculates the directional characteristic at least for each newly arriving item of position information.
- updated position information can be received from the tracking system at regular intervals of, for example, 40 ms to a maximum of 100 ms.
- the distance r between the tracking position and the position of the microphone array is taken into account by forming the narrowest possible beam for large distances r > r MAX , as described above.
- Known methods are used for this purpose, e.g. B. Delay, superimposition and filtering of the microphone signals.
- the width or the aperture angle (azimuth angle) of the directional characteristic is variable and dependent on the speed of the moving object 10 .
- a higher speed of the moving object 10 leads to a larger width or a larger opening angle of the directional characteristic.
- the minimum width or the minimum opening angle of the directional characteristic is reached, which is then not fallen below (eg 5°-10°).
- the variable directivity can be achieved by z.
- the filters are modified in a filter-and-sum beamformer. Modified filter coefficients can be used for this, which can be retrieved from a memory 235 in which they are stored. To change the orientation, the delay values for the individual microphone signals can be modified.
- appropriate delay values corresponding to the direction can also be retrieved from the memory 235 .
- other values can be changed that determine the beam width or the opening angle, e.g. B. Weighting factors for ambisonic signals in a modal beamformer.
- figure 5 shows in one embodiment of the invention a flow chart of a method according to the invention. It is an automatically executed method 100 for controlling a microphone array 40.
- the method 100 comprises the steps of receiving 110 position information, which contains a position p TR and a speed of a moving object 10, from a tracking system, and receiving 120 a plurality of microphone signals AS in, 1 ,...,AS in,N from several microphone capsules.
- the microphone signals contain sound from a sound event emanating from the moving object 10 .
- a directivity pattern is calculated from the plurality of microphone signals, the directivity pattern being based on beamforming and having at least one preferred direction of high sensitivity according to the position information.
- the width or the opening angle ⁇ of the directional characteristic is variable over time and depends on the speed of the moving object 10 from, whereby a higher speed of the moving object leads to a larger width or a larger opening angle of the directional characteristic.
- the width or the opening angle of the directional characteristic is also changed 140 depending on the tracking latency, with a larger tracking latency leading to a larger width or a larger opening angle of the directional characteristic.
- the width or the opening angle of the directional characteristic is also changed 150 depending on the distance of the moving object 10 from the microphone array, with a greater distance leading to a smaller width or a smaller opening angle of the directional characteristic, and the width or the opening angle of the directional characteristic does not fall below a minimum value.
- different microphone capsules are located in different microphones, each with a directivity, the angle of the directivity characteristic being calculated in only one dimension and the angle of the directivity characteristic being determined in another dimension by the directivity of the microphones.
- updated position information is received 110 from the tracking system at regular intervals of no more than 100 ms.
- B. can be video-based, and the width or the opening angle ⁇ of the beam of the directional characteristic is adapted to the updated position information.
- the invention can be implemented with a configurable computer or processor.
- the configuration is carried out by means of a computer-readable data carrier with instructions stored thereon which are suitable for programming the computer or processor in such a way that it executes the steps of the method described above.
Landscapes
- Health & Medical Sciences (AREA)
- Otolaryngology (AREA)
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Acoustics & Sound (AREA)
- Signal Processing (AREA)
- General Health & Medical Sciences (AREA)
- Circuit For Audible Band Transducer (AREA)
- Measurement Of Velocity Or Position Using Acoustic Or Ultrasonic Waves (AREA)
Description
Die Erfindung betrifft ein Verfahren zur Steuerung eines Mikrofonarrays. Die Erfindung betrifft ebenfalls eine Vorrichtung zur Steuerung eines Mikrofonarrays.The invention relates to a method for controlling a microphone array. The invention also relates to a device for controlling a microphone array.
Für die Erfassung einzelner akustischer Ereignisse in einem großen planaren Erfassungsbereich bei gleichzeitigem hohen Störgeräuschpegel wird in
Ein typisches Beispiel für einen solchen großen planaren Erfassungsbereich bei gleichzeitigem hohen Störgeräuschpegel ist die Tonaufnahme einzelner Balltrittgeräusche oder der Schiedsrichterpfiff während eines Fußballspiels. Für eine solche Aufgabe ist der mögliche Erfassungsbereich das Fußballfeld. Außerdem ist in einem Fußballstadion während eines Spiels typischerweise ein hoher Störgeräuschpegel vorhanden, der hauptsächlich von den Zuschauertribünen rund um das Spielfeld ausgeht. Eine Besonderheit von Ballsportarten im Allgemeinen ist die Tatsache, dass sich sowohl der Ball als auch die Spieler in der Regel sehr schnell bewegen und daher eine hohe Geschwindigkeit für die Strahllenkung benötigt wird, um die Balltrittgeräusche erfassen zu können. Das Mikrofonarray soll nicht auf dem Spielfeld positioniert sein, sondern kann sich z. B. am Spielfeldrand befinden.A typical example of such a large planar detection area with a high background noise level at the same time is the sound recording of individual ball kick noises or the referee's whistle during a soccer game. For such a task is the possible Detection area the soccer field. In addition, there is typically a high level of background noise in a football stadium during a game, mainly from the spectator stands around the field. A special feature of ball sports in general is the fact that both the ball and the players usually move very quickly and therefore a high speed for the jet steering is required in order to be able to capture the ball kicking noises. The microphone array should not be positioned on the playing field, but can be e.g. B. are located on the edge of the field.
Wenn die Position des akustischen Ziels relativ zur Position des Arrays automatisch verfolgt werden kann (z. B. visuell unter Verwendung von Videokameras), kann die Strahllenkung vollautomatisch durchgeführt werden, wobei ein menschlicher Bediener nicht notwendig ist. Ein automatisches Verfolgungssystem (Trackingsystem, Tracker) liefert hierbei Positions- und Geschwindigkeitsdaten verschiedener Zielobjekte, die als Trackingdaten bezeichnet werden. Das wichtigste Zielobjekt in diesem Zusammenhang ist der Ball. Es treten jedoch folgende Probleme auf.If the acoustic target's position relative to the array's position can be tracked automatically (e.g., visually using video cameras), beam steering can be performed fully automatically, with no human operator necessary. An automatic tracking system (tracking system, tracker) supplies position and speed data of various target objects, which are referred to as tracking data. The most important target object in this context is the ball. However, the following problems arise.
Erstens weisen die Trackingdaten eine Verzögerung (Latenz) und eine Unsicherheit dieser Verzögerung auf. Die Trackingdaten zur Steuerung der Strahlrichtung sind üblicherweise mit einer gewissen Latenz versehen, die beispielsweise durch im Rahmen des visuellen Trackings angewendete Bildverarbeitungsalgorithmen oder durch die Übertragung der Trackingdaten selbst vom Trackingsystem zum Mikrofonarray verursacht wird. Für die Erfassung sich bewegender Schallobjekte mit dem Mikrofonarray bedeutet dies, dass sich das Objekt zu dem Zeitpunkt, zu dem die Informationen über die Objektposition am Mikrofonarray ankommen, üblicherweise bereits an einer anderen Position befindet, was zu einer fehlangepassten Strahllenkung führt. In der Regel ist die Latenz der Trackingdaten zeitinvariant und, was noch wichtiger ist, normalerweise nicht genau bekannt.First, the tracking data has a delay (latency) and an uncertainty of that delay. The tracking data for controlling the beam direction usually has a certain latency, which is caused, for example, by image processing algorithms used in the context of visual tracking or by the transmission of the tracking data itself from the tracking system to the microphone array. For the detection of moving sound objects with the microphone array, this means that by the time the information about the object position arrives at the microphone array, the object is usually already in a different position, which leads to mismatched beam steering. Typically, the latency of the tracking data is time-invariant and, more importantly, is usually not precisely known.
Zweitens gibt es eine Unsicherheit der Genauigkeit der Trackingdaten: Trackingsysteme können in der Regel nicht die genaue Position der verfolgten Objekte angeben, sondern sie können die Position nur mit einer bestimmten Positionsgenauigkeit angeben, beispielsweise in Form eines Konfidenzintervalls.Second, there is an uncertainty in the accuracy of the tracking data: tracking systems usually cannot indicate the exact position of the tracked objects, but they can only indicate the position with a certain positional accuracy, for example in the form of a confidence interval.
Drittens ist die Schallausbreitung mit einer Verzögerung verbunden. Der Schall benötigt eine bestimmte Zeit, um sich von dem Objekt, das das Schallereignis auslöst, zur Mikrofonanordnung auszubreiten. Unter der Annahme, dass sich die zu erfassenden Schallobjekte in einem bestimmten Maximalabstand von der Anordnung (z. B. bis zu 50 m) bewegen, kann dieser Effekt als eine Art "negative Latenz" in Bezug auf die Trackingdatenverarbeitung angesehen werden, für die die Strahlsteuerung warten muss, bis der einer bestimmten Position entsprechende Schall am Mikrofonarray angekommen ist. Im Gegensatz zur Latenz der Trackingdaten ist die "negative Latenz" aufgrund der Schallausbreitung zeitlich variabel, weil sie dem Abstand zwischen dem Schallobjekt und dem Mikrofonarray entspricht.Third, there is a delay in sound propagation. The sound takes a certain amount of time to propagate from the object that triggers the sound event to the microphone array. Assuming that the sound objects to be detected move at a certain maximum distance from the arrangement (e.g. up to 50 m), this effect can be seen as a kind of "negative latency" in terms of tracking data processing, for which the beam steering has to wait until the sound corresponding to a certain position has arrived at the microphone array. In contrast to the latency of the tracking data, the "negative latency" is variable over time due to the sound propagation because it corresponds to the distance between the sound object and the microphone array.
Beide Effekte führen zu einer schlechten Aufnahmequalität des Schallobjekts, da die Strahlrichtung zeitlich nicht korrekt ausgerichtet ist (z.B. wird der Strahl zu spät oder zu früh in die entsprechende Richtung gerichtet).Both effects lead to poor recording quality of the sound object, since the beam direction is not aligned correctly in terms of time (e.g. the beam is directed too late or too early in the corresponding direction).
Gegenwärtig gibt es für das Problem der Latenz der Trackingdaten in einem Echtzeiterfassungssystem nur eine suboptimale Lösung. Dabei wird das Audiosignal einfach um die erwartete mittlere Latenz verzögert, bevor Strahlformung (beamforming) angewendet wird. Diese Lösung berücksichtigt jedoch weder Unsicherheiten bei der Latenz der Trackingdaten noch zeitvariable Abstände zwischen Objekt und Array. Diese Effekte führen häufig zu einer zeitlichen Fehlausrichtung, d.h. zu einer Differenz zwischen der eingestellten Richtung und der tatsächlichen Richtung des zu erfassenden Schallobjekts zu diesem Zeitpunkt.Currently, there is only a suboptimal solution to the problem of tracking data latency in a real-time acquisition system. It simply delays the audio signal by the expected mean latency before applying beamforming. However, this solution neither takes into account uncertainties in the latency of the tracking data nor time-varying distances between object and array. These effects often lead to temporal misalignment, i.e. a difference between the set direction and the actual direction of the sound object to be detected at that time.
Dokument
In der prioritätsbegründenden deutschen Patentanmeldung hat das Deutsche Patent- und Markenamt die folgenden Dokumente recherchiert:
Die Erfindung löst dieses Problem. Ein erfindungsgemäßes Verfahren ist in Anspruch 1 angegeben. Eine entsprechende Vorrichtung ist in Anspruch 9 angegeben. Anspruch 8 betrifft einen computer-lesbaren Datenträger mit darauf gespeicherten Instruktionen, die geeignet sind, einen Computer oder Prozessor derart zu programmieren, dass dieser die Schritte des Verfahrens ausführt. Weitere vorteilhafte Ausführungsformen werden in den Ansprüchen 2-7 und 10-12 beschrieben.The invention solves this problem. A method according to the invention is specified in
Erfindungsgemäß werden die Latenz (einschließlich der Latenzunsicherheit) der Trackingdaten und die Schallausbreitung berücksichtigt, indem die Breite des gelenkten Audiostrahls (beam) zeitlich geändert wird, so dass der Strahl zwar noch so eng wie möglich, aber so breit wie erforderlich ist, um den gewünschten Objektschall sicher vollständig zu erfassen. Somit entsteht eine zeitvariante Strahlbreitenregelung für das Mikrofonarray. Dabei ist die Strahlbreite abhängig von folgenden Parametern: den Trackingdaten, d.h. der Geschwindigkeit des beweglichen Objekts und seiner Entfernung zum Mikrofonarray, sowie von der zeitlichen Dauer bis zum Eintreffen der Trackingdaten (Trackinglatenz). Dadurch kann der gleichzeitig eintreffende Schall eines Schallereignisses, das durch das bewegliche Objekt ausgelöst wird, sicher erfasst werden.According to the invention, the latency (including the latency uncertainty) of the tracking data and accounts for sound propagation by temporally altering the width of the steered audio beam (beam) so that the beam is still as narrow as possible, but as wide as necessary to safely completely capture the desired object sound capture. This creates a time-varying beam width control for the microphone array. The beam width depends on the following parameters: the tracking data, ie the speed of the moving object and its distance from the microphone array, and the time it takes for the tracking data to arrive (tracking latency). As a result, the simultaneously arriving sound of a sound event that is triggered by the moving object can be reliably recorded.
Weitere Einzelheiten und vorteilhafte Ausführungsformen sind in den Zeichnungen dargestellt. Darin zeigt
- Fig. 1
- einen skizzierten Ablauf von Positionsmessung, Schallereignis und Eintreffen des Schalls und der Positionsdaten am Mikrofonarray;
- Fig. 2
- eine Draufsicht auf ein Spielfeld in einer ersten Situation;
- Fig. 3
- eine Draufsicht auf ein Spielfeld in einer zweiten Situation;
- Fig. 4
- ein Blockdiagramm einer erfindungsgemäßen Vorrichtung; und
- Fig. 5
- ein Flussdiagramm eines erfindungsgemäßen Verfahrens.
- 1
- a sketched sequence of position measurement, sound event and arrival of the sound and the position data at the microphone array;
- 2
- a top view of a playing field in a first situation;
- 3
- a plan view of a playing field in a second situation;
- 4
- a block diagram of a device according to the invention; and
- figure 5
- a flowchart of a method according to the invention.
In
Allerdings beziehen sich die Trackingdaten auf eine Position des Balls zum Zeitpunkt tTR, während die Schallwellen vom Schallereignis zum Zeitpunkt tE stammen. Wenn die Zeit der Schallausbreitung der Trackinglatenz entspricht, wie im oben beschriebenen Fall, passen die Trackingdaten und der Schall zusammen. Anderenfalls wird das Schallereignis zu einem früheren oder späteren Zeitpunkt und damit an einer anderen Position erzeugt. Da die Position, die Trajektorie Tr0 (d.h. die Bewegungsrichtung) und die Geschwindigkeit des Balls zum Zeitpunkt tTR aus den Trackingdaten bekannt sind, und auch die Trackinglatenz und die Tracking-Genauigkeit bekannt ist oder zumindest abgeschätzt werden kann, kann aber die Ballposition zum Zeitpunkt tE berechnet bzw. abgeschätzt werden. Der Strahl kann dann so breit gemacht werden, dass der Ort des Schallereignisses zum Zeitpunkt tE mit erfasst wird.However, the tracking data relate to a position of the ball at time t TR , while the sound waves originate from the sound event at time t E . If the sound propagation time equals the tracking latency, as in the case described above, the tracking data and the sound match. Otherwise the sound event is generated at an earlier or later point in time and thus at a different position. Since the position, the trajectory Tr 0 (i.e. the direction of movement) and the speed of the ball at time t TR are known from the tracking data, and the tracking latency and tracking accuracy are also known or can at least be estimated, the ball position at Time t E are calculated or estimated. The beam can then be made so wide that the location of the sound event at time t E is also recorded.
Wenn der Abstand zwischen der vom Trackingsystem bereitgestellten Position und dem Mikrofonarray größer ist als ein Maximalwert rMAX, kann der Schall diese Entfernung nicht innerhalb der Latenz des Trackingsystems zurücklegen. Daher liegen die (zum Schallereignis gehörenden) Trackingdaten in diesem Fall bereits am Mikrofonarray 40 vor, wenn die Schallwellen 50 eintreffen. Diese Entfernung ergibt sich aus rMAX = vS * dTRACK (mit der Schallgeschwindigkeit vs und der Trackinglatenz dTRACK). In
Darüber hinaus kann es auch Fälle geben, in denen die Latenz des Trackingsystems nicht genau bekannt ist oder die vom Trackingsystem kommenden Positionsdaten fehlerbehaftet sind. In diesen Fällen kann aber als obere Schranke eine maximal mögliche Latenz angegeben werden. Daher lässt sich in diesen Fällen ebenfalls die Breite des akustische Strahls adaptiv steuern, um diesen Unsicherheiten Rechnung zu tragen. Allgemein ist es dann sinnvoll, die Strahlbreite umso mehr zu vergrößern, je schneller sich das Objekt bewegt, das den Schall erzeugt, und je kleiner der Abstand des Objektes vom Array ist.In addition, there may also be cases in which the latency of the tracking system is not precisely known or the position data coming from the tracking system is inaccurate. In these cases, however, a maximum possible latency can be specified as the upper limit. In these cases, therefore, the width of the acoustic beam can also be controlled adaptively in order to take these uncertainties into account. In general, it then makes sense to increase the beam width the more the faster the object that generates the sound moves and the smaller the distance of the object from the array.
Kritisch ist jedoch der Fall, in dem der Abstand zwischen der vom Trackingsystem bereitgestellten Position und dem Mikrofonarray kleiner ist als der Maximalwert rMAX. Dieser Fall wird im Folgenden betrachtet.However, the case in which the distance between the position provided by the tracking system and the microphone array is smaller than the maximum value r MAX is critical. This case is considered below.
Dazu wird, zum Zeitpunkt to des Eintreffens der Trackingdaten, zunächst der Bereich BTr der möglichen wahren Ballposition im Tracking-Zeitpunkt tTR konstruiert, der in
Wenn jedoch die Schallausbreitung von der Balltritt-Position zum Array berücksichtigt wird, kann eine engere geeignete Strahlbreite errechnet werden. Insbesondere gibt es für alle möglichen Balltritt-Positionen p1,K,p2,K,p3,K eine Mindestzeitdauer dAIR,min, die der Balltritt-Schall benötigt, um sich durch die Luft zum Mikrofonarray auszubreiten. Dementsprechend gibt es eine maximale Zeitdauer dBALL,max, in der sich der Ball bewegt haben kann, bevor er dann getreten wurde, so dass der dabei entstandene Schall zum Zeitpunkt to am Array ankommt. Beide Fälle treten gemeinsam auf, wenn sich der Ball von der Trackingposition pTR entlang der Trajektorie Tr3 direkt in Richtung des Arrays bewegt und auf diesem Weg in einem Abstand rreal, max von der Trackingposition pTR getreten wird. Die Distanz rreal,max kann aus der Tatsache abgeleitet werden, dass sich beide Zeiten dBALL,max (= tE-tTR) und dAIR,min (= t0-tE) zur Tracker-Latenz addieren müssen, damit der Schall des Balltritts zum Zeitpunkt to am Array ankommt, d.h.
Wenn die Zeitangaben durch die entsprechenden Entfernungen und Geschwindigkeiten angegeben werden mit
Allgemein wird der Bereich Breal der möglichen Ballposition kleiner, wenn die Entfernung von der Trackingposition pTR zum Array größer wird, wenn die Ballgeschwindigkeit vBALL kleiner wird, oder wenn die maximale Latenz des Trackers kleiner wird. Außerdem kann auch die Tracking-Genauigkeit in die Steuerung der Strahlbreite einbezogen werden, wobei die Strahlbreite umso mehr zu erhöhen ist, je ungenauer das Tracking ist. Je kleiner der errechnete Bereich Breal der möglichen Ballposition ist, umso geringer ist die Strahlbreite und umso weniger Umgebungsgeräusche werden ungewollt erfasst. Die erfindungsgemäß erhöhte Fokussierung führt daher zu einer verbesserten Audiosignalqualität.In general, the range B real of the possible ball position becomes smaller when the distance from the tracking position p TR to the array increases, when the ball speed v BALL decreases , or when the maximum latency of the tracker decreases. In addition, tracking accuracy can also be factored into beamwidth control, with the less precise the tracking, the more the beamwidth needs to be increased. The smaller the calculated area B real of the possible ball position, the narrower the beam width and the less ambient noise is unintentionally recorded. The increased focus according to the invention therefore leads to an improved audio signal quality.
Ein Grundgedanke der vorgeschlagenen Strahlbreitenregelung ist, dass vom Eintreten des Schallereignisses an der Schallquelle bis zu dem Zeitpunkt, an dem der Schall das Mikrofonarray erreicht, bereits eine gewisse Zeit vergangen ist, in der sich die Schallquelle bereits weiterbewegt hat.A basic idea of the proposed beam width control is that a certain amount of time has already elapsed between the occurrence of the sound event at the sound source and the time at which the sound reaches the microphone array, during which the sound source has already moved on.
Für kleinere Abstände r < rMAX jedoch ist die Breite bzw. der Öffnungswinkel (Azimutwinkel) der Richtcharakteristik variabel und von der Geschwindigkeit des beweglichen Objekts 10 abhängig. Dabei führt eine höhere Geschwindigkeit des beweglichen Objekts 10 zu einer größeren Breite bzw. einem größeren Öffnungswinkel der Richtcharakteristik. Für r=rMAX wird die minimale Breite bzw. der minimale Öffnungswinkel der Richtcharakteristik erreicht, der dann auch nicht unterschritten wird (z.B. 5°-10°). Die variable Richtcharakteristik lässt sich erreichen, indem z. B. in einem Filter-and-Sum Beamformer die Filter modifiziert werden. Dazu können geänderte Filterkoeffizienten eingesetzt werden, die aus einem Speicher 235 abgerufen werden können, in dem sie gespeichert sind. Um die Ausrichtung zu ändern, können die Verzögerungswerte für die einzelnen Mikrofonsignale modifiziert werden. Dazu können in einer Ausführungsform entsprechend der Richtung passende Verzögerungswerte ebenfalls aus dem Speicher 235 abgerufen werden. Für andere Beamformer können andere Werte geändert werden, die die Strahlbreite bzw. den Öffnungswinkel bestimmen, z. B. Gewichtungsfaktoren für Ambisonicssignale in einem Modal-Beamformer.However, for smaller distances r<r MAX , the width or the aperture angle (azimuth angle) of the directional characteristic is variable and dependent on the speed of the moving
In einer Ausführungsform wird die Breite bzw. der Öffnungswinkel der Richtcharakteristik auch in Abhängigkeit von der Trackinglatenz geändert 140, wobei eine größere Trackinglatenz zu einer größeren Breite oder einem größeren Öffnungswinkel der Richtcharakteristik führt. In einer weiteren Ausführungsform wird die Breite bzw. der Öffnungswinkel der Richtcharakteristik auch in Abhängigkeit von der Entfernung des beweglichen Objekts 10 von dem Mikrofonarray geändert 150, wobei eine größere Entfernung zu einer kleineren Breite bzw. einem kleineren Öffnungswinkel der Richtcharakteristik führt, und wobei die Breite bzw. der Öffnungswinkel der Richtcharakteristik einen Mindestwert nicht unterschreiten.In one embodiment, the width or the opening angle of the directional characteristic is also changed 140 depending on the tracking latency, with a larger tracking latency leading to a larger width or a larger opening angle of the directional characteristic. In a further embodiment, the width or the opening angle of the directional characteristic is also changed 150 depending on the distance of the moving
In einer Ausführungsform der Erfindung befinden sich verschiedene der Mikrofonkapseln in verschiedenen Mikrofonen mit jeweils einer Richtwirkung, wobei der Winkel der Richtcharakteristik nur in einer Dimension berechnet wird und in einer anderen Dimension der Winkel der Richtcharakteristik durch die Richtwirkung der Mikrofone bestimmt wird.In one embodiment of the invention, different microphone capsules are located in different microphones, each with a directivity, the angle of the directivity characteristic being calculated in only one dimension and the angle of the directivity characteristic being determined in another dimension by the directivity of the microphones.
In einer Ausführungsform wird in regelmäßigen Abständen von maximal 100 ms aktualisierte Positionsinformation vom Trackingsystem empfangen 110, das z. B. videobasiert sein kann, und die Breite bzw. der Öffnungswinkel α des Strahls der Richtcharakteristik wird an die aktualisierte Positionsinformation angepasst.In one embodiment, updated position information is received 110 from the tracking system at regular intervals of no more than 100 ms. B. can be video-based, and the width or the opening angle α of the beam of the directional characteristic is adapted to the updated position information.
Die Erfindung kann mit einem konfigurierbaren Computer oder Prozessor implementiert werden. Die Konfiguration erfolgt durch einen computerlesbaren Datenträger mit darauf gespeicherten Instruktionen, die geeignet sind, den Computer oder Prozessor so zu programmieren, dass dieser die Schritte des oben beschriebenen Verfahrens ausführt.The invention can be implemented with a configurable computer or processor. The configuration is carried out by means of a computer-readable data carrier with instructions stored thereon which are suitable for programming the computer or processor in such a way that it executes the steps of the method described above.
Claims (12)
- A method (100) for controlling a microphone array (40), comprising- receiving (110) from a tracking system position information including a position (pTR) and a velocity of a moving object (10);- receiving (120) a plurality of microphone signals from a plurality of microphone capsules, the microphone signals comprising sound of a sound event emanating from the moving object (10);- calculating (130) of a directional characteristic from the plurality of microphone signals, wherein the directional characteristic is based on beamforming and has at least one preferred direction of high sensitivity corresponding to the position information, and wherein an audio output signal is generated that includes the sound coming from the preferred direction of high sensitivity; and- providing (160) the audio output signal at an output;- wherein a beam width or opening angle (α) of the directional characteristic is variable over time and depends on the velocity of the moving object (10), wherein a higher velocity of the moving object (10) results in a larger beam width or larger opening angle of the directional characteristic.
- The method according to claim 1, wherein the tracking system has a tracking latency that corresponds to a time duration between measuring the position information and receiving the measured position information at the microphone array, and wherein the beam width or opening angle (α) of the directional characteristic depends also from the tracking latency, wherein a larger tracking latency leads to a larger beam width or larger opening angle of the directional characteristic.
- The method according to claim 1 or 2, wherein the beam width or opening angle (α) of the directional characteristic depends also from a distance between the moving object (10) and the microphone array, wherein a larger distance leads to a smaller beam width or smaller opening angle of the directional characteristic, and wherein the beam width or the opening angle of the directional characteristic remains above a given minimum value.
- The method according to any one of the claims 1 to 3, wherein the microphone capsules are in different microphones, each having a directional characteristic, and wherein the beam angle or opening angle of the directional characteristic is calculated and variable only in one dimension while in another dimension it is determined by a directional characteristic of the microphones.
- The method according to any one of the claims 1 to 4, wherein updated positional information is received (10) in regular time intervals of up to 100 ms from the tracking system, and wherein the beam width or the opening angle (α) of a beam of the directional characteristic is adapted to the updated positional information.
- The method according to any one of the claims 1 to 5, wherein the tracking system is video-based.
- The method according to any one of the claims 1 to 6, wherein the moving object (10) is a ball, a moving playing device or a moving sports device.
- A computer-readable non-transitory storage medium having stored thereon computer-readable instructions that when executed on a computer or processor cause the computer or processor to execute the method according to any one of the claims 1 to 7.
- A device (200) for controlling a microphone array (40), the device comprising- a first input interface (210) for position information, the position information comprising a position and a velocity of a moving object (10);- a second input interface (220) comprising a plurality of inputs for microphone signals coming from a plurality of microphones;- a processing unit (230) configured for calculating (130) a directional characteristic from the plurality of microphone signals, wherein the directional characteristic is based on beamforming and has at least one preferred direction of high sensitivity corresponding to the position information, and wherein an audio output signal comprising sound from the at least one preferred direction of high sensitivity is generated; and- an output interface (240) for providing the audio output signal;- wherein a width or an opening angle (α) of the directional characteristic is variable over time and depends on the velocity of the moving object (10), wherein a higher velocity of the moving object (10) leads to a larger width or larger opening angle of the directional characteristic.
- The device according to claim 9, wherein the position information relates to a first point in time (tTR) at which it was measured, and wherein the sound from the at least one preferred direction of high sensitivity results from a sound event that occurred at a second point in time (tE) different from the first point in time.
- The device according to claim 9 or 10, wherein the width or the opening angle (α) of the directional characteristic depends also from the distance between the moving object (10) and the microphone array, wherein a larger distance results in a smaller width or smaller opening angle of the directional characteristic, and wherein the width or opening angle of the directional characteristic remains above a given minimum value.
- The device according to any one of the claims 9 to 11, wherein the microphone capsules are in different microphones, each having a directional characteristic, and wherein the width or opening angle of the directional characteristic is calculated in only one dimension, while in another dimension it is determined by a width or an opening angle of the microphones.
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
DE102019134541.3A DE102019134541A1 (en) | 2019-12-16 | 2019-12-16 | Method for controlling a microphone array and device for controlling a microphone array |
Publications (2)
Publication Number | Publication Date |
---|---|
EP3843419A1 EP3843419A1 (en) | 2021-06-30 |
EP3843419B1 true EP3843419B1 (en) | 2022-12-28 |
Family
ID=73554322
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP20209487.6A Active EP3843419B1 (en) | 2019-12-16 | 2020-11-24 | Method for controlling a microphone array and device for controlling a microphone array |
Country Status (3)
Country | Link |
---|---|
US (1) | US11303997B2 (en) |
EP (1) | EP3843419B1 (en) |
DE (1) | DE102019134541A1 (en) |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6914854B1 (en) * | 2002-10-29 | 2005-07-05 | The United States Of America As Represented By The Secretary Of The Army | Method for detecting extended range motion and counting moving objects using an acoustics microphone array |
KR100493172B1 (en) * | 2003-03-06 | 2005-06-02 | 삼성전자주식회사 | Microphone array structure, method and apparatus for beamforming with constant directivity and method and apparatus for estimating direction of arrival, employing the same |
DE102004005998B3 (en) * | 2004-02-06 | 2005-05-25 | Ruwisch, Dietmar, Dr. | Separating sound signals involves Fourier transformation, inverse transformation using filter function dependent on angle of incidence with maximum at preferred angle and combined with frequency spectrum by multiplication |
WO2007007446A1 (en) * | 2005-07-14 | 2007-01-18 | Yamaha Corporation | Array speaker system and array microphone system |
US20080247567A1 (en) * | 2005-09-30 | 2008-10-09 | Squarehead Technology As | Directional Audio Capturing |
EP2942975A1 (en) * | 2014-05-08 | 2015-11-11 | Panasonic Corporation | Directivity control apparatus, directivity control method, storage medium and directivity control system |
CN106486147A (en) * | 2015-08-26 | 2017-03-08 | 华为终端(东莞)有限公司 | The directivity way of recording, device and sound pick-up outfit |
DE102018110759A1 (en) | 2018-05-04 | 2019-11-07 | Sennheiser Electronic Gmbh & Co. Kg | microphone array |
-
2019
- 2019-12-16 DE DE102019134541.3A patent/DE102019134541A1/en not_active Withdrawn
-
2020
- 2020-11-24 EP EP20209487.6A patent/EP3843419B1/en active Active
- 2020-12-15 US US17/121,833 patent/US11303997B2/en active Active
Also Published As
Publication number | Publication date |
---|---|
DE102019134541A1 (en) | 2021-06-17 |
EP3843419A1 (en) | 2021-06-30 |
US11303997B2 (en) | 2022-04-12 |
US20210185433A1 (en) | 2021-06-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP2800982B1 (en) | Method and device for measuring the speed of a vehicle independently of the wheels | |
DE102009024339B3 (en) | Bearing method and direction finding system for detecting and tracking temporally successive bearing angles | |
DE102016221049A1 (en) | Apparatus and method for receiving a reflected light pulse in a lidar system | |
EP1405100A1 (en) | Method for correcting data of several opto-electronic sensors | |
DE10138001A1 (en) | Echo signal monitoring device, for use on motor vehicle, has mode for determining relative position of transmitter and receiver units with respect to each other using returned echo | |
WO2013083112A2 (en) | Goal recognition system, and method for recognizing a goal | |
DE102020107484A1 (en) | METHOD AND DEVICE FOR CONFIRMING A RADAR DETECTION | |
DE102018119877A1 (en) | Wide-spaced radar nodes with a clear beam pattern | |
EP2920606A1 (en) | Apparatus and method for determining the elevation angle in a radar system | |
EP3843419B1 (en) | Method for controlling a microphone array and device for controlling a microphone array | |
DE69011566T2 (en) | Acoustic detection device. | |
EP0437649B1 (en) | Method of ultrasound imaging | |
WO2021185722A1 (en) | Method for monitoring a pivoting region of a door during a pivoting process, computer program product, computer-readable storage medium, and pivoting region monitoring system | |
DE2106035B2 (en) | Device for checking the conformity of the shape of an electromagnetically generated object image with a predetermined scheme | |
DE60113834T2 (en) | DOUBLE DETECTION PROCESSING FOR DETECTING SIGNALS WITH HIGH ACCELERATION DETERMINANCE | |
EP1271175B1 (en) | Method for determining the position of a sound emitting target | |
DE102009019405A1 (en) | Apparatus and method for binaural reproduction of audio sonar signals | |
EP2589977B1 (en) | Method and apparatus for correcting systematic DF errors | |
DE102018009571A1 (en) | Method and device for the automatic evaluation and provision of video signals of an event | |
DE2440742B2 (en) | Device for the elimination of interference and fixed signals | |
WO2007045104A1 (en) | Method and device for recording the point of impact of flying objects on a defined target area | |
DE102021100430A1 (en) | Process for the resource-saving arrangement of pulse sequences of tasks for a multifunctional radar system | |
EP2333574B1 (en) | Method and device for improving measurement accuracy and sonar assembly | |
DE3345021C2 (en) | ||
DE2917510C2 (en) | Process for the automatic detection of the width, the head and the side contours of sheet-metal or band-shaped test pieces when testing materials with ultrasound |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20220103 |
|
RBV | Designated contracting states (corrected) |
Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: H04R 1/40 20060101ALI20220610BHEP Ipc: H04R 1/32 20060101ALI20220610BHEP Ipc: H04R 3/00 20060101ALI20220610BHEP Ipc: H04R 1/00 20060101AFI20220610BHEP |
|
INTG | Intention to grant announced |
Effective date: 20220713 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D Free format text: NOT ENGLISH |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 502020002261 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 1541215 Country of ref document: AT Kind code of ref document: T Effective date: 20230115 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D Free format text: LANGUAGE OF EP DOCUMENT: GERMAN |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG9D |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20221228 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230328 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20221228 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20221228 |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20221228 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20221228 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20221228 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20221228 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230329 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20221228 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20221228 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20221228 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230428 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20221228 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20221228 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20221228 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20221228 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20221228 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230428 Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20221228 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 502020002261 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20221228 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
26N | No opposition filed |
Effective date: 20230929 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20221228 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20231123 Year of fee payment: 4 Ref country code: DE Payment date: 20231103 Year of fee payment: 4 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20221228 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R081 Ref document number: 502020002261 Country of ref document: DE Owner name: SENNHEISER ELECTRONIC SE & CO. KG, DE Free format text: FORMER OWNER: SENNHEISER ELECTRONIC GMBH & CO. KG, 30900 WEDEMARK, DE |