EP1257146B1 - Procédé et sytème de traitement de son - Google Patents

Procédé et sytème de traitement de son Download PDF

Info

Publication number
EP1257146B1
EP1257146B1 EP02009933A EP02009933A EP1257146B1 EP 1257146 B1 EP1257146 B1 EP 1257146B1 EP 02009933 A EP02009933 A EP 02009933A EP 02009933 A EP02009933 A EP 02009933A EP 1257146 B1 EP1257146 B1 EP 1257146B1
Authority
EP
European Patent Office
Prior art keywords
sound
microphone array
source
location
enhancing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Lifetime
Application number
EP02009933A
Other languages
German (de)
English (en)
Other versions
EP1257146A3 (fr
EP1257146A2 (fr
Inventor
David John Benjamin Pearce
James Alexander Rex
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Motorola Solutions Inc
Original Assignee
Motorola Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Motorola Inc filed Critical Motorola Inc
Publication of EP1257146A2 publication Critical patent/EP1257146A2/fr
Publication of EP1257146A3 publication Critical patent/EP1257146A3/fr
Application granted granted Critical
Publication of EP1257146B1 publication Critical patent/EP1257146B1/fr
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R3/00—Circuits for transducers, loudspeakers or microphones
    • H04R3/005—Circuits for transducers, loudspeakers or microphones for combining the signals of two or more microphones

Definitions

  • the present invention relates to sound signal processing.
  • the invention relates to enhancing sound signal quality for use in a sound processing device.
  • pick-up of noise and reverberation may be reduced by employing an array of remote microphones.
  • the remote microphones' outputs are processed and then combined into a single signal that has an improved signal quality.
  • the best processing parameters to use in the microphone array depend on the location of the desired sound source relative to the microphones. Many arrays reject noise by having much higher sensitivity to sound coming from the direction (or region) of the desired source than to sounds from elsewhere, where noise sources are located. However, the source's relative location is often unknown initially, and may change over time. For example, people often move their heads whilst speaking, and if an array is in a hand-held device, it is quite likely to move somewhat during use.
  • the location of a sound source relative to a microphone array can be estimated by measuring certain characteristics of the signals received from that source by the array of microphones.
  • An example of one such characteristic is cross-power spectral phase.
  • the other method of location of a sound source relative to a microphone array is estimation by visually imaging the source, with a camera mounted on the same base as the array.
  • the camera image is processed to locate the source within it, using known visual characteristics of the source. For example, a speaker's mouth may be located by looking for his/her face in the image.
  • Granted US patent US-A-4984087 shows a prior art microphone apparatus for a video camera.
  • the methods and system of the invention are for use in portable sound processing devices
  • the methods and system according to the present invention allow processing of sound signals that result in increased quality.
  • the output signal has decreased content of acoustic noise and reverberation caused by surroundings.
  • the output signal is also less sensitive to changes of location of a sound source relative to the microphones. These changes of location are caused both by movement of the sound source, and movement of the device equiped with these microphones.
  • the sound source would be a person speaking and the device might be held in his/her hand.
  • a method for enhancing sound signal quality as defined in claim 1.
  • the method comprises steps of receiving sound from sound source by a microphone array and detecting a movement of this microphone array.
  • the movement is detected by a platform motion sensor.
  • Current location of the sound source relative to the microphone array is estimated on the basis of the sound signal from the microphone array and signals from the platform motion sensor. Results of this estimation are used for enhancing output sound signal quality.
  • An advantage of the present invention is that the motion sensor can continously track the motion of the microphone array, and use this to adjust the current estimate of the relative location of the desired sound source.
  • the changes of relative location of the sound source and microphone array can be tracked even when there is no sound signal.
  • the sound-based source location estimation can only track changes in source location when sound is present.
  • a system for enhancing sound signal quality as defined in claim 5.
  • the system comprises a microphone array and a platform motion sensor that are connected to a source location estimator. Additionally the microphone array is connected to a sound enhancing processor. The sound enhancing processor is connected to a source location estimator.
  • a “movement signal” herein below refers to a signal generated and provided by a platform motion sensor.
  • a “microphone array” mentioned herein below refers to a set of microphones that are a part of a portable device. Such a portable device is typically in a user's hand, during use, and may be subject to various rotational and translational movements.
  • a sound from a sound source 412 which may be moving, is received by a microphone array 402.
  • a platform motion sensor 400 detects a movement of the microphone array 402.
  • a sound signal is provided to a sound enhancing processor 408 and to a source location estimator 406.
  • the movement signal is provided to the source location estimator 406.
  • step 112 the sound enhancing processor 408 enhances the quality of the sound signal. This enhancement may be performed according to methods known in the art, and is not described further here.
  • step 104 on the basis of this sound signal, an estimation of a location of the sound source 412 relative to the microphone array 402 is performed. For the purpose of this estimation, long time intervals of the sound signal are analysed.
  • step 106 on the basis of the movement signal, an estimation of a change of position of the microphone array 402 is performed. For the purpose of this estimation, short time intervals of the movement signal are analysed.
  • step 108 on the basis of results of estimations performed in steps 104 and 106, a current change of source's relative location is estimated.
  • step 110 the estimated current location of the sound source 412 relative to the microphone array 402 is obtained, by combining the current change of source's relative location from step 108 with the location of the sound source 412 relative to the microphone array 402.
  • Results of the estimation obtained in step 110 and the enhanced signal received in step 112 can be used for further processing in other devices, e.g. speech recognition devices.
  • step 200 a sound from a sound source 412, which may be moving, is received by a microphone array 402.
  • platform motion sensor 400 detects a movement of the microphone array 402. Afterwards, a sound signal is provided to a sound enhancing processor 408 and to a source location estimator 406. At the same time the movement signal is provided to the source location estimator 406.
  • step 204 on the basis of this sound signal, an estimation of a location of the sound source 412 relative to the microphone array 402 is performed. For the purpose of this estimation, long time intervals of the sound signal are analysed. Results of this estimation are provided to the sound enhancing processor 408.
  • step 212 the sound enhancing processor 408 enhances the quality of the sound signal using the results of the estimation performed in step 204.
  • step 206 on the basis of the movement signal, an estimation of a change of position of the microphone array 402 is performed. For the purpose of this estimation, short time intervals of the movement signal are analysed.
  • step 208 on the basis of results of estimations performed in steps 204 and 206, a current change of the source's relative location is estimated.
  • step 210 an estimation of current location of the sound source 412 relative to the microphone array 402 is performed, by combining the current change of the source's relative location from step 208 with the location of the sound source 412 relative to the microphone array 402.
  • Results of the estimation obtained in step 210 and the enhanced signal received in step 212 can be used for further processing in other devices, e.g. speech recognition devices.
  • step 300 a sound from a sound source 412, which may be moving, is received by a microphone array 402.
  • platform motion sensor 400 detects a movement of the microphone array 402.
  • a sound signal is provided to a sound enhancing processor 408 and to a source location estimator 406.
  • the movement signal is provided to the source location estimator 406.
  • step 304 on the basis of this sound signal, an estimation of a location of the sound source 412 relative to the microphone array 402 is performed. For the purpose of this estimation, long time intervals of the sound signal are analysed. Simultaneously, in step 306, on the basis of the movement signal an estimation of a change of position of the microphone array 402 is performed. For the purpose of this estimation, short time intervals of the movement signal are analysed.
  • step 308 on the basis of results of estimations performed in steps 304 and 306, a current change of the source's relative location is estimated.
  • step 310 by combining the current change of source's relative location from step 308 with the location of the sound source 412 relative to the microphone array 402, the estimated current location of the sound source 412 relative to the microphone array 402 is obtained.
  • step 312 the sound enhancing processor 408 enhances the quality of the sound signal using the results of the estimation performed in step 310.
  • the sound signal of enhanced quality obtained in step 312 can be used for further processing in other devices, e.g. speech recognition devices.
  • the long time interval referenced in steps 104, 204 and 304 is typically in the range from 10 ms up to 1 s.
  • the short time interval referenced in steps 106, 206 and 306 is typically in the range from 0.1 ⁇ s up to 10 ⁇ s.
  • FIG. 4 depicts a system for enhancing quality of an output sound signal, operative in accordance with the embodiment of the present invention as illustrated in Fig. 3.
  • a system for enhancing sound signal quality comprises a microphone array 402, that is connected to a sound enhancing processor 408, and to a source location estimator 406.
  • the system also comprises a platform motion sensor 400 that is connected to the source location estimator 406.
  • the source location estimator 406 is connected to the sound enhancing processor 408.
  • the microphone array 402 consists of two or more microphones 404.
  • An array of at least two microphones can provide output signals that can be used to estimate both the direction and distance of a source of sound.
  • the platform motion sensor 400 comprises at least one gyroscope and/or accelerometer.
  • the platform motion sensor 400 is fixed to the microphone array 402, or to any other part of the portable sound processing device 410 that forms one unit with the microphone array 402.
  • the source location estimator 406 and sound enhancing processor 408 have respectively a first signal output connection and a second signal output connection.
  • the invention uses the signal from platform motion sensor 400 to provide an indication of movements of the microphone array 402. Therefore it is necessary that sensor 400 and microphone array 402 be linked so that the signal from sensor 400 relates to the motion of microphone array 402.
  • a link may simply include sensor 400 being attached to array 402, or both being attached to the same housing. Therefore, although figure 4 shows elements 400, 402, 406 and 408 all in one unit, elements 406 and 408 may in fact be located elsewhere.
  • a system in accordance with the invention, or the methods of the invention may be used in various portable devices.
  • the invention is usable in portable radio communication devices. Therefore the system may be used in a mobile telephone or a portable or mobile PMR radio.
  • the invention also may be used in a personal digital assistant (PDA) or laptop computer, linked for example by a radio or infra-red communication link to a cellular network.
  • PDA personal digital assistant
  • Such a network may be in a building, or a cellular telephone, or UMTS/3G network.
  • the invention may form part of a Distributed Speech Recognition (DSR) system.
  • DSR Distributed Speech Recognition
  • some speech processing would be performed remotely, i.e. the processing would be carried out at at least two different locations.

Landscapes

  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Otolaryngology (AREA)
  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Acoustics & Sound (AREA)
  • Signal Processing (AREA)
  • Circuit For Audible Band Transducer (AREA)
  • Measurement Of Velocity Or Position Using Acoustic Or Ultrasonic Waves (AREA)
  • Electrophonic Musical Instruments (AREA)
  • Devices For Supply Of Signal Current (AREA)

Claims (8)

  1. Procédé d'amélioration de la qualité d'un signal sonore, destiné à être utilisé dans un dispositif portatif de traitement du son, le procédé comprenant les opérations suivantes :
    - recevoir (300) un son de la part d'une source de sons (412) au moyen d'un ensemble de microphones (402) ;
    - détecter (302) le mouvement dudit ensemble de microphones (402) au moyen d'un capteur (400) de mouvement de plate-forme et produire un signal de mouvement indicatif de ce mouvement ;
    - fournir un signal sonore venant dudit ensemble de microphones (402) à un dispositif (408) de traitement d'amélioration du son et à un estimateur (406) de la position de la source ;
    - fournir ledit signal de mouvement venant dudit capteur de mouvement de plate-forme (400) audit estimateur (406) de la positon de la source ;
    - ledit estimateur (406) de la position de la source estimant la position courante de ladite source sonore (412) par rapport audit ensemble de microphones (402) en utilisant le signal sonore venant dudit ensemble de microphones et le signai produit par le capteur de mouvement de plate-forme ;
    - foumir, audit dispositif (408) de traitement d'amélioration du son, le résultat de ladite estimation de ladite position courante de ladite source sonore (412) par rapport audit ensemble de microphones (402) ;
    - ledit dispositif (408) de traitement d'amélioration du son améliorant (312) ladite qualité du signal sonore ;
    - fournir ledit signal sonore amélioré à un dispositif de traitement du son.
  2. Procédé selon la revendication 1, où ladite estimation dudit emplacement courant de ladite source sonore par rapport audit ensemble de microphones comprend les opérations suivantes :
    - estimer (206) le changement de la position dudit ensemble de microphones sur la base dudit signal de mouvement venant dudit capteur de mouvement de plate-forme ;
    - estimer (208) le changement courant de la position relative de la source sur la base du résultat de ladite estimation de ladite position de ladite source sonore par rapport audit ensemble de microphones et de ladite estimation dudit changement de position dudit ensemble de microphones ;
    - combiner (210) ledit changement courant de la position relative de la source avec ladite position de ladite source sonore par rapport audit ensemble de microphones.
  3. Procédé selon la revendication 1 ou 2, où, dans le but d'effectuer ladite estimation de la position de ladite source sonore par rapport audit ensemble de microphones, on analyse de longs intervalles de temps dudit signal sonore, les longs intervalles de temps étant compris dans l'intervalle de 10 ms à 1 s.
  4. Procédé selon l'une quelconque des revendications précédentes, où, dans le but d'effectuer ladite estimation dudit changement de position dudit ensemble de microphones, on analyse de courts intervalles de temps dudit signal de mouvement, les courts intervalles de temps étant compris dans l'intervalle de 0,1 µs à 10 µs.
  5. Appareil permettant d'améliorer la qualité d'un signal sonore, destiné à être utilisé dans un dispositif portatif (410) de traitement du son portatif, ledit appareil comprenant :
    un dispositif (408) de traitement d'amélioration du son ;
    un estimateur (406) de la position de la source, connecté audit dispositif de traitement d'amélioration du son ;
    un ensemble de microphones (402), connecté audit dispositif (408) de traitement d'amélioration du son et audit estimateur (406) de la position de la source ; et
    un capteur (400) de mouvement de plate-forme, connecté audit estimateur (406) de la position de la source ;
    où ledit ensemble de microphones (402) a pour fonction de fournir un signal sonore audit dispositif (408) de traitement d'amélioration du son et audit estimateur (406) de la position de la source ;
    ledit capteur (400) de mouvement de plate-forme a pour fonction de fournir un signal de mouvement audit estimateur (406) de la position de la source ;
    ledit estimateur (406) de la position de la source a pour fonction d'estimer la position courante de ladite source sonore (412) par rapport audit ensemble de microphones (402) en utilisant le signal sonore venant dudit ensemble de microphones et le signal de mouvement produit par le capteur de mouvement de plate-forme ;
    ledit estimateur de la position de la source a pour fonction de fournir audit dispositif (408) de traitement d'amélioration du son le résultat de ladite estimation de ladite position courante de ladite source sonore (412) par rapport audit ensemble de microphones (402) ;
    ledit dispositif (408) de traitement d'amélioration du son a pour fonction d'améliorer (312) ladite qualité du signal sonore et de fournir un signal sonore amélioré à un dispositif de traitement du son.
  6. Appareil selon la revendication 5, où ledit capteur de mouvement de plate-forme (400) est fixé audit ensemble de microphones.
  7. Appareil selon la revendication 5 ou 6, où ledit dispositif de traitement du son est un module de reconnaissance de la parole.
  8. Téléphone mobile, dispositif radio PMR portatif ou mobile, assistant numérique personnel noté PDA, ou ordinateur portatif, qui comporte un appareil permettant d'améliorer la qualité du signal sonore selon l'une quelconque des revendications 5 à 8.
EP02009933A 2001-05-03 2002-05-03 Procédé et sytème de traitement de son Expired - Lifetime EP1257146B1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GB0110861A GB2375276B (en) 2001-05-03 2001-05-03 Method and system of sound processing
GB0110861 2001-05-03

Publications (3)

Publication Number Publication Date
EP1257146A2 EP1257146A2 (fr) 2002-11-13
EP1257146A3 EP1257146A3 (fr) 2004-01-07
EP1257146B1 true EP1257146B1 (fr) 2006-07-19

Family

ID=9913962

Family Applications (1)

Application Number Title Priority Date Filing Date
EP02009933A Expired - Lifetime EP1257146B1 (fr) 2001-05-03 2002-05-03 Procédé et sytème de traitement de son

Country Status (5)

Country Link
EP (1) EP1257146B1 (fr)
AT (1) ATE333774T1 (fr)
DE (1) DE60213155T2 (fr)
GB (1) GB2375276B (fr)
PT (1) PT1257146E (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12126311B2 (en) 2021-08-27 2024-10-22 Nokia Technologies Oy Processing audio with an audio processing operation

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6675027B1 (en) 1999-11-22 2004-01-06 Microsoft Corp Personal mobile computing device having antenna microphone for improved speech recognition
US7447630B2 (en) 2003-11-26 2008-11-04 Microsoft Corporation Method and apparatus for multi-sensory speech enhancement
US7499686B2 (en) 2004-02-24 2009-03-03 Microsoft Corporation Method and apparatus for multi-sensory speech enhancement on a mobile device
US7283850B2 (en) 2004-10-12 2007-10-16 Microsoft Corporation Method and apparatus for multi-sensory speech enhancement on a mobile device
US7346504B2 (en) 2005-06-20 2008-03-18 Microsoft Corporation Multi-sensory speech enhancement using a clean speech prior
US7680656B2 (en) 2005-06-28 2010-03-16 Microsoft Corporation Multi-sensory speech enhancement using a speech-state model
US7406303B2 (en) 2005-07-05 2008-07-29 Microsoft Corporation Multi-sensory speech enhancement using synthesized sensor signal
US7930178B2 (en) 2005-12-23 2011-04-19 Microsoft Corporation Speech modeling and enhancement based on magnitude-normalized spectra
CN102687529B (zh) * 2009-11-30 2016-10-26 诺基亚技术有限公司 用于处理音频信号的方法和装置
JP5407848B2 (ja) * 2009-12-25 2014-02-05 富士通株式会社 マイクロホンの指向性制御装置
EP2638694A4 (fr) 2010-11-12 2017-05-03 Nokia Technologies Oy Appareil de traitement audio
US20120183156A1 (en) * 2011-01-13 2012-07-19 Sennheiser Electronic Gmbh & Co. Kg Microphone system with a hand-held microphone
CN104244142B (zh) * 2013-06-21 2018-06-01 联想(北京)有限公司 一种麦克风阵列、实现方法及电子设备
GB2521175A (en) * 2013-12-11 2015-06-17 Nokia Technologies Oy Spatial audio processing apparatus
US10477304B2 (en) 2016-06-15 2019-11-12 Mh Acoustics, Llc Spatial encoding directional microphone array
US10356514B2 (en) 2016-06-15 2019-07-16 Mh Acoustics, Llc Spatial encoding directional microphone array

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3025359A (en) * 1960-02-25 1962-03-13 Gulton Ind Inc Vibration-compensated pressure sensitive microphone
US4984087A (en) * 1988-05-27 1991-01-08 Matsushita Electric Industrial Co., Ltd. Microphone apparatus for a video camera
US5717767A (en) * 1993-11-08 1998-02-10 Sony Corporation Angle detection apparatus and audio reproduction apparatus using it
DE19943872A1 (de) * 1999-09-14 2001-03-15 Thomson Brandt Gmbh Vorrichtung zur Anpassung der Richtcharakteristik von Mikrofonen für die Sprachsteuerung

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12126311B2 (en) 2021-08-27 2024-10-22 Nokia Technologies Oy Processing audio with an audio processing operation

Also Published As

Publication number Publication date
ATE333774T1 (de) 2006-08-15
EP1257146A3 (fr) 2004-01-07
GB2375276A (en) 2002-11-06
EP1257146A2 (fr) 2002-11-13
GB0110861D0 (en) 2001-06-27
PT1257146E (pt) 2006-11-30
DE60213155D1 (de) 2006-08-31
DE60213155T2 (de) 2007-07-26
GB2375276B (en) 2003-05-28

Similar Documents

Publication Publication Date Title
EP1257146B1 (fr) Procédé et sytème de traitement de son
JP4694700B2 (ja) 話者の方向をトラッキングする方法およびシステム
CN112019976B (zh) 用于处理音频信号的设备和方法
CN100370830C (zh) 用于音频/图像的说话者检测和定位的方法和装置
CN106782589B (zh) 移动终端及其语音输入方法和装置
US8635066B2 (en) Camera-assisted noise cancellation and speech recognition
US20180132042A1 (en) Laser microphone utilizing mirrors having different properties
CN111641794B (zh) 声音信号采集方法和电子设备
US8868413B2 (en) Accelerometer vector controlled noise cancelling method
US20060182291A1 (en) Acoustic processing system, acoustic processing device, acoustic processing method, acoustic processing program, and storage medium
WO2014161309A1 (fr) Procédé et appareil pour qu'un terminal mobile mette en œuvre un suivi de source vocale
JP2008191662A (ja) 音声制御システムおよび音声制御方法
US20030171932A1 (en) Speech recognition
JP2007221300A (ja) ロボット及びロボットの制御方法
WO2002041632A1 (fr) Enregistrement d'images en mouvement
CN113810825A (zh) 在存在强噪声干扰的情况下的鲁棒的扬声器定位系统和方法
WO2021244056A1 (fr) Procédé et appareil de traitement de données, et support lisible
CN110390953B (zh) 啸叫语音信号的检测方法、装置、终端及存储介质
CN110970046A (zh) 一种音频数据处理的方法及装置、电子设备、存储介质
US6959095B2 (en) Method and apparatus for providing multiple output channels in a microphone
CN117480554A (zh) 语音增强方法及相关设备
WO2022062531A1 (fr) Procédé et appareil d'acquisition de signal audio multicanal, et système
CN110572600A (zh) 一种录像处理方法及电子设备
EP1266538B1 (fr) Systeme de commande sonore spatial
US8831681B1 (en) Image guided audio processing

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU MC NL PT SE TR

AX Request for extension of the european patent

Free format text: AL;LT;LV;MK;RO;SI

PUAL Search report despatched

Free format text: ORIGINAL CODE: 0009013

AK Designated contracting states

Kind code of ref document: A3

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU MC NL PT SE TR

AX Request for extension of the european patent

Extension state: AL LT LV MK RO SI

RIC1 Information provided on ipc code assigned before grant

Ipc: 7H 04R 1/40 B

Ipc: 7H 04R 3/00 A

Ipc: 7G 10K 11/34 B

17P Request for examination filed

Effective date: 20040707

17Q First examination report despatched

Effective date: 20040805

AKX Designation fees paid

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU MC NL PT SE TR

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE CH CY DE DK ES FI FR GR IE IT LI LU MC NL PT SE TR

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT;WARNING: LAPSES OF ITALIAN PATENTS WITH EFFECTIVE DATE BEFORE 2007 MAY HAVE OCCURRED AT ANY TIME BEFORE 2007. THE CORRECT EFFECTIVE DATE MAY BE DIFFERENT FROM THE ONE RECORDED.

Effective date: 20060719

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20060719

Ref country code: BE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20060719

Ref country code: CH

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20060719

Ref country code: LI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20060719

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REF Corresponds to:

Ref document number: 60213155

Country of ref document: DE

Date of ref document: 20060831

Kind code of ref document: P

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20061019

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20061030

REG Reference to a national code

Ref country code: SE

Ref legal event code: TRGR

REG Reference to a national code

Ref country code: PT

Ref legal event code: SC4A

Free format text: AVAILABILITY OF NATIONAL TRANSLATION

Effective date: 20061006

ET Fr: translation filed
PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20070412

Year of fee payment: 6

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20070420

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: IT

Payment date: 20070518

Year of fee payment: 6

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20070531

REG Reference to a national code

Ref country code: PT

Ref legal event code: MM4A

Free format text: LAPSE DUE TO NON-PAYMENT OF FEES

Effective date: 20080204

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20061020

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20080204

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20070503

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20080530

Year of fee payment: 7

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20070503

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20081201

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20080503

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20060719

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20070503

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20060719

REG Reference to a national code

Ref country code: FR

Ref legal event code: ST

Effective date: 20100129

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20090602

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20080424

Year of fee payment: 7

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20091201

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: SE

Payment date: 20160506

Year of fee payment: 15

REG Reference to a national code

Ref country code: SE

Ref legal event code: EUG

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170504