JP3732681B2 - Audio sound canceling device - Google Patents

Audio sound canceling device Download PDF

Info

Publication number
JP3732681B2
JP3732681B2 JP18306199A JP18306199A JP3732681B2 JP 3732681 B2 JP3732681 B2 JP 3732681B2 JP 18306199 A JP18306199 A JP 18306199A JP 18306199 A JP18306199 A JP 18306199A JP 3732681 B2 JP3732681 B2 JP 3732681B2
Authority
JP
Japan
Prior art keywords
signal
audio
adaptive
coefficient
microphone
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
JP18306199A
Other languages
Japanese (ja)
Other versions
JP2001016678A (en
Inventor
真吾 木内
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alpine Electronics Inc
Original Assignee
Alpine Electronics Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alpine Electronics Inc filed Critical Alpine Electronics Inc
Priority to JP18306199A priority Critical patent/JP3732681B2/en
Publication of JP2001016678A publication Critical patent/JP2001016678A/en
Application granted granted Critical
Publication of JP3732681B2 publication Critical patent/JP3732681B2/en
Anticipated expiration legal-status Critical
Expired - Fee Related legal-status Critical Current

Links

Images

Landscapes

  • Circuit For Audible Band Transducer (AREA)
  • Soundproofing, Sound Blocking, And Sound Damping (AREA)

Description

【0001】
【発明の属する技術分野】
本発明は適応信号処理によりオーディオ音をキャンセルする装置に係わり、特に、適応フィルタの係数値を監視して参照信号を増幅する増幅器のゲインを制御し、参照信号と目標応答のレベル比を最適値に自動設定するオーディオ音キャンセル装置に関する。
【0002】
【従来の技術】
音声認識装置やハンズフリー電話機等に話者音声信号を入力する場合、オーディオ音や周囲ノイズを低減したSN比の大きな音声信号を入力することが、音声認識率を向上する上で、あるいは、快適な通信を行う上で必要である。このため、ノイズやオーディオ音を低減するシステムが種々提案されている。
図3はオーディオ音をキャンセルする装置の構成図であり、1はオーディオソース、2はオーディオ信号を入力され、オーディオ音を音響空間に放射するスピーカ、3は音響空間に放射されたオーディオ音を検出するマイクロホン、4はエラー信号eのパワーが最小となるように適応信号処理を行い、適応フィルタの係数Wを更新する適応信号処理部、5、6はそれぞれオーディオソース1とスピーカ2、オーディオソース1と適応信号処理部4間に設けられたアンプ、7はマイクロホン出力信号と適応フィルタ出力の差を演算してエラー信号eを出力する減算器である。
【0003】
適応信号処理部4は、図示しないがLMS演算部と、FIR型ディジタルフィルタ構成の適応フィルタを有している。適応信号処理部4は、オーディオ信号を参照信号、マイクロホンから出力する信号を目標信号とし、エラー信号eのパワーが最小となるように適応フィルタの係数Wを更新し、これにより、オーディオ音の抑圧を行う。すなわち、適応信号処理により、適応フィルタはスピーカ2からマイクロホン3までの伝搬特性を模擬するようになり、オーディオ音がキャンセルされる。
【0004】
【発明が解決しようとする課題】
図3のオーディオキャンセル装置において、オーディオ音の抑圧性能を高くするためには、適応フィルタに設定する係数Wの精度を上げる必要がある。このためには、参照信号と目標応答の振幅比が適切な値になっている必要がある。しかし、一般に、目標応答の振幅はスピーカー2とマイク3の位置関係によって変化する。このため、音響空間を車室とする場合、装置を搭載する車種が決まらないと、参照信号と目標応答の振幅比が適切な値になるようアンプ4,5のゲインを設定することができない。以上より、従来は、装置を車に搭載してからアンプ4,5のゲインを調整してフィルタ係数が適切値となるように調整する。しかし、この調整作業は手間がかかり、ユーザの負担が大きい問題がある。
以上から本発明の目的は、自動的にフィルタ係数が適切値となるように参照信号と目標応答の振幅比を設定することである。
【0005】
【課題を解決するための手段】
上記課題は本発明によれば、オーディオ信号を出力するオーディオソース、オーディオ信号を入力され、オーディオ音を音響空間に放射するスピーカ、音響空間に放射されたオーディオ音を検出するマイクロホン、オーディオ信号を参照信号、マイクロホンから出力する信号を目標信号とし、エラー信号のパワーが最小となるように適応信号処理を行い、適応フィルタの係数を更新して該適応フィルタにより前記スピーカからマイクロホンまでの伝達特性を模擬させる適応信号処理部、オーディオソースと適応信号処理部間に設けられ、オーディオ信号を増幅するゲイン可変の増幅器、前記スピーカからマイクロホンまでの伝達特性を正しく模擬できるように前記適応フィルタ係数の最大値を設定し、該適応フィルタの係数を監視し、該適応フィルタの最大係数値が前記設定値となるように前記増幅器のゲインを制御するゲイン設定部を有するオーディオ音キャンセル装置により達成される。
【0006】
【発明の実施の形態】
図1は本発明のオーディオ音キャンセル装置の構成図である。
図中、11はオーディオソース、12はオーディオ信号を入力され、オーディオ音を音響空間に放射するスピーカ、13は音響空間に放射されたオーディオ音を検出するマイクロホン、14はエラー信号eのパワーが最小となるように適応信号処理を行い、適応フィルタの係数Wを更新する適応信号処理部、15はオーディオソース11とスピーカ12の間に設けられたゲイン固定のアンプ、16はオーディオソース11と適応信号処理部14間に設けられたゲイン可変のアンプ、17は目標信号であるマイクロホン出力と適応フィルタ出力の差を演算してエラー信号eを出力する減算器、18はゲイン可変のアンプ16のゲインを設定するゲイン設定部であ。
【0007】
適応信号処理部14は、図示しないがLMS演算部と、FIR型ディジタルフィルタ構成の適応フィルタを有している。適応信号処理部14は、オーディオ信号を参照信号、マイクロホンから出力する信号を目標信号とし、エラー信号eのパワーが最小となるように適応フィルタ係数Wを更新する。
【0008】
ゲイン設定部18は適応フィルタ係数Wが適切な値になるようにアンプ16のゲインを設定する。フィルタ係数Wの絶対値は0〜1の値を取ることが可能で、大きい程係数の精度を上げることができる。しかし、適応信号処理の計算において係数が1より大きくなると1にクランプされ、適応フィルタは正確にスピーカからマイクロホン迄の伝搬特性を模擬できなくなる。又、フィルタ係数が小さすぎると係数の精度が低下し、同様に、適応フィルタは正確にスピーカからマイクロホン迄の伝搬特性を模擬できなくなる。
そこで、ゲイン設定部18は適応フィルタの係数Wを監視し、フィルタ係数の最大値が設定値たとえば0.95となるようにアンプ16のゲインを制御する。すなわち、フィルタ係数Wの最大値が0.95より小さければフィルタ係数が大きくなるようにゲインを制御し、フィルタ係数の最大値が0.95より大きければフィルタ係数が小さくなるようにゲインを制御する。
【0009】
参照信号が目標信号より小さいと(参照信号<目標信号)、適応信号処理によりフィルタ係数Wが大きくなって参照信号=目標信号となる。逆に、参照信号が目標信号より大きいと(参照信号>目標信号)、適応信号処理によりフィルタ係数Wが小さくなって参照信号=目標信号となる。従って、
▲1▼フィルタ係数が0.95より小さく、該フィルタ係数を大きくするには、参照信号<目標信号となるようにアンプ16のゲインを小さくすれば良く、又、
▲2▼フィルタ係数が0.95より大きく、フィルタ係数を小さくするには、参照信号>目標信号となるようにアンプ16のゲインを大きくすればよい。
ゲイン設定部18は上記▲1▼,▲2▼に従ってアンプ16のゲインを制御し、フィルタ係数の最大値が設定値たとえば0.95となるように制御する。
【0010】
図2は本発明のゲイン設定制御の処理フローである。
適応信号処理部14は、適応信号処理によりフィルタ係数Wの更新を開始し(学習開始、ステップ101)、フィルタ係数Wが一定値に収束した時、フィルタ係数の更新を停止する(学習停止、ステップ102)。ゲイン設定部18はフィルタ係数を読み取り(ステップ103)、最大のフィルタ係数値を求め、該最大のフィルタ係数が設定値(=0.95)に等しくなったかチェックし(ステップ104)、等しければゲイン設定制御を終了する。最大フィルタ係数が設定値(=0.95)より大きければ、アンプ16のゲインを大きくし、最大フィルタ係数が設定値(=0.95)より小さければ、アンプ16のゲインを小さくする(ステップ105)。以後、ステップ101に戻り、フィルタ係数の更新を再開し、最大フィルタ係数=設定値(0.95)となるまで上記処理を繰り返す。
【0011】
以上では設定値を0.95としたが本発明はかかる値に限定するものではない。又、フィルタ係数の適正範囲、たとえば、0.90〜0.95を設定し、この範囲に入るようにゲインを制御するように構成することもできる。
以上、本発明を実施例により説明したが、本発明は請求の範囲に記載した本発明の主旨に従い種々の変形が可能であり、本発明はこれらを排除するものではない。
【0012】
【発明の効果】
以上本発明によれば、フィルタ係数の精度を高くすることができるため、システムが持つ性能を十分に発揮することができる。また、それらの設定は自動で行われるため、それらの設定をおこなう場合のユーザーに対する負荷を従来と比較して著しく軽減することが可能となる。
【図面の簡単な説明】
【図1】本発明のオーディオ音キャンセル装置の構成図である。
【図2】本発明のゲイン設定処理フローである。
【図3】従来のオーディオ音キャンセル装置の構成図である。
【符号の説明】
11・・オーディオソース
12・・スピーカ
13・・マイクロホン
14・・適応信号処理部
15・・ゲイン固定アンプ
16・・ゲイン可変アンプ
17・・減算器
18・・ゲイン設定部
[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an apparatus for canceling audio sound by adaptive signal processing, and in particular, controls the gain of an amplifier that amplifies a reference signal by monitoring the coefficient value of the adaptive filter, and sets the level ratio between the reference signal and the target response to an optimum value. The present invention relates to an audio sound canceling apparatus that is automatically set to “.
[0002]
[Prior art]
When a speaker voice signal is input to a voice recognition device, a hands-free telephone, etc., input of a voice signal having a large S / N ratio with reduced audio sound and ambient noise improves the voice recognition rate or is comfortable. Necessary for proper communication. For this reason, various systems for reducing noise and audio sound have been proposed.
FIG. 3 is a block diagram of an apparatus for canceling an audio sound, where 1 is an audio source, 2 is an audio signal input, and a speaker that radiates the audio sound into the acoustic space. 3 is an audio sound radiated into the acoustic space. The microphones 4 perform adaptive signal processing so that the power of the error signal e is minimized, and the adaptive signal processing units 5 and 6 update the coefficient W of the adaptive filter. And an amplifier 7 provided between the adaptive signal processing unit 4 and a subtractor 7 which calculates the difference between the microphone output signal and the adaptive filter output and outputs an error signal e.
[0003]
Although not shown, the adaptive signal processing unit 4 includes an LMS calculation unit and an adaptive filter having an FIR digital filter configuration. The adaptive signal processing unit 4 uses the audio signal as the reference signal and the signal output from the microphone as the target signal, and updates the coefficient W of the adaptive filter so that the power of the error signal e is minimized, thereby suppressing the audio sound. I do. That is, by the adaptive signal processing, the adaptive filter comes to simulate the propagation characteristics from the speaker 2 to the microphone 3, and the audio sound is canceled.
[0004]
[Problems to be solved by the invention]
In the audio cancellation apparatus of FIG. 3, in order to increase the audio sound suppression performance, it is necessary to increase the accuracy of the coefficient W set in the adaptive filter. For this purpose, the amplitude ratio between the reference signal and the target response needs to be an appropriate value. However, in general, the amplitude of the target response varies depending on the positional relationship between the speaker 2 and the microphone 3. For this reason, when the acoustic space is a vehicle compartment, the gains of the amplifiers 4 and 5 cannot be set so that the amplitude ratio between the reference signal and the target response becomes an appropriate value unless the vehicle type on which the device is mounted is determined. As described above, conventionally, after the apparatus is mounted on a vehicle, the gains of the amplifiers 4 and 5 are adjusted to adjust the filter coefficient to an appropriate value. However, this adjustment work is troublesome and has a problem that the burden on the user is great.
As described above, an object of the present invention is to set the amplitude ratio of the reference signal and the target response so that the filter coefficient automatically becomes an appropriate value.
[0005]
[Means for Solving the Problems]
According to the present invention, an audio source that outputs an audio signal, a speaker that receives an audio signal and radiates audio sound to the acoustic space, a microphone that detects audio sound radiated to the acoustic space, and the audio signal The target signal is the signal and the signal output from the microphone, adaptive signal processing is performed so that the power of the error signal is minimized, the coefficient of the adaptive filter is updated, and the transfer characteristic from the speaker to the microphone is simulated by the adaptive filter An adaptive signal processing unit that is provided between the audio source and the adaptive signal processing unit, a variable gain amplifier that amplifies the audio signal, and a maximum value of the adaptive filter coefficient so that transfer characteristics from the speaker to the microphone can be correctly simulated. Set, monitor the coefficients of the adaptive filter, and Maximum coefficient value of filter is achieved by an audio sound canceling device having a gain setting section for controlling the gain of the amplifier so that the set value.
[0006]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 is a block diagram of an audio sound canceling apparatus according to the present invention.
In the figure, 11 is an audio source, 12 is an audio signal input, a speaker that radiates audio sound into the acoustic space, 13 is a microphone that detects audio sound radiated into the acoustic space, and 14 is the power of the error signal e is minimum. An adaptive signal processing unit that performs adaptive signal processing so that the coefficient W of the adaptive filter is updated, 15 is a gain-fixed amplifier provided between the audio source 11 and the speaker 12, and 16 is the audio source 11 and the adaptive signal. A variable gain amplifier provided between the processing units 14, a subtractor 17 that calculates a difference between a microphone output as a target signal and an adaptive filter output and outputs an error signal e, and 18 a gain of the variable gain amplifier 16. It is a gain setting part to set.
[0007]
Although not shown, the adaptive signal processing unit 14 includes an LMS calculation unit and an adaptive filter having an FIR type digital filter configuration. The adaptive signal processing unit 14 uses the audio signal as the reference signal and the signal output from the microphone as the target signal, and updates the adaptive filter coefficient W so that the power of the error signal e is minimized.
[0008]
The gain setting unit 18 sets the gain of the amplifier 16 so that the adaptive filter coefficient W becomes an appropriate value. The absolute value of the filter coefficient W can take a value from 0 to 1, and the larger the coefficient coefficient, the higher the accuracy of the coefficient. However, when the coefficient becomes larger than 1 in the calculation of adaptive signal processing, it is clamped to 1, and the adaptive filter cannot accurately simulate the propagation characteristic from the speaker to the microphone. On the other hand, if the filter coefficient is too small, the accuracy of the coefficient is lowered, and similarly, the adaptive filter cannot accurately simulate the propagation characteristic from the speaker to the microphone.
Therefore, the gain setting unit 18 monitors the coefficient W of the adaptive filter and controls the gain of the amplifier 16 so that the maximum value of the filter coefficient becomes a set value, for example, 0.95. That is, the gain is controlled so that the filter coefficient increases if the maximum value of the filter coefficient W is smaller than 0.95, and the gain is controlled so that the filter coefficient decreases if the maximum value of the filter coefficient is larger than 0.95.
[0009]
When the reference signal is smaller than the target signal (reference signal <target signal), the filter coefficient W is increased by adaptive signal processing, and the reference signal = target signal. On the contrary, if the reference signal is larger than the target signal (reference signal> target signal), the filter coefficient W is decreased by the adaptive signal processing, and the reference signal = target signal. Therefore,
(1) The filter coefficient is smaller than 0.95, and in order to increase the filter coefficient, the gain of the amplifier 16 may be decreased so that the reference signal <target signal,
(2) In order to make the filter coefficient larger than 0.95 and reduce the filter coefficient, the gain of the amplifier 16 may be increased so that the reference signal> the target signal.
The gain setting unit 18 controls the gain of the amplifier 16 in accordance with the above (1) and (2), so that the maximum value of the filter coefficient becomes a set value, for example, 0.95.
[0010]
FIG. 2 is a processing flow of gain setting control of the present invention.
The adaptive signal processing unit 14 starts updating the filter coefficient W by adaptive signal processing (start learning, step 101), and stops updating the filter coefficient when the filter coefficient W converges to a certain value (learning stop, step 101). 102). The gain setting unit 18 reads the filter coefficient (step 103), obtains the maximum filter coefficient value, and checks whether the maximum filter coefficient is equal to the set value (= 0.95) (step 104). Exit. If the maximum filter coefficient is larger than the set value (= 0.95), the gain of the amplifier 16 is increased. If the maximum filter coefficient is smaller than the set value (= 0.95), the gain of the amplifier 16 is decreased (step 105). Thereafter, the process returns to step 101, the update of the filter coefficient is resumed, and the above processing is repeated until the maximum filter coefficient = the set value (0.95).
[0011]
In the above, the set value is 0.95, but the present invention is not limited to this value. Further, an appropriate range of filter coefficients, for example, 0.90 to 0.95 can be set, and the gain can be controlled so as to fall within this range.
The present invention has been described with reference to the embodiments. However, the present invention can be variously modified in accordance with the gist of the present invention described in the claims, and the present invention does not exclude these.
[0012]
【The invention's effect】
As described above, according to the present invention, since the accuracy of the filter coefficient can be increased, the performance of the system can be sufficiently exhibited. In addition, since these settings are performed automatically, it is possible to significantly reduce the load on the user when performing these settings compared to the conventional case.
[Brief description of the drawings]
FIG. 1 is a configuration diagram of an audio sound canceling apparatus according to the present invention.
FIG. 2 is a gain setting process flow of the present invention.
FIG. 3 is a configuration diagram of a conventional audio sound canceling apparatus.
[Explanation of symbols]
11. Audio source 12 Speaker 13 Microphone 14 Adaptive signal processing unit 15 Gain fixed amplifier 16 Gain variable amplifier 17 Subtractor 18 Gain setting unit

Claims (1)

オーディオ音をキャンセルする装置において、
オーディオ信号を出力するオーディオソース、
オーディオ信号を入力され、オーディオ音を音響空間に放射するスピーカ、
音響空間に放射されたオーディオ音を検出するマイクロホン、
オーディオ信号を参照信号、マイクロホンから出力する信号を目標信号とし、エラー信号のパワーが最小となるように適応信号処理を行い、適応フィルタの係数を更新して該適応フィルタにより前記スピーカからマイクロホンまでの伝達特性を模擬させる適応信号処理部、
オーディオソースと適応信号処理部間に設けられ、オーディオ信号を増幅するゲイン可変の増幅器、
前記スピーカからマイクロホンまでの伝達特性を正しく模擬できるように前記適応フィルタ係数の最大値を設定し、該適応フィルタの係数を監視し、該適応フィルタの最大係数値が前記設定値となるように前記増幅器のゲインを制御するゲイン設定部、
を有することを特徴とするオーディオ音キャンセル装置。
In a device that cancels audio sound,
An audio source that outputs audio signals,
A speaker that receives an audio signal and radiates an audio sound into an acoustic space;
A microphone that detects audio sound radiated into the acoustic space,
The audio signal is the reference signal, the signal output from the microphone is the target signal, adaptive signal processing is performed so that the power of the error signal is minimized, the coefficient of the adaptive filter is updated, and the adaptive filter processes the signals from the speaker to the microphone. An adaptive signal processor that simulates the transfer characteristics ;
A variable gain amplifier that is provided between the audio source and the adaptive signal processing unit and amplifies the audio signal,
The maximum value of the adaptive filter coefficient is set so that the transfer characteristic from the speaker to the microphone can be correctly simulated, the coefficient of the adaptive filter is monitored, and the maximum coefficient value of the adaptive filter becomes the set value. A gain setting unit for controlling the gain of the amplifier;
An audio sound canceling device comprising:
JP18306199A 1999-06-29 1999-06-29 Audio sound canceling device Expired - Fee Related JP3732681B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP18306199A JP3732681B2 (en) 1999-06-29 1999-06-29 Audio sound canceling device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP18306199A JP3732681B2 (en) 1999-06-29 1999-06-29 Audio sound canceling device

Publications (2)

Publication Number Publication Date
JP2001016678A JP2001016678A (en) 2001-01-19
JP3732681B2 true JP3732681B2 (en) 2006-01-05

Family

ID=16129069

Family Applications (1)

Application Number Title Priority Date Filing Date
JP18306199A Expired - Fee Related JP3732681B2 (en) 1999-06-29 1999-06-29 Audio sound canceling device

Country Status (1)

Country Link
JP (1) JP3732681B2 (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5868216B2 (en) * 2012-02-27 2016-02-24 三菱電機株式会社 Clustering apparatus and clustering program
CN106023998B (en) * 2016-05-27 2019-12-13 北京视觉世界科技有限公司 Camera audio input device, denoising method and camera

Also Published As

Publication number Publication date
JP2001016678A (en) 2001-01-19

Similar Documents

Publication Publication Date Title
US10957301B2 (en) Headset with active noise cancellation
EP0694197B1 (en) Improved audio reproduction system
US8296136B2 (en) Dynamic controller for improving speech intelligibility
GB2465064A (en) Active noise cancellation system with split digital filter
US20080298615A1 (en) Hearing aid and method of utilizing gain limitation in a hearing aid
JP3219770B2 (en) Silencer
JPH07325588A (en) Muffler
JPH08223089A (en) Method and equipment of echo cancellation for all dual connection
US20210193103A1 (en) Feedfoward active noise control
WO2009081184A1 (en) Noise cancellation system and method with adjustment of high pass filter cut-off frequency
JP3732681B2 (en) Audio sound canceling device
WO2021016001A1 (en) Input signal-based frequency domain adaptive filter stability control
JP2001005463A (en) Acoustic system
JP7461815B2 (en) Voice Concealment System
EP0639962B1 (en) Hands free noise canceling headset
JPH086573A (en) Active noise control device
JP3002049B2 (en) Silencer
US20240015436A1 (en) Audio device with anc and hear-through
US11462205B2 (en) In-vehicle communication support system
US11462203B2 (en) In-vehicle communication support system
JP3358463B2 (en) Loudspeaker
JP4235062B2 (en) Output sound processing system
JPH10198386A (en) Sound regenerating device
JP3523401B2 (en) Electronic silencer
JPH06284491A (en) Adaptive noise reduction device

Legal Events

Date Code Title Description
A977 Report on retrieval

Free format text: JAPANESE INTERMEDIATE CODE: A971007

Effective date: 20050606

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20050726

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20050909

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20051011

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20051013

R150 Certificate of patent or registration of utility model

Free format text: JAPANESE INTERMEDIATE CODE: R150

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20091021

Year of fee payment: 4

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20091021

Year of fee payment: 4

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20101021

Year of fee payment: 5

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20101021

Year of fee payment: 5

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20111021

Year of fee payment: 6

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20111021

Year of fee payment: 6

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20121021

Year of fee payment: 7

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20121021

Year of fee payment: 7

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20131021

Year of fee payment: 8

LAPS Cancellation because of no payment of annual fees