EP2129051B1 - A method and system for realizing the voice compensation in the mobile communication network - Google Patents

A method and system for realizing the voice compensation in the mobile communication network Download PDF

Info

Publication number
EP2129051B1
EP2129051B1 EP07702031.1A EP07702031A EP2129051B1 EP 2129051 B1 EP2129051 B1 EP 2129051B1 EP 07702031 A EP07702031 A EP 07702031A EP 2129051 B1 EP2129051 B1 EP 2129051B1
Authority
EP
European Patent Office
Prior art keywords
frame
voice
invalid
frames
network side
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP07702031.1A
Other languages
German (de)
English (en)
French (fr)
Other versions
EP2129051A1 (en
EP2129051A4 (en
Inventor
Donghua Lu
Wei Ruan
Jian Cao
Hongwei Lou
Wanchun Zhang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
ZTE Corp
Original Assignee
ZTE Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ZTE Corp filed Critical ZTE Corp
Publication of EP2129051A1 publication Critical patent/EP2129051A1/en
Publication of EP2129051A4 publication Critical patent/EP2129051A4/en
Application granted granted Critical
Publication of EP2129051B1 publication Critical patent/EP2129051B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/005Correction of errors induced by the transmission channel, if related to the coding algorithm

Definitions

  • the present invention relates to voice compensation technology, and more specifically, to a method and system for voice compensation when the device at network side does not use or partially use a vocoder.
  • vocoder at network side mainly has the following two important roles: in the uplink, after the user equipment (UE) sends the compressively encoded voice to the network side, the vocoder at the network side decodes the received compressed voice to make the voice suitable to be transmitted in the network; in the downlink, the vocoder at the network side compressively encodes the voice stream transmitted in the network to make the voice suitable to be transmitted in the radio link.
  • UE user equipment
  • CDMA2000 Code Division Multiple Access 2000
  • EVRC Enhanced Variable Rate Coder
  • QCELP-13k Quadrature-Qualcomm Code Excited Linear Predictive Coding-Qualcomm-13k
  • QCELP-8k Quadrature-Qualcomm-8k
  • EVRC is the most-widely used mainstream encoding and decoding format.
  • MS1 Mobile Station
  • MS2 they use the same voice encoding and decoding method (such as EVRC).
  • the voice of the MS1 subscriber arrives at the ears of the MS2 subscriber through the following way: Firstly, MS1 sends the encoded EVRC compressed voice frame to network side 1 through the radio uplink, and network side 1 uses vocoder 1 to decode the received EVRC voice frames and transform them into circuit PCM (Pulse Coded Modulation) code stream, and then perform circuit switching. After network side 2 receives the circuit-switched PCM code stream from network side 1, it uses vocoder 2 to transform the PCM code stream to EVRC compressed voice frames and sends them to MS2 through radio downlink.
  • PCM Pulse Coded Modulation
  • the voice encoding and decoding by the vocoder is lossy compression, each encoding and decoding will decrease the voice quality. Still taking the call between MS1 and MS2 as an example, since MS1 and MS2 use the same encoding and decoding format, two voice encoding and decoding processes can be avoided at the network side if the encoding and decoding of EVRC compressed voice frames added at the network side can be avoided, and the process that the voice of MS1 subscriber arrives at the ears of MS2 subscriber is as follows: firstly, MS1 sends the encoded EVRC compressed voice frames to network side 1 through radio uplink, and network side 1 directly switches the received EVRC voice frames and sends them to network side 2; network side 2 receives the switched EVRC compressed voice frames and sends them to MS2 through radio downlink.
  • Patent application WO 02/054744 A1 discloses a network element and a method for enhancing the quality of digitised analogue signals transmitted in parameterised coded form via a digital network.
  • the network element comprises means for extracting signals from and insert signals into the network, first processing means for processing the extracted parameters in the parameter domain with functions suitable to enhance the quality of the digitised analogue signals and second processing means for processing the extracted parameters in the linear domain with functions suitable to enhance the quality of the digitised analogue signals.
  • the all-IP mobile communication network has the problem of how to simultaneously support several kinds of vocoders with lower cost, that is, the problem of supporting so-called TrFO (Transcoder Free Operation) and RTO (Remote Transcoder Opearation).
  • TrFO Transcoder Free Operation
  • RTO Remote Transcoder Opearation
  • TrFO means making the network negotiate the encoding and decoding type and the mode of the vocoder before call establishment through some out-band negotiation mechanism. Through negotiation, calls among the mobile subscribers do not need to pass through the vocoder at network side, thus improving the voice quality, saving expensive vocoder resource and the followed power consumption.
  • RTO is a special case of TrFO, and since the encoding and decoding methods of both sides of communication are not consistent during out-band negotiation, it is necessary that there is a vocoder at network side to transform the code type of one side of communication into the code type of the other side.
  • the main difference between RTO and TDM circuit transmission networks is that: in TDM network, there are two encoding and decoding transformations at network side, while only one in RTO network.
  • MS1 uses EVRC while MS2 uses QCELP-13k
  • MS2 uses QCELP-13k
  • the voice of MS1 subscriber arrives at the ears of MS2 subscriber through the following process: firstly, MS1 sends the encoded EVRC compressed voice frames to network side 1 through radio uplink; network side 1 directly switches the received EVRC voice frames and sends them to network side 2; network side 2 receives the switched EVRC compressed voice frames and transforms them into QCELP-13k compressed voice frames by a vocoder, and sends them to MS2 through radio downlink.
  • CDMA2000 LMSD Legacy Mobile Station Domain
  • out-band negotiation of TrFO is implemented through signaling negotiation between access network and MSCe. Since CDMA2000 LMSD applies IP switching technology, network side can directly make the compressed voice data encoded by the UE be RTP (Real-Time Transport Protocol) packets and transfer them through IP network, and there is no need to transform encoding and decoding scheme to PCM and to be transmit through TDM circuit.
  • RTP Real-Time Transport Protocol
  • EVRC as an example, as inter alia disclosed in 3 rd Generation Partnership Project 2, "3GPP2", "Enhanced Variable Rate Codec, Speech Service Option 3 and 68 for Wideband Spread Spectrum Digital Systems", 3GPP2 C.S0014-B, Version 1.0, May 2006, the maximal transmission rate of EVRC is 8kpbs (the transmission rate of full-rate frame), and EVRC also has lots of half-rate frames and 1/8-rate frames.
  • EVRC can be packed to be transmitted in the network to save the overhead of IP header. Taking packing 3 EVRC frames into a RTP message for example, the average transmission rate of EVRC in the nework is 11.7kbps with the overhead of IP header.
  • TrFO has some problems in the practical applications. For example, suppose MS1 and MS2 are in a TrFO call, if the quality of radio link is not good enough, there is possibility that network side 1 cannot correctly receive and analyze the content of some frames sent by MS 1 to network side 1 through uplink, that is, frame error in the air.
  • network side 1 only can fill these frame errors with to-be-compensated frames specified in protocol (for example, in EVRC, half-rate or full-rate frames with all bits being "0" are defined as to-be-compensated frames, and so on) and switch them to network side 2 which sends these to-be-compensated frames specified by protocol to MS2.
  • protocol for example, in EVRC, half-rate or full-rate frames with all bits being "0" are defined as to-be-compensated frames, and so on
  • RTO technology also has the same problem. Since RTO call uses vocoder at network side, suppose MS1 and MS2 are in RTO call, some of the frames sent by MS1 to network side 1 through uplink may have errors if the quality of radio link is not good enough, network side 1 can uses the vocoder at network side to perform voice compensation for these frame errors. However, when the compensated voice frames arrives at network side 2, there is still possibility of frame loss and frame jittering due to the problem of network transmission quality, at this time, network side 2 will fill the frame errors with to-be-compensated frames specified in protocol and send them to MS2. Therefore, these to-be-compensated frames will have big influence on the whole voice quality of RTO if voice compensation cannot be performed effectively for these frames by MS2.
  • TrFO and RTO reducing the times of encoding and decoding by the vocoder at network side will improve the voice quality, but they cannot use the vocoder at network side for voice compensation as the original circuit-switched mobile communication system does when the environment of radio link is not good and the quality of network transmission is poor.
  • voice compensation totally relies on the vocoder at the UE side.
  • TrFO and RTO the voice qualities of TrFO and RTO severely rely on the compensation performance of the vocoder in the UE and on whether the vocoder compensates all kinds of to-be-compensated voice frames or not, which will significantly affect the whole voice quality of TrFO and RTO.
  • the present invention offers a method and system for realizing voice compensation in a mobile communication network to solve the above disadvantages and to approximately compensate the voice in the condition that the transmission quality is poor and the device at network side does not use or partially uses vocoder, thus improving the quality of whole voice.
  • a method for realizing voice compensation in mobile communication network as set forth in independent claim 1.
  • the present invention also offers a system for realizing voice compensation in mobile communication network, as set forth in independent claim 3.
  • the system and method of the present invention can effectively solve the problem that the voice quality makes the ears uncomfortable, including discontinuous phenomenon, trilling and swallowing words during the call, due to the bad radio link environment or poor network transmission quality when there is no vocoder at the network side participates or vocoder at the network side partially participates the call.
  • the scheme of the present invention implements voice compensation in the device at the network side to effectively reduce the dependency of the call on the user terminal and the vocoder's performance, thus satisfying all kinds of voice quality requirements by the user terminals.
  • the main idea of the present invention is: during the call, full-rate frames and half-rate frames gives the greatest contribution to the voice, if there are full-rate frames or half-rate frames damaged or lost, the voice quality will be easily affected.
  • a lot of experiments show that, the loss or damage of one or more full-rate frames will cause discontinuous voice, swallowing words, especially in the condition of consecutive full-half-rate frames; and the loss or damage of one or more half-rate frames will cause trilling especially in the condition of consecutive full-half-rate frames, thus making the ears uncomfortable, and the degree of uncomfortable feeling relies on the encoding and decoding performance of the vocoder of the UE. Therefore, the objective of the present invention is to mainly compensate full-rate frames or half-rate ones.
  • the present invention offers a method for realizing voice compensation in mobile communication network to apply in the condition that the radio environment is bad or transmission quality is poor, and the network side does not use or partially uses vocoder (such as TrFO or RTO), as shown in FIG. 1 , and the method comprises the following steps:
  • the method for judging whether the invalid frame is a frame in the condition of non-1/8-rate is as follows:
  • valid frame means that the frame can be normally encoded and decoded by the vocoder during the call, that is, those frames other than the invalid frames are called valid frames.
  • Said "last valid frame” means the valid frame received or to be sent at the last time of processing frames; if the frame received or to be sent at the last time of processing frames is an invalid frame, it means the valid frame received or to be sent at the time just before the last one of processing frames, and so on.
  • Step 3 the device at the network side judges whether the frame distance between the invalid frame and its last valid frame is less than or equals to the threshold of voice compensation:
  • Said compensation threshold depends on the compensation effect and performance of the mobile communication system, the compensation threshold to achieve the optimal compensation effect can be obtained according to the voice quality by comparing the results of several experiments; For example, the compensation threshold is set as 6, 6 consecutive invalid frames are compensated; while if the compensation threshold is set as 2, only two invalid frames are compensated, and the third one of the consecutive frames will not be compensated.
  • the frame distance between frame A and frame B means the number of frames between frame A and frame B plus 1 for a group of sequentially arrived frames. For example, for a group of sequentially arrived frames, frame a, frame b, frame c, frame d ... ..., the frame distance between frame a and frame d is 3.
  • Step 4 the device at the network side performs voice frame compensation for the invalid frame, uses the compensated voice frame to replace the invalid frame to become the voice frame to be processed and output at this time.
  • the voice frame compensation methods applied by the device at the network side comprise one of the following: valid frame copying method, 1/4-rate frame filling method, simulation approximating method, and so on.
  • Valid frame copying method the last valid frame is used to replace the current invalid frame
  • 1/4-rate frame filling method this method can only be applied in EVRC encoded and decoded voice call; wherein a 1/4-rate frame with any content is used to replace the current invalid frame to be compensated
  • Simulation approximating method a frame is simulated using the rate and content of the last valid frame and the frame distance between the current invalid frame and its last valid frame according to the discipline obtained from simulation, and the frame obtained from simulation is used to replace the current invalid frame.
  • the compensated voice frame is normally processed and output.
  • the present invention offers a system for realizing voice compensation in mobile communication network, the system is set in the device at the network side and is applied in the condition that the radio environment is bad or transmission quality is poor and the device at the network side does not use or partially use vocoder, as shown in FIG. 2 , and the system comprises:
  • the compensated voice frames are sent to the unit for processing voice frames in the device at the network side.
  • Said device at the network side can be one of the following: a base station, a base station controller, a radio network controller or a mobile switching center.
  • the present invention can be used for voice call in which the device at the network side does not use or partially uses vocoder for voice compensation, including the mobile communication system applying TrFO technology, RTO technology or TFO (Tandem Free Operation) technology.
  • the present invention also can be used in a mobile communication system of CDMA2000, WCDMA (Wideband-Code Division Multiple Access) and TDS-CDMA (Time Division Synchronization- Code Division Multiple Access).
  • the method for voice compensation applied in this embodiment is valid frame copying method, and the threshold of frame distance of voice compensation in this embodiment is 1, that is, voice compensation is performed only for the first invalid frame after the valid frames in the condition of full-rate, while voice compensation is not performed for the continuously arrived invalid frames after the first invalid frame;
  • the specific steps which should be performed are as follows:
  • this embodiment only describes the steps of judging and compensating the forward voice frames from the network side by the device at the network side, it is also applicable for the steps of judging and compensating the backward voice frames from the UE by the device at the network side, which will not be described in detail here.
  • This embodiment can be applied for EVRC encoded and decoded calls, and the legal encoding format of EVRC does not include 1/4-rate frames.
  • a lot of experiments show that, vocoder of each UE will perform voice compensation after receiving EVRC encoded and decoded 1/4-rate frames. As shown in FIG 4 , the specific steps of this embodiment are as follows:
  • the main idea of this embodiment is replacing a group of consecutive frames right after the full-rate frames with 1/4-rate frames; For each invalid frame whose frame distance from its last full-rate valid frame is less than or equals to the predetermined threshold, it is replaced with a 1/4-rate frame; No extra voice compensation is performed for those whose frame distance is greater than the threshold; that is, if the number of consecutive invalid frames right after the full-rate valid frame is greater than the maximal threshold, no extra voice compensation will be performed for those ones which exceed the threshold; said maximal threshold of the number of invalid frames is the compensation threshold.
  • the compensation threshold of this method can be set as infinite, that is, all consecutive invalid frames right after the full-rate frames will be replaced with 1/4-rate frames.
  • this embodiment only describes the steps of judging and compensating the forward voice frames from the network side by the device at the network side, it is also applicable for the steps of judging and compensating the backward voice frames from UE by the device at the network side, which will not be described here.
  • a large amount of full-rate voice data are statistically inducted according to the previous actual situations to obtain the approximation rule of the content of frames and rate change; when the invalid frames are compensated, a frame can be simulated and used to replace the rate and content of the invalid frame according to the approximation rule and by using the content and rate of the last validate frame and the frame distance between the invalid frame and the last valid frame; in this specification, the frames obtained through simulation are called as pseudo-full-rate frames.
  • the predetermined compensation threshold in this embodiment is 6. As shown in FIG. 5 , the specific steps of this embodiment are as follows:
  • the main idea of this embodiment is replacing the consecutive invalid frames right after the full-rate frames with simulated voice frames, in simulation, the 6 consecutive invalid frames right after the full-rate frames can be compensated through statistical discipline according to the content of the full-rate frames and the frame distance between the invalid frame and the full-rate frame.
  • this embodiment only describes the steps of judging and compensating the forward voice frames from the network side by the device at the network side, it is also applicable for the steps of judging and compensating the backward voice frames from the UE by the device at the network side, which will not be described here.
  • the voice quality obtained through simulation approximating method is the best, and moreover, the simulation approximating method can compensate several deleted frames in the condition of consecutive full-rate frames, and its overhead is not heavy, only the content of the latest full-rate frame should be saved.
  • the above three embodiments mainly compensate the frames in the condition of full-rate; in practical applications, it may be supposed that voice compensation will be performed when the last valid frame is a full-rate or half-rate frame.
  • the compensation threshold can be set according to the practical situations.
  • the present invention offers a system and method for realizing voice compensation at the network side to solve the problem that the whole voice quality is poor and makes human ears uncomfortable because vocoder is not used or partially used in the network side for voice quality compensation and linear prediction so as to make the voice quality largely rely on whether the vocoder in the UE compensates some to-be-compensated frames and on the compensation performance.
  • the technical scheme of the present invention can relatively compensate the voice when vocoder is not used or partially used at the network side to reduce the uncomfortable feelings of ears due to swallowing words, trilling and discontinuous voice, to promote the whole voice quality, and to reduce the dependency of calls on the performance of UE and its vocoder.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Mobile Radio Communication Systems (AREA)
  • Telephonic Communication Services (AREA)
EP07702031.1A 2007-01-10 2007-01-10 A method and system for realizing the voice compensation in the mobile communication network Active EP2129051B1 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2007/000099 WO2008083517A1 (fr) 2007-01-10 2007-01-10 Procédé et système permettant de réaliser une compensation vocale dans un réseau de communication mobile

Publications (3)

Publication Number Publication Date
EP2129051A1 EP2129051A1 (en) 2009-12-02
EP2129051A4 EP2129051A4 (en) 2010-07-28
EP2129051B1 true EP2129051B1 (en) 2017-08-09

Family

ID=39608312

Family Applications (1)

Application Number Title Priority Date Filing Date
EP07702031.1A Active EP2129051B1 (en) 2007-01-10 2007-01-10 A method and system for realizing the voice compensation in the mobile communication network

Country Status (3)

Country Link
EP (1) EP2129051B1 (zh)
CN (1) CN101529830B (zh)
WO (1) WO2008083517A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102242260B1 (ko) 2014-10-14 2021-04-20 삼성전자 주식회사 이동 통신 네트워크에서 음성 품질 향상 방법 및 장치
CN107393559B (zh) * 2017-07-14 2021-05-18 深圳永顺智信息科技有限公司 检校语音检测结果的方法及装置

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2531352B2 (ja) * 1993-07-19 1996-09-04 日本電気株式会社 音声パケット受信装置
US7539615B2 (en) * 2000-12-29 2009-05-26 Nokia Siemens Networks Oy Audio signal quality enhancement in a digital network
FI20010235A (fi) * 2001-02-08 2002-08-09 Nokia Corp Menetelmä informaatiokehysten prosessoimiseksi
JP2005223375A (ja) * 2004-02-03 2005-08-18 Elwing Co Ltd データ伝送方法及びその装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Also Published As

Publication number Publication date
EP2129051A1 (en) 2009-12-02
WO2008083517A1 (fr) 2008-07-17
CN101529830B (zh) 2013-01-30
EP2129051A4 (en) 2010-07-28
CN101529830A (zh) 2009-09-09

Similar Documents

Publication Publication Date Title
KR100933159B1 (ko) 이동통신시스템에서 음성 데이터 전송을 위한 동기화 방법 및 시스템
US20050286536A1 (en) Reducing backhaul bandwidth
EP2107818B1 (en) Gsm bearer set up method, apparatus and system
RU2008120004A (ru) Способ и система адаптивного кодирования информации в режиме реального времени в беспроводных сетях
WO1999056456A2 (en) Alternating speech and data transmission in digital communications systems
EP1275261A1 (en) Application of rtp and rtcp in the amr transport in voice over ip networks
US20040179555A1 (en) System and method for compressing data in a communications environment
JP3650800B2 (ja) 異なるネットワークのターミナル間のスピーチ送信
JP4856251B2 (ja) 無線通信ネットワークにおけるヘッダの抑制
CN107886961A (zh) 基于VoLTE承载的自适应多速率中低速话音优化方法
US8209187B2 (en) Speech coding arrangement for communication networks
EP2108193B1 (en) Methods, systems, and computer program products for silence insertion descriptor (sid) conversion
CN1233186C (zh) 用于在移动无线网内传输语音数据的方法和装置
EP2129051B1 (en) A method and system for realizing the voice compensation in the mobile communication network
CN103871415B (zh) 实现异系统间语音互通的方法、系统与tfo转换装置
US8634326B2 (en) Method and node for the control of a connection in a communication network
RU2426250C2 (ru) Способ и система для речевой компенсации в сети мобильной связи
WO2009036693A1 (fr) Procédé et système de traitement de données de liaison montante et descendante dans un réseau de communication sans fil
CN101365189B (zh) 一种移动网中实现语音透传的方法和系统
WO2007118392A1 (fr) Procédé et dispositif de transmission de données vocales
US8380249B2 (en) Transmission method in a mobile communication system
US20050195861A1 (en) Sound communication system and mobile station
WO2009099373A1 (en) A method of transmitting speech
AU4299499A (en) Alternating speech and data transmission in digital communications systems

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20090717

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC NL PL PT RO SE SI SK TR

DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20100630

RIC1 Information provided on ipc code assigned before grant

Ipc: H04L 12/56 20060101ALI20100624BHEP

Ipc: G10L 19/00 20060101AFI20100624BHEP

17Q First examination report despatched

Effective date: 20160726

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602007051912

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: H04L0012560000

Ipc: G10L0019005000

RIC1 Information provided on ipc code assigned before grant

Ipc: G10L 19/005 20130101AFI20170113BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

INTG Intention to grant announced

Effective date: 20170308

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC NL PL PT RO SE SI SK TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

Ref country code: AT

Ref legal event code: REF

Ref document number: 917620

Country of ref document: AT

Kind code of ref document: T

Effective date: 20170815

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602007051912

Country of ref document: DE

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20170809

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 917620

Country of ref document: AT

Kind code of ref document: T

Effective date: 20170809

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 12

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170809

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170809

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170809

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170809

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170809

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171209

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171109

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170809

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170809

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171110

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170809

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170809

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170809

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170809

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602007051912

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170809

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170809

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170809

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20180511

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170809

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180110

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20180131

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180131

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180131

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180131

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180110

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170809

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170809

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170809

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20070110

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170809

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20231130

Year of fee payment: 18

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20231212

Year of fee payment: 18

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20231205

Year of fee payment: 18