EP3149727B1 - Method for forming the excitation signal for a glottal pulse model based parametric speech synthesis system - Google Patents

Method for forming the excitation signal for a glottal pulse model based parametric speech synthesis system Download PDF

Info

Publication number
EP3149727B1
EP3149727B1 EP14893138.9A EP14893138A EP3149727B1 EP 3149727 B1 EP3149727 B1 EP 3149727B1 EP 14893138 A EP14893138 A EP 14893138A EP 3149727 B1 EP3149727 B1 EP 3149727B1
Authority
EP
European Patent Office
Prior art keywords
glottal
glottal pulse
database
pulse
pulses
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP14893138.9A
Other languages
German (de)
English (en)
French (fr)
Other versions
EP3149727A4 (en
EP3149727A1 (en
Inventor
Rajesh DACHIRAJU
Aravind GANAPATHIRAJU
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Interactive Intelligence Group Inc
Original Assignee
Interactive Intelligence Group Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Interactive Intelligence Group Inc filed Critical Interactive Intelligence Group Inc
Publication of EP3149727A1 publication Critical patent/EP3149727A1/en
Publication of EP3149727A4 publication Critical patent/EP3149727A4/en
Application granted granted Critical
Publication of EP3149727B1 publication Critical patent/EP3149727B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/90—Pitch determination of speech signals
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L13/00—Speech synthesis; Text to speech systems
    • G10L13/02—Methods for producing synthetic speech; Speech synthesisers

Definitions

  • the present invention generally relates to telecommunications systems and methods, as well as speech synthesis. More particularly, the present invention pertains to the formation of the excitation signal in a Hidden Markov Model based statistical parametric speech synthesis system.
  • Tuomo Raitio et al. present a study of the performance of glottal flow signal based excitation methods in statistical parametric speech synthesis in their contribution " Comparing Glottal-Flow-Excited Statistical Parametric Speech Synthesis Methods", 2103 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vancouver, BC, 26-31 May 2013, Institute of Electrical and Electronics Engineers, Piscataway, NJ, US, 26 May 2013, pages 7830-7834 .
  • Tamas Gabor Csapo and Geza Nemeth disclose an excitation model for the use in speech synthesis in " A Novel Codebook-Based Excitation Model for us in Speech Synthesis", Cognitive Info Communications (COGINFOCOM), 2012 IEEE 3rd International Conference on, IEEE, 2 December 2012, pages 661 to 665 . None of those contributions to the prior art discloses a method to form parametric models including a combination of steps of clustering a glottal pulse database and forming a corresponding vector database.
  • a method is presented to form parametric models, comprising the steps of: computing a glottal pulse distance metric between a number of glottal pulses; clustering a glottal pulse database into a number of clusters to determine centroid glottal pulses; forming a corresponding vector database by associating a vector with each glottal pulse in the glottal pulse database, wherein the vector associated with each glottal pulse is defined based on the glottal pulse, the centroid glottal pulses and the distance metric; and forming parametric models by associating a glottal pulse from the glottal pulse database to each determined Eigenvector.
  • the excitation, in unvoiced regions, is modeled as white noise. In voiced regions, the excitation is not actually impulse sequences.
  • the excitation is instead a sequence of voice source pulses which occur due to vibration of the vocal folds.
  • the pulses' shapes may vary depending on various factors such as the speaker, the mood of the speaker, the linguistic context, emotions, etc.
  • Source pulses have been treated mathematically as vectors by length normalization (through resampling) and impulse alignment, as described in European Patent EP 2242045 (granted June 27, 2012, inventors Thomas Drugman, et al .)
  • the final length of normalized source pulse signal is resampled to meet the target pitch.
  • the source pulse is not chosen from a database, but obtained over a series of calculations which compromise the pulse characteristics in the frequency domain.
  • the approximate excitation signal used for creating a pulse database does not capture low frequency source content as there is no pre-filtering done while determining the Linear Prediction (LP) coefficients, which are used for inverse filtering.
  • LP Linear Prediction
  • speech unit signals are represented by a set of parameters which can be used to synthesize speech.
  • the parameters may be learned by statistical models, such as HMMs, for example.
  • speech may be represented as a source-filter model, wherein source/excitation is a signal which when passed through an appropriate filter produces a given sound.
  • Figure 1 is a diagram illustrating an embodiment of a Hidden Markov Model (HMM) based Text to Speech (TTS) system.
  • HMM Hidden Markov Model
  • TTS Text to Speech
  • the Speech Database 105 may contain an amount of speech data for use in speech synthesis.
  • a speech signal 106 is converted into parameters.
  • the parameters may be comprised of excitation parameters and spectral parameters.
  • Excitation Parameter Extraction 110 and Spectral Parameter Extraction 115 occurs from the speech signal 106 which travels from the Speech Database 105.
  • a Hidden Markov Model 120 may be trained using these extracted parameters and the Labels 107 from the Speech Database 105. Any number of HMM models may result from the training and these context dependent HMMs are stored in a database 125.
  • the synthesis phase begins as the context dependent HMMs 125 are used to generate parameters 140.
  • the parameter generation 140 may utilize input from a corpus of text 130 from which speech is to be synthesized from.
  • the text 130 may undergo analysis 135 and the extracted labels 136 are used in the generation of parameters 140.
  • excitation and spectral parameters may be generated in 140.
  • the excitation parameters may be used to generate the excitation signal 145, which is input, along with the spectral parameters, into a synthesis filter 150.
  • Filter parameters are generally Mel frequency cepstral coefficients (MFCC) and are often modeled by a statistical time series by using HMMs.
  • MFCC Mel frequency cepstral coefficients
  • the predicted values of the filter and the fundamental frequency as time series values may be used to synthesize the filter by creating an excitation signal from the fundamental frequency values and the MFCC values used to form the filter.
  • Synthesized speech 155 is produced when the excitation signal passes through the filter.
  • the formation of the excitation signal 145 is integral to the quality of the output, or synthesized, speech 155. Low frequency information of the excitation is not captured. It will thus be appreciated that an approach is needed to capture the low frequency source content of the excitation signal and to improve the quality of synthetic speech.
  • Figure 2 is a graphical illustration of an embodiment of the signal regions of a speech segment, indicated generally at 200.
  • the signal has been broken down into segments based on fundamental frequency values for categories such as voiced, unvoiced, and pause segments.
  • the vertical axis 205 illustrates fundamental frequency in Hertz ( Hz ) while the horizontal axis 210 represents the passage of milliseconds ( ms ).
  • the time series, F 0 , 215 represents the fundamental frequency.
  • the voiced region, 220 can be seen as a series of peaks and may be referred to as a non-zero segment.
  • the non-zero segments 220 may be concatenated to form an excitation signal for the entire speech, as described in further detail below.
  • the unvoiced region 225 is seen as having no peaks in the graphical illustration 200 and may be referred to as zero segments.
  • the zero segments may represent a pause or an unvoiced segment given by the phone labels.
  • Figure 3 is a diagram illustrating an embodiment of excitation signal creation indicated generally at 300.
  • Figure 3 illustrates the creation of the excitation signal for both unvoiced and pause segments.
  • the fundamental frequency time series values, represented as F 0 represent signal regions 305 that are broken down into voiced, unvoiced, and pause segments based on the F 0 values.
  • An excitation signal 320 is created for unvoiced and pause segments. Where pauses occur, zeroes (0) are placed in the excitation signal. In unvoiced regions, white noise of appropriate energy (in one embodiment, this may be determined empirically by listening tests) is used as the excitation signal.
  • the signal regions, 305, along with the Glottal Pulse 310 are used for excitation generation 315 and subsequent generation of the excitation signal 320.
  • the Glottal Pulse 310 comprises an Eigen glottal pulse that has been identified from the glottal pulse database, the creation of which is described in further detail in Figure 8 below.
  • FIG 4 is a diagram illustrating an embodiment of excitation signal creation for a voiced segment, indicated generally at 400. It is assumed that a Eigen glottal pulse has been identified from the glottal pulse database (described in further detail in Figure 7 below).
  • the signal region 405 comprises F 0 values, which may be predicted by models, from the voiced segment.
  • f s represents the sampling frequency of the signal.
  • the value of 5/1000 represents the interval of 5 ms durations that the F 0 values are determined for. It should be noted that any interval of a designated duration of a unit time may be used.
  • Another array, designated as F 0 ′ n is obtained by linearly interpolating the F 0 array.
  • glottal boundaries are created, 410, which mark the pitch boundaries of the excitation signal of the voiced segments in the signal region 405.
  • the glottal pulse 415 is used along with the identified glottal boundaries 410 in the overlap adding 420 of a glottal pulse beginning at each glottal boundary.
  • the excitation signal 425 is then created through the process of "stitching", or segment joining, to avoid boundary effects which are further described in Figures 5 and 6 .
  • Figure 5 is a diagram illustrating an embodiment of overlap boundaries, indicated generally at 500.
  • the illustration 500 represents a series of glottal pulses 515 and overlapping glottal pulses 520 in the segment.
  • the vertical axis 505 represents the amplitude of excitation.
  • the horizontal axis 510 may represent the frame number.
  • FIG 6 is a diagram illustrating an embodiment of excitation signal creation for a voiced segment, indicated generally at 600.
  • Switching may be used to form the final excitation signal of voiced segments (from Figure 4 ), which is ideally devoid of boundary effects.
  • any number of different excitation signals may have been formed through the overlap add method illustrated in Figure 4 and in the diagram 500 ( Figure 5 ).
  • the different excitation signals may have a constantly increasing amount of shifts in glottal boundaries 605 and an equal amount of circular left shift 630 for the glottal pulse signal.
  • the glottal pulse signal 615 is of a length less than the corresponding pitch period, then the glottal pulse may be zero extended 625 to the length of the pitch period before circular left shifting 630 is performed.
  • w is generally taken as 1 msec or, in terms of samples, f s 1000 .
  • f s 16,000
  • w 16 for example.
  • the highest pitch period present in the given voice segment is represented as m ⁇ w .
  • Glottal pulses are created and associated with each pitch boundary array P m .
  • the glottal pulses 620 may be obtained from the glottal pulse signal of some length N by first zero extending it to the pitch period and then circularly left shifting it by m ⁇ w samples.
  • an excitation signal 635 is formed by initializing the glottal pulses to zero (0).
  • the formed signal is as a single stitched excitation, corresponding to the shift, m .
  • the arithmetic mean of all of the single stitched excitation signals is then computed 640, which represents the final excitation signal for the voiced segment 645.
  • FIG 7 is a diagram illustrating an embodiment of glottal pulse identification, indicated generally at 700.
  • any two given glottal pulses may be used to compute the distance metric/dissimilarity between them. These are taken from the glottal pulse database 840 created in process 800 (further described in Figure 8 below).
  • the computation may be performed by decomposing the two given glottal pulses x i , y i into sub-band components x i 1 , x i 2 , x i 3 and y i 1 , y i 2 , y i 3 .
  • the given glottal pulse may be transformed into the frequency domain by using a method such as Discrete Cosine Transform (DCT), for example.
  • DCT Discrete Cosine Transform
  • the frequency band may be split into a number of bands, which are demodulated and converted into time domain. In this example, three bands are used for illustrative purposes.
  • the sub-band distance metric is then computed between corresponding sub-band components of each glottal pulses, denoted as d s x i 1 y i 1 .
  • the sub-band metric which may be represented as d s ( f,g ), where d s represents the distance between the two sub-band components f and g , may be computed as described in the following paragraphs.
  • the Discrete Hilbert Transform of normalized circular cross correlation is computed and denoted as R f , g h n .
  • the clustering iterations are terminated when there is no shift in any of the centroids of the k clusters.
  • a vector a set of N real numbers, for example 256, is associated with every glottal pulse 710 in the glottal pulse database 840 to form a corresponding vector database 715.
  • step 720 Principal Component Analysis (PCA) is performed to compute Eigenvectors of the vector database 715.
  • PCA Principal Component Analysis
  • any one Eigenvector may be chosen 725.
  • the closest matching vector 730 to the chosen Eigenvector from the vector database 715 is then determined in the sense of Euclidean distance.
  • the glottal pulse from the pulse database 840 which corresponds to the closest matching vector 730 is regarded as the resulting Eigen glottal pulse 735 associated with an Eigenvector.
  • FIG. 8 is a diagram illustrating an embodiment of glottal pulse database creation indicated generally at 800.
  • a speech signal, 805, undergoes pre-filtering, such as pre-emphasis 810.
  • Linear Prediction (LP) Analysis, 815 is performed using the pre-filtered signal to obtain the LP coefficients.
  • LP Linear Prediction
  • Low frequency information of the excitation may be captured.
  • the coefficients are determined, they are used to inverse filter, 820, the original speech signal, 805, which is not pre-filtered, to compute the Integrated Linear Prediction Residual (ILPR) signal 825.
  • the ILPR signal 825 may be used as an approximation to the excitation signal, or voice source signal.
  • the ILPR signal 825 is segmented 835 into glottal pulses using the glottal segment/cycle boundaries that have been determined from the speech signal 805.
  • the segmentation 835 may be performed using the Zero Frequency Filtering Technique (ZFF) technique.
  • ZFF Zero Frequency Filtering Technique
  • the resulting glottal pulses may then be energy normalized. All of the glottal pulses for the entire speech training data are combined in order to form the glottal pulse database 840.

Landscapes

  • Engineering & Computer Science (AREA)
  • Acoustics & Sound (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)
  • Circuit For Audible Band Transducer (AREA)
  • Measuring Pulse, Heart Rate, Blood Pressure Or Blood Flow (AREA)
  • Mobile Radio Communication Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
EP14893138.9A 2014-05-28 2014-05-28 Method for forming the excitation signal for a glottal pulse model based parametric speech synthesis system Active EP3149727B1 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2014/039722 WO2015183254A1 (en) 2014-05-28 2014-05-28 Method for forming the excitation signal for a glottal pulse model based parametric speech synthesis system

Publications (3)

Publication Number Publication Date
EP3149727A1 EP3149727A1 (en) 2017-04-05
EP3149727A4 EP3149727A4 (en) 2018-01-24
EP3149727B1 true EP3149727B1 (en) 2021-01-27

Family

ID=54699420

Family Applications (1)

Application Number Title Priority Date Filing Date
EP14893138.9A Active EP3149727B1 (en) 2014-05-28 2014-05-28 Method for forming the excitation signal for a glottal pulse model based parametric speech synthesis system

Country Status (8)

Country Link
EP (1) EP3149727B1 (zh)
JP (1) JP6449331B2 (zh)
AU (2) AU2014395554B2 (zh)
BR (1) BR112016027537B1 (zh)
CA (2) CA2947957C (zh)
NZ (1) NZ725925A (zh)
WO (1) WO2015183254A1 (zh)
ZA (1) ZA201607696B (zh)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10255903B2 (en) 2014-05-28 2019-04-09 Interactive Intelligence Group, Inc. Method for forming the excitation signal for a glottal pulse model based parametric speech synthesis system
US10014007B2 (en) 2014-05-28 2018-07-03 Interactive Intelligence, Inc. Method for forming the excitation signal for a glottal pulse model based parametric speech synthesis system
CA3030133C (en) * 2016-06-02 2022-08-09 Genesys Telecommunications Laboratories, Inc. Technologies for authenticating a speaker using voice biometrics
JP2018040838A (ja) * 2016-09-05 2018-03-15 国立研究開発法人情報通信研究機構 音声のイントネーション構造を抽出する方法及びそのためのコンピュータプログラム
CN114913844A (zh) * 2022-04-11 2022-08-16 昆明理工大学 一种基音归一化重构的广播语种识别方法

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5400434A (en) * 1990-09-04 1995-03-21 Matsushita Electric Industrial Co., Ltd. Voice source for synthetic speech system
US6795807B1 (en) * 1999-08-17 2004-09-21 David R. Baraff Method and means for creating prosody in speech regeneration for laryngectomees
JP2002244689A (ja) * 2001-02-22 2002-08-30 Rikogaku Shinkokai 平均声の合成方法及び平均声からの任意話者音声の合成方法
CA2724753A1 (en) * 2008-05-30 2009-12-03 Nokia Corporation Method, apparatus and computer program product for providing improved speech synthesis
JP5075865B2 (ja) * 2009-03-25 2012-11-21 株式会社東芝 音声処理装置、方法、及びプログラム
PL2242045T3 (pl) * 2009-04-16 2013-02-28 Univ Mons Sposób kodowania i syntezy mowy
JP5085700B2 (ja) * 2010-08-30 2012-11-28 株式会社東芝 音声合成装置、音声合成方法およびプログラム
US8744854B1 (en) * 2012-09-24 2014-06-03 Chengjun Julian Chen System and method for voice transformation

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Also Published As

Publication number Publication date
NZ725925A (en) 2020-04-24
ZA201607696B (en) 2019-03-27
JP6449331B2 (ja) 2019-01-09
AU2014395554A1 (en) 2016-11-24
WO2015183254A1 (en) 2015-12-03
AU2020227065A1 (en) 2020-09-24
EP3149727A4 (en) 2018-01-24
CA2947957A1 (en) 2015-12-03
CA2947957C (en) 2023-01-03
AU2020227065B2 (en) 2021-11-18
BR112016027537A2 (zh) 2017-08-15
CA3178027A1 (en) 2015-12-03
EP3149727A1 (en) 2017-04-05
BR112016027537B1 (pt) 2022-05-10
AU2014395554B2 (en) 2020-09-24
JP2017520016A (ja) 2017-07-20

Similar Documents

Publication Publication Date Title
US10621969B2 (en) Method for forming the excitation signal for a glottal pulse model based parametric speech synthesis system
AU2020227065B2 (en) Method for forming the excitation signal for a glottal pulse model based parametric speech synthesis system
US10014007B2 (en) Method for forming the excitation signal for a glottal pulse model based parametric speech synthesis system
Le Cornu et al. Generating intelligible audio speech from visual speech
EP2337006A1 (en) Speech processing and learning
CN114694632A (zh) 语音处理装置
US9466285B2 (en) Speech processing system
CN108369803B (zh) 用于形成基于声门脉冲模型的参数语音合成系统的激励信号的方法
JP2017520016A5 (ja) パラメトリック音声合成システムに基づく声門パルスモデルの励起信号形成方法
Sung et al. Excitation modeling based on waveform interpolation for HMM-based speech synthesis.
US10446133B2 (en) Multi-stream spectral representation for statistical parametric speech synthesis
Niu et al. On the Audio-visual Synchronization for Lip-to-Speech Synthesis
Tantisatirapong et al. Comparison of feature extraction for accent dependent Thai speech recognition system
Prasad et al. Identification of F1 and F2 in Speech Using Modified Zero Frequency Filtering.
Vasudev et al. Speaker identification using FBCC in Malayalam language
Narendra et al. Excitation modeling for HMM-based speech synthesis based on principal component analysis
Boonkla et al. Robust Speech Analysis Based on Source-Filter Model Using Multivariate Empirical Mode Decomposition in Noisy Environments
Tran Noise-robust voice conversion
Toth Using articulatory position data to improve voice transformation
Khorram et al. Context-dependent deterministic plus stochastic model
Hu et al. Speech recognition using non-linear trajectories in a formant-based articulatory layer of a multiple-level segmental HMM.
Phu A Study on Efficient Algorithms for Temporal Decomposition of Speech
Goswami Formant estimation for noise robust vowel recognition based on spectral domain ramp cepstrum model
KR20030075328A (ko) 화자간 변별력 향상을 위하여 개인별 켑스트럼 가중치를적용한 화자 인증 장치 및 그 방법

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20161213

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAX Request for extension of the european patent (deleted)
REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602014074670

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G10L0013000000

Ipc: G10L0013020000

A4 Supplementary search report drawn up and despatched

Effective date: 20180103

RIC1 Information provided on ipc code assigned before grant

Ipc: G10L 13/02 20130101AFI20171220BHEP

Ipc: G10L 25/90 20130101ALI20171220BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20200921

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1359074

Country of ref document: AT

Kind code of ref document: T

Effective date: 20210215

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602014074670

Country of ref document: DE

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20210127

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1359074

Country of ref document: AT

Kind code of ref document: T

Effective date: 20210127

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210428

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210527

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210427

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210427

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210527

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602014074670

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

26N No opposition filed

Effective date: 20211028

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210531

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210528

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210531

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20210531

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210527

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210531

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20140528

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230510

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20230519

Year of fee payment: 10

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: IE

Payment date: 20240509

Year of fee payment: 11

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20240510

Year of fee payment: 11

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20240522

Year of fee payment: 11

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210127