EP2755205A1 - Subband-Verarbeitung zur Komplexitätsverringerung - Google Patents

Subband-Verarbeitung zur Komplexitätsverringerung Download PDF

Info

Publication number
EP2755205A1
EP2755205A1 EP14157074.7A EP14157074A EP2755205A1 EP 2755205 A1 EP2755205 A1 EP 2755205A1 EP 14157074 A EP14157074 A EP 14157074A EP 2755205 A1 EP2755205 A1 EP 2755205A1
Authority
EP
European Patent Office
Prior art keywords
band
representative
sub
phase value
value
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP14157074.7A
Other languages
English (en)
French (fr)
Other versions
EP2755205B1 (de
Inventor
Shreyas Paranjpe
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
2236008 Ontario Inc
Original Assignee
QNX Software Systems Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US12/696,533 external-priority patent/US8457976B2/en
Application filed by QNX Software Systems Ltd filed Critical QNX Software Systems Ltd
Publication of EP2755205A1 publication Critical patent/EP2755205A1/de
Application granted granted Critical
Publication of EP2755205B1 publication Critical patent/EP2755205B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/03—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters
    • G10L25/18—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters the extracted parameters being spectral information of each sub-band
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
    • G10L19/0204—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders using subband decomposition
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
    • G10L19/0212—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders using orthogonal transformation
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • G10L2021/02082—Noise filtering the noise being echo, reverberation of the speech

Definitions

  • Wideband networks receive and transmit data through radio frequency signals through inbound and outbound transmissions.
  • the networks may transmit data, voice, and video simultaneously through multiple channels that may be distinguished in frequency.
  • Some wideband networks are capable of high speed operations and may have a considerably higher throughput than some narrowband networks.
  • the increased bandwidth of these networks may increase the processing loads and memory requirements of other applications.
  • Frequency domain based adaptive filtering may be computationally intensive because it translates a time domain signal into multiple frequency components that are separately processed. Translating a time domain signal into multiple frequency components increases the computational complexity and memory usage of some systems when a signal's bandwidth increases. As the number of frequency components increase with bandwidth, the computational load and the required memory increase.
  • a sub-band processing system processes data such that, after it is compressed and decompressed it is restored to its original format.
  • the system may compress video, sound, text, code, and/or numeric data such that little or no data is lost after a bin or file is decompressed. While the data may contain more information than may be heard or seen (e.g., perceived by a user), some systems preserve the original data (or a representative data set) while compressing and decompressing operating data through a lossy compression.
  • the sub-band processing system analysis may occur on frequency domain characteristics.
  • the signal may be broken into intervals though a multiplier function (retained in a local or a distributed computer readable medium) or multiplier device that multiplies the signal by a "window" function or a "frame” of fixed duration.
  • smooth window functions such as Hann, Hamming, etc. retained in the local or the distributed computer readable medium
  • a window filter may be used for the short-time spectral analysis.
  • a time-to-frequency transform device, a Discrete Fourier Transform (DFT) device, or a Fast Fourier Transform (FFT) device may transform (or decompose) the short-time based signals into a complex spectrum.
  • DFT Discrete Fourier Transform
  • FFT Fast Fourier Transform
  • the spectrum may be separated into bins of magnitude and phase data or substantially equivalent complex (e.g. real and imaginary) data.
  • a sub-band (or band) may be represented by a single bin of magnitude and phase spectra, or a collection of consecutive or successive bins represented by a common or single magnitude and phase spectra. Table 1 shows representative characteristics of an exemplary FFT device.
  • the sub-band processing system may reduce M to a lowest possible integer that does not affect the performance or quality of a later process.
  • the system may generate a number of sub-bands that minimize perceptual error.
  • the applications may exploit the sensitivity of the human auditory system or other systems that do not detect or process certain frequencies or are affected by certain signal distortions.
  • Figures 1 and 2 respectively, describe exemplary non-overlapping and band-like overlapping compressions.
  • the uncompressed bins are shown above the corresponding compressed sub-bands.
  • the compressions divide a variable sequence of uncompressed bins into a substantially equal sequence of compressed sub-bands.
  • a substantially equal gain or a variable gain may be applied to render compressed sub-bands that are substantially flat across the frequency spectrum. Perceptual distortions may be minimized by applying lower compression ratios at lower frequencies while applying higher compression ratios at higher frequencies.
  • Table 2 describes an exemplary non-overlapping compression scheme in which each sub-band represents 2 q bins. Approximate freq range (kHz) Input bin numbers Compression ratio Output sub-bands #s 0-1 0..31 1:1 0..31 1-2 32..63 2:1 32..47 2-4 64..127 4:1 48..63 4-Nyquist 128..M 8:1 64..xx
  • Other systems may apply a more perceptually based scheme that partitions the frequency spectrum into non-overlapping regions.
  • the compression may be based on an auditory filter estimate.
  • Each sub-band may be approximately equal to a first predetermined frequency band such as 1 ⁇ 2 ERB (Equivalent Rectangular Bandwidth) for frequencies below about 4 kHz, and a second predetermined frequency band such as 1 ERB for frequencies above about 4 kHz. More aggressive compression schemes may be applied when the level of distortion or artifacts do not affect (or have little affect on) the performance of other systems.
  • a first predetermined frequency band such as 1 ⁇ 2 ERB (Equivalent Rectangular Bandwidth) for frequencies below about 4 kHz
  • 1 ERB Equivalent Rectangular Bandwidth
  • More aggressive compression schemes may be applied when the level of distortion or artifacts do not affect (or have little affect on) the performance of other systems.
  • An average may comprise a logical operation stored in a local or remote central or distributed memory such as an arithmetic mean of the phases within each sub-band.
  • the weights of a weighted average may be based on the phase correlations common to one or all of the bins that comprise one or more sub-bands.
  • the selected magnitudes an average magnitude (e.g., an average of bins that makeup a band), peaks in the magnitude spectrum, or a function or algorithm that selects or synthesizes a magnitude of each sub-band may be designated as representative.
  • an average magnitude e.g., an average of bins that makeup a band
  • peaks in the magnitude spectrum or a function or algorithm that selects or synthesizes a magnitude of each sub-band
  • a function or algorithm that selects or synthesizes a magnitude of each sub-band may be designated as representative.
  • the bin containing that magnitude is indexed, stored in memory, and the magnitude is rotated or shifted (e.g., through a phase shifter) to attain the selected or designated phase.
  • a resulting sub-band value may be transformed to a maximum magnitude selected from its constituent bins and the phase of the "preserved" or selected bin (through a rotation through or shift by a phase differential, e.g., beta sub-band1 , beta sub-band2
  • Figure 4 is an uncompressed spectrum of complex vectors representing bins 4, 5, 6 and 7 that comprise an exemplary sub-band 3.
  • bin 5 has the largest magnitude and is therefore designated as representative (e.g., through a peak magnitude detector).
  • the phase of bin 4 is the designated phase.
  • the vector representing bin 5 is rotated counterclockwise or otherwise adjusted to substantially match the phase of bin 4 while maintaining its original maximum magnitude (as shown in Figure 5 ).
  • the rotated or adjusted version of bin 5 represents sub-band 3, which effectively attenuates the remaining spectrum within the sub-band (e.g., effectively setting the remaining spectrum to substantially to zero) as shown in Figure 6 .
  • the magnitudes and phases of the sparse spectrum may be further processed before the spectrum is reconstructed.
  • the original spectrum (or the representative data set) may be retained in a computer readable medium or memory so that the original relative magnitude and phase relationships may be maintained or restored in the decompressed spectrum. This retention potentially reduces audible artifacts that may be introduced by a compression scheme.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Computational Linguistics (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)
EP14157074.7A 2010-01-29 2011-01-24 Subband-Verarbeitung zur Komplexitätsverringerung Active EP2755205B1 (de)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US12/696,533 US8457976B2 (en) 2009-01-30 2010-01-29 Sub-band processing complexity reduction
EP11151856.9A EP2355094B1 (de) 2010-01-29 2011-01-24 Subband zur Verarbeitung der Komplexitätsverringerung

Related Parent Applications (2)

Application Number Title Priority Date Filing Date
EP11151856.9A Division-Into EP2355094B1 (de) 2010-01-29 2011-01-24 Subband zur Verarbeitung der Komplexitätsverringerung
EP11151856.9A Division EP2355094B1 (de) 2010-01-29 2011-01-24 Subband zur Verarbeitung der Komplexitätsverringerung

Publications (2)

Publication Number Publication Date
EP2755205A1 true EP2755205A1 (de) 2014-07-16
EP2755205B1 EP2755205B1 (de) 2019-12-11

Family

ID=44023086

Family Applications (2)

Application Number Title Priority Date Filing Date
EP11151856.9A Active EP2355094B1 (de) 2010-01-29 2011-01-24 Subband zur Verarbeitung der Komplexitätsverringerung
EP14157074.7A Active EP2755205B1 (de) 2010-01-29 2011-01-24 Subband-Verarbeitung zur Komplexitätsverringerung

Family Applications Before (1)

Application Number Title Priority Date Filing Date
EP11151856.9A Active EP2355094B1 (de) 2010-01-29 2011-01-24 Subband zur Verarbeitung der Komplexitätsverringerung

Country Status (2)

Country Link
EP (2) EP2355094B1 (de)
CA (1) CA2729707C (de)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3764353A1 (de) * 2019-07-09 2021-01-13 BlackBerry Limited Verfahren zur mehrstufigen kompression in der unterbandverarbeitung

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0527374A2 (de) * 1991-08-12 1993-02-17 Alcatel SEL Aktiengesellschaft Codierverfahren für Audiosignale mit 32 kbit/s
US20070016404A1 (en) * 2005-07-15 2007-01-18 Samsung Electronics Co., Ltd. Method and apparatus to extract important spectral component from audio signal and low bit-rate audio signal coding and/or decoding method and apparatus using the same

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100363259B1 (ko) * 2000-05-16 2002-11-30 삼성전자 주식회사 인지 특성 가중 함수를 이용한 음성신호의 위상 양자화장치 및 방법
EP1852848A1 (de) * 2006-05-05 2007-11-07 Deutsche Thomson-Brandt GmbH Verfahren und Vorrichtung für verlustfreie Kodierung eines Quellensignals unter Verwendung eines verlustbehafteten kodierten Datenstroms und eines verlustfreien Erweiterungsdatenstroms

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0527374A2 (de) * 1991-08-12 1993-02-17 Alcatel SEL Aktiengesellschaft Codierverfahren für Audiosignale mit 32 kbit/s
US20070016404A1 (en) * 2005-07-15 2007-01-18 Samsung Electronics Co., Ltd. Method and apparatus to extract important spectral component from audio signal and low bit-rate audio signal coding and/or decoding method and apparatus using the same

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
ANDERSON DAVID V: "SPEECH ANALYSIS AND CODING USING A MULTI-RESOLUTION SINUSOIDAL TRANSFORM", CONFERENCE PROCEEDINGS / THE 1996 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, MAY 7 - 10, 1996, MARRIOTT MARQUIS HOTEL, ATLANTA, GEORGIA, USA, IEEE SERVICE CENTER, PISCATAWAY, NJ, 7 May 1996 (1996-05-07), pages 1037 - 1040, XP002662475, ISBN: 978-0-7803-3192-1 *
MIKSIC A ET AL: "SUBBAND ECHO CANCELLATION IN AUTOMATIC SPEECH DIALOG SYSTEMS", 5TH EUROPEAN CONFERENCE ON SPEECH COMMUNICATION AND TECHNOLOGY. EUROSPEECH '97. RHODES, GREECE, SEPT. 22 - 25, 1997; [EUROPEAN CONFERENCE ON SPEECH COMMUNICATION AND TECHNOLOGY. (EUROSPEECH)], GRENOBLE : ESCA, FR, 22 September 1997 (1997-09-22), pages 2579 - 2582, XP001045216 *
PARANJPE SHREYAS ET AL: "Acoustic Echo Cancellation for Wideband Audio and Beyond", AES CONVENTION 128; MAY 2010, AES, 60 EAST 42ND STREET, ROOM 2520 NEW YORK 10165-2520, USA, 1 May 2010 (2010-05-01), XP040509430 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3764353A1 (de) * 2019-07-09 2021-01-13 BlackBerry Limited Verfahren zur mehrstufigen kompression in der unterbandverarbeitung
US10984808B2 (en) 2019-07-09 2021-04-20 Blackberry Limited Method for multi-stage compression in sub-band processing

Also Published As

Publication number Publication date
CA2729707C (en) 2014-04-01
EP2355094B1 (de) 2017-04-12
EP2755205B1 (de) 2019-12-11
CA2729707A1 (en) 2011-07-29
EP2355094A1 (de) 2011-08-10

Similar Documents

Publication Publication Date Title
US9225318B2 (en) Sub-band processing complexity reduction
US10529347B2 (en) Methods, apparatus and systems for determining reconstructed audio signal
KR100913987B1 (ko) 다중-채널 출력 신호를 발생시키기 위한 다중-채널합성장치 및 방법
US8249861B2 (en) High frequency compression integration
CA2550654C (en) Frequency extension of harmonic signals
US6915264B2 (en) Cochlear filter bank structure for determining masked thresholds for use in perceptual audio coding
JP5252661B2 (ja) エイリアシング効果抑制のための実副帯信号の処理装置及び処理方法
JP3093179B2 (ja) 高品質オーディオ用短時間遅延変換エンコーダ及びデコーダ
TWI501661B (zh) 參數式立體聲轉換系統及方法
US20070078645A1 (en) Filterbank-based processing of speech signals
RU2256293C2 (ru) Усовершенствование исходного кодирования с использованием дублирования спектральной полосы
US20020177995A1 (en) Method and arrangement for performing a fourier transformation adapted to the transfer function of human sensory organs as well as a noise reduction facility and a speech recognition facility
EP2261894A1 (de) Signalanalyse-/steuerungssystem und -verfahren, signalsteuerungsvorrichtung und -verfahren sowie programm dafür
JP3297050B2 (ja) デコーダスペクトル歪み対応電算式適応ビット配分符号化方法及び装置
EP3764353B1 (de) Verfahren zur mehrstufigen kompression in der unterbandverarbeitung
EP2755205B1 (de) Subband-Verarbeitung zur Komplexitätsverringerung
JP5491193B2 (ja) 音声コード化の方法および装置
EP0707761B1 (de) Vorrichtung zur bestimmung des signalspektrums eines digitalen breitbandsignals und zur ableitung einer bitzuweisungsinformation
EP0720146A1 (de) Verfahren zur Messung von Sprachmaskierungseigenschaften
Hayashi et al. Speech quality enhancement using wavelet reconstruction filters

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20140227

AC Divisional application: reference to earlier application

Ref document number: 2355094

Country of ref document: EP

Kind code of ref document: P

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

R17P Request for examination filed (corrected)

Effective date: 20150115

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 1199974

Country of ref document: HK

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20180528

RIC1 Information provided on ipc code assigned before grant

Ipc: G10L 19/02 20130101ALN20190426BHEP

Ipc: G10L 21/0208 20130101ALN20190426BHEP

Ipc: G10L 25/18 20130101AFI20190426BHEP

RIC1 Information provided on ipc code assigned before grant

Ipc: G10L 25/18 20130101AFI20190516BHEP

Ipc: G10L 21/0208 20130101ALN20190516BHEP

Ipc: G10L 19/02 20130101ALN20190516BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20190711

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: 2236008 ONTARIO INC.

AC Divisional application: reference to earlier application

Ref document number: 2355094

Country of ref document: EP

Kind code of ref document: P

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1212994

Country of ref document: AT

Kind code of ref document: T

Effective date: 20191215

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602011064031

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20191211

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200311

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200311

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200312

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

REG Reference to a national code

Ref country code: DE

Ref legal event code: R081

Ref document number: 602011064031

Country of ref document: DE

Owner name: MALIKIE INNOVATIONS LTD., IE

Free format text: FORMER OWNER: 2236008 ONTARIO INC., WATERLOO, ONTARIO, CA

Ref country code: DE

Ref legal event code: R081

Ref document number: 602011064031

Country of ref document: DE

Owner name: BLACKBERRY LIMITED, WATERLOO, CA

Free format text: FORMER OWNER: 2236008 ONTARIO INC., WATERLOO, ONTARIO, CA

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

REG Reference to a national code

Ref country code: GB

Ref legal event code: 732E

Free format text: REGISTERED BETWEEN 20200723 AND 20200729

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200411

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602011064031

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1212994

Country of ref document: AT

Kind code of ref document: T

Effective date: 20191211

Ref country code: BE

Ref legal event code: MM

Effective date: 20200131

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200124

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

26N No opposition filed

Effective date: 20200914

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200131

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200131

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200131

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200124

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191211

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20240129

Year of fee payment: 14

Ref country code: GB

Payment date: 20240123

Year of fee payment: 14

REG Reference to a national code

Ref country code: DE

Ref legal event code: R081

Ref document number: 602011064031

Country of ref document: DE

Owner name: MALIKIE INNOVATIONS LTD., IE

Free format text: FORMER OWNER: BLACKBERRY LIMITED, WATERLOO, ONTARIO, CA

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20240125

Year of fee payment: 14