EP2580737B1 - Tissue classification - Google Patents

Tissue classification Download PDF

Info

Publication number
EP2580737B1
EP2580737B1 EP11722170.5A EP11722170A EP2580737B1 EP 2580737 B1 EP2580737 B1 EP 2580737B1 EP 11722170 A EP11722170 A EP 11722170A EP 2580737 B1 EP2580737 B1 EP 2580737B1
Authority
EP
European Patent Office
Prior art keywords
voxel
tissue
normal
voxels
abnormal
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP11722170.5A
Other languages
German (de)
English (en)
French (fr)
Other versions
EP2580737A1 (en
Inventor
Yechiel Lamash
Jonathan Lessick
Asher Gringauz
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Koninklijke Philips NV
Original Assignee
Koninklijke Philips NV
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Koninklijke Philips NV filed Critical Koninklijke Philips NV
Publication of EP2580737A1 publication Critical patent/EP2580737A1/en
Application granted granted Critical
Publication of EP2580737B1 publication Critical patent/EP2580737B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/40—Analysis of texture
    • G06T7/41—Analysis of texture based on statistical description of texture
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/50—Extraction of image or video features by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
    • G06V10/507—Summing image-intensity values; Histogram projection analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/12—Classification; Matching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30048—Heart; Cardiac
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/03—Recognition of patterns in medical or anatomical images

Definitions

  • CT computed tomography
  • Coronary artery disease one of the major causes of morbidity and mortality, often manifests as myocardial infarction or ischemia.
  • the size and severity of these defects are among the major determinants of prognosis in this disease.
  • Multi Slice Computed Tomography MSCT
  • MSCT can be used for non-invasive imaging for visualization and assessment of coronary heart disease.
  • MSCT does not provide information regarding the functional significance of coronary stenosis.
  • Perfusion defect and infarct size are currently acquired with a variety of resources such as echocardiography and single photon emission computed tomography (SPECT) myocardial perfusion imaging (MPI).
  • SPECT single photon emission computed tomography
  • MPI myocardial perfusion imaging
  • the literature has shown that intramyocardial distribution of contrast during the arterial phase of enhancement is related to myocardial perfusion. However, the visualization of these hypoenhanced areas is operator-dependent and requires manipulation of the image windowing. Estimation of the infarction extent requires further manual scrolling and marking across adjacent slices.
  • the literature has also revealed various attempts to perform automatic quantification of perfusion defects using SPECT or PET images with simple threshold based methods compared to a normal population and delayed enhancement images to measure myocardial infarct size based on automated feature analysis and combined thresholding.
  • the literature provides a description of objective 3D quantification of perfusion defects based on histogram analysis and comparison with normal values and measured transmural perfusion through a ratio of subendocardial to subepicardial voxel densities. Unfortunately, no fully automatic methods have been disclosed or exist for CT.
  • the invention may take form in various components and arrangements of components, and in various steps and arrangements of steps.
  • the drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention.
  • FIGURE 1 illustrates an imaging system 100 such as a computed tomography (CT) scanner.
  • the imaging system 100 includes a stationary gantry 102 and a rotating gantry 104, which is rotatably supported by the stationary gantry 102.
  • the rotating gantry 104 rotates around an examination region 106 about a longitudinal or z-axis.
  • a radiation source 108 such as an x-ray tube, is supported by the rotating gantry 104 and rotates with the rotating gantry 104, and emits radiation.
  • a radiation sensitive detector array 110 located opposite the source 108 detects radiation that traverses the examination region 106 and generates projection data indicative thereof.
  • a reconstructor 112 reconstructs projection data and generates volumetric image data indicative of the examination region 106.
  • a support 114 such as a couch, supports a subject in the examination region 106.
  • the support 114 can be used to variously position the subject with respect to x, y, and/or z axes before, during and/or after scanning.
  • a general purpose computing system serves as an operator console 116, which includes human readable output devices such as a display and/or printer and input devices such as a keyboard and/or mouse.
  • Software resident on the console 116 allows the operator to control the operation of the system 100.
  • a segmentor 118 is used to variously segment the volumetric image data.
  • the segmentation may include the segmentation of the myocardial voxels from the volumetric image data.
  • the segmentation may be automatic, semi-automatic (with partial user interaction), or manual (with user interaction).
  • a tissue classifier 120 classifies at least a sub-portion of the segmented image data.
  • the classification includes classifying voxels (and hence the tissue represented in the segmented image data) as normal or abnormal tissue.
  • the tissue classifier 120 employs a classification algorithm that is based on a multi-dimensional histogram to classify tissue as normal or abnormal.
  • the 2D representation is based on voxel de-noised CT numbers and voxel normalized distances from the endocardium, and in an embodiment the classification takes into account the gradient (appearance) of CT numbers in the transmural direction.
  • tissue classifier 120 may be part of the system 100 (as shown) or remote therefrom, for example, in a computing system such as a workstation or the like. In either instance, one or more processors may execute computer readable instructions encoded and/or embodied on local or remote computer readable storage medium such as memory to implement the tissue classifier 120.
  • FIGURE 2 illustrates an example tissue classifier 120 for a cardiac application. As shown, the tissue classifier 120 receives segmented image data, which may have been segmented by the segmentor 118 or otherwise.
  • a reformatter 202 can be used to reformat the segmented image data.
  • the original segmented image data is formed from slices along the long axis of the subject.
  • the reformatter 202 identifies the long axis of heart in the segmented image data, identifies the short axis of the heart based on the long axis (e.g., the axis perpendicular to the long axis), and reformats the segmented image data along the short axis of the heart.
  • the reformatter 202 transforms (e.g., rotates and/or translates) the data to the heart axis coordinate system. Where the segmented image data is already in a suitable format, the reformatter 202 can be omitted.
  • a coordinate system transformer 204 transforms the reformatted data into a suitable coordinate system, if the reformatted data is not already in a suitable coordinate system. For example, where the coordinate system of interest is a cylindrical coordinate system and the reformatted data is in another coordinate system such as a rectangular coordinate system, the coordinate system transformer 204 transforms the reformatted data into the cylindrical coordinate system. Coordinate transformations between other coordinate systems are also contemplated herein. Where the segmented image data is already in a suitable coordinate system, the coordinate system transformer 204 can be omitted.
  • a filter 206 can be used to filter the transformed image data.
  • the filtering includes edge preserved de-noising, which facilitates distinguishing tissue types in the image data.
  • An example of a suitable fast edge preserved de-noising filter is a two-dimensional median filter. This filter smoothes the image data at circumferential and radial orientations while preserving edges.
  • the image data is re-formatted to align the heart muscle by mapping the endocardial border into a straight line prior to filtering.
  • the filter 206 is omitted.
  • a 2D voxel representation generator 208 generates a 2D histogram from the filtered image data.
  • the 2D representation can represent voxels in the data of the myocardium using various values such as a voxel de-noised gray level, obtained by the edge preserving image filtering (e.g., by filter 206 or otherwise), and a voxel normalized distance from the endocardium.
  • a classification component 210 classifies voxels, and hence the tissue represented thereby, based on the 2D voxel representation.
  • the classification component 210 can use various algorithms to classify the voxels in the two-dimensional space, at a reduced subspace (e.g., based on principle component analysis), or otherwise. In the following example, a reduced subspace approach based on principle component analysis is utilized.
  • a transformation component 212 processes the 2D histogram.
  • the data transformation component 212 is configured to perform a principle component analysis (PCA) on the 2D representation.
  • PCA principle component analysis
  • the PCA is applied using the normal voxels only. In one instance, since the normal voxels may not be known a priori , it is assumed that most of the myocardial voxels are normal, and the voxels with the values that are close to the 2D histogram's ridge are taken.
  • the resulting histogram includes several main modes for materials such as fat, normal muscle, ischemic muscle, and over enhanced tissue (e.g., due to blood and image artifacts).
  • the relative location of the different modes depends on the concentration and distribution of the iodine in the blood and differs from one patient to the other.
  • One approach to finding thresholds for segmenting abnormal regions is based on probabilistic tissue modeling in which a voxel is classified according to its maximum a-posteriori probability (MAP) to belong to a normal or abnormal tissue segment.
  • MAP a-posteriori probability
  • a suitable mixture model includes a Gaussian mixture model.
  • Such a mixture model assumes that the data distribution is represented as the summation of several Gaussians.
  • the distribution is determined based on a mixture of four Gaussians, which correspond to the above-noted tissues: fat, abnormal tissue (ischemia), normal tissue, and over-enhanced tissue.
  • Other models and threshold techniques for classifying abnormal and normal tissues may alternatively be used.
  • the parameters of the model include: p j
  • j 1 2 ⁇ ⁇ ⁇ ⁇ j 1 / 2 ⁇ exp - 1 2 x - ⁇ j T ⁇ j - 1 ⁇ x - ⁇ j
  • x is a gray level voxel value
  • j is a class (fat, abnormal tissue, normal tissue and over-enhanced tissue)
  • x ) is the a-posteriori probability density function
  • j ) is the conditional probability density function of x belonging to class j
  • P ( j ) is the prior probability to belong to class j
  • ⁇ j is the mean of class j
  • ⁇ j is the covariance matrix of class j .
  • LiklihoodRatio p Y ⁇
  • Abnormal ) are conditional probability density functions, P ( Normal ) and P ( Abnormal ) are the priors, and Y ' V T ⁇ Y , where Y ' represents the one-dimensional voxel value after transforming to the first principle of the normal voxels and V is the Eigen vector of the first principle component.
  • a parameter estimator 214 which include a numerical non-linear optimization solver can be used to estimate the model parameters.
  • the logic 216 performs the fitting to the probability density function using a Genetic Algorithm (GA) optimization or other nonlinear optimization technique.
  • GA Genetic Algorithm
  • a typical GA searches for the optimal solution by first defining an initial population of potential solutions called chromosomes and then subsequently evaluating each chromosome using a pre-defined fitness function.
  • the fitness scores are used in creating a new population (generation) using three genetic operators of recombination, crossover and mutation. This process of creating a new population from the old one iterates until a pre-defined termination condition is satisfied.
  • Other approaches are also contemplated herein.
  • FIGURES 3, 4 , and 5 show the results of the above approach.
  • FIGURE 3 shows a first unprocessed histogram 302 of an abnormal myocardium, and a second histogram 304 of the abnormal myocardium after the above PCA processing. Note that the first histogram 302 does not have distinct peaks while the second histogram 304 includes distinct peaks 306 and 308.
  • FIGURE 4 shows a fitting of the distinct peaks 306 and 308 to Gaussian mixture models 402 and 404.
  • FIGURE 5 shows a histogram 502 of normal myocardium after the above PCA processing and a fitting of the histogram 502 to a single Gaussian mixture model 504.
  • the histogram 502 generally has a single narrow and symmetric shape, unlike the histogram 302 of FIGURE 3 , which neither narrow nor symmetric.
  • the classification component 120 uses these histograms identify and distinguish abnormal tissue from normal tissue.
  • a metric determiner 218 generates various maps such as polar maps, overlay, and/or other maps.
  • the metric determiner 218 determines a metric indicative of the percentage of voxels with gray level below a maximum a-posteriori probability (MAP) threshold along radial ray crossing the muscle. Overlays showing such information can be superimposed over the image voxels according to the probability of being normal or abnormal.
  • the overlay provides a measure of the likelihood ratio (to be normal/ abnormal) taken in a logarithmic domain. This metric is well suited for separating healthy and ischemic tissue, for example, since a large percentage of the voxels will have low values.
  • the metric determiner 218 determines a metric indicative of the transmural gradient from the subendocardium to the subepicardium.
  • a metric indicative of the transmural gradient from the subendocardium to the subepicardium Generally, normal myocardium typically has a negative slope and true perfusion defects usually have a positive slope (e.g., since the subendocardial regions are always affected first by ischemia).
  • This metric provides a measure of the slope of the Hounsfield values through the radial thickness of the ventricular wall, or the difference in the enhancement of the subendocardial compared to the subepicardial segments along radial rays crossing the muscle. This metric yields good separation between dark artifacts and real ischemic tissue, for example, since artifacts do not tend to have such a difference between the subendocardial and subepicardial tissues.
  • the classification component 210 can use one or both of the above-described metrics, and/or optionally one or more other metrics, for determining optimal separation between healthy and ischemic tissue and between ischemic tissue and image artifacts. This allows the classification component 210 to identify each radial ray crossing the muscle as normal or abnormal according to the percentage of abnormal voxels and by its subendocardial to subepicardial voxel density difference, such as through a weighted combination of these factors to determine the likelihood of abnormality.
  • the transmural density difference should be close to zero for a normal region or for a transmural defect, and a value below zero indicates a perfusion defect whereas a value above zero suggests an artifact.
  • FIGURE 6 illustrates a method for identifying normal and/or abnormal tissue.
  • the method is described in connection with cardiac data and quantifying myocardial perfusion defects or infarction from volumetric images data of the heart.
  • segmented volumetric image data is obtained.
  • the segmented data includes segmentation of the myocardial voxels from the 3D volume.
  • the segmented volumetric image data is filtered.
  • this applying an edge-preserving de-noising filter can enhance regional differences in voxel values, which may enhance the separation between normal and abnormal voxels.
  • a two-dimensional voxel representation is generated for the segmented image data.
  • each voxel (or a subset of voxels) in the myocardium is represented through a voxel de-noised CT number (obtained from the filter image data) and a voxel normalized distance from the endocardium.
  • the normalization factor can be the myocardium width or other distance.
  • the voxels are classified as normal or abnormal tissue. In one instance, this is achieved by applying a dimension reduction such as a principle component analysis (PCA). In another instance, voxel classification can be performed in the higher dimensional space or otherwise.
  • PCA principle component analysis
  • false classifications are identified. This can be done through a second pass in the spatial domain using Hounsfield gradients to differentiate between true defects and artifacts.
  • normal myocardium typically has a negative slope
  • true perfusion defects usually have a positive slope since the subendocardial regions are affected first by ischemia. In another embodiment, this act is omitted.
  • infarct regions do not tend to have straight, long and thin or sharp and small geometry but rather a continuous and smooth shape since they occur in the territory of a diseased coronary artery distal to an occlusion or severe stenosis. Therefore, a spatial evaluation of the classified voxels can be performed to remove small isolated regions below a predefined threshold which are likely to be artifacts.
  • the above described acts may be implemented by way of computer readable instructions, which, when executed by a computer processor(s), causes the processor(s) to carry out the acts described herein.
  • the instructions are stored in a computer readable storage medium such as memory associated with and/or otherwise accessible to the relevant computer.
  • CT computed tomography
  • other imaging modalities such as MRI, PET, SPECT, and/or other modalities.
  • the approach described herein can also be employed with echocardiography using specialized contrast agents to identify and size perfusion defects.
  • the approach described herein can be employed for various applications.
  • the approach described herein can be used to automatically identify and quantify perfusion defects in the myocardium from CT studies.
  • the approach can be used in detection and quantification of myocardial delayed enhancement. In this case the algorithm would search for voxels with increased rather than decreased CT numbers.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Medical Informatics (AREA)
  • Quality & Reliability (AREA)
  • Radiology & Medical Imaging (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Probability & Statistics with Applications (AREA)
  • Apparatus For Radiation Diagnosis (AREA)
  • Magnetic Resonance Imaging Apparatus (AREA)
EP11722170.5A 2010-06-14 2011-04-27 Tissue classification Active EP2580737B1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US35429110P 2010-06-14 2010-06-14
PCT/IB2011/051845 WO2011158135A1 (en) 2010-06-14 2011-04-27 Tissue classification

Publications (2)

Publication Number Publication Date
EP2580737A1 EP2580737A1 (en) 2013-04-17
EP2580737B1 true EP2580737B1 (en) 2015-08-26

Family

ID=44120316

Family Applications (1)

Application Number Title Priority Date Filing Date
EP11722170.5A Active EP2580737B1 (en) 2010-06-14 2011-04-27 Tissue classification

Country Status (4)

Country Link
US (1) US8781192B2 (zh)
EP (1) EP2580737B1 (zh)
CN (1) CN102939616B (zh)
WO (1) WO2011158135A1 (zh)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8965484B2 (en) * 2011-04-27 2015-02-24 General Electric Company Method and apparatus for generating a perfusion image
US20170071470A1 (en) * 2015-09-15 2017-03-16 Siemens Healthcare Gmbh Framework for Abnormality Detection in Multi-Contrast Brain Magnetic Resonance Data
EP3392804A1 (en) * 2017-04-18 2018-10-24 Koninklijke Philips N.V. Device and method for modelling a composition of an object of interest
US11238303B2 (en) * 2019-05-15 2022-02-01 Getac Technology Corporation Image scanning method for metallic surface and image scanning system thereof
US11423318B2 (en) * 2019-07-16 2022-08-23 DOCBOT, Inc. System and methods for aggregating features in video frames to improve accuracy of AI detection algorithms
US10671934B1 (en) 2019-07-16 2020-06-02 DOCBOT, Inc. Real-time deployment of machine learning systems
CN111311525A (zh) * 2019-11-20 2020-06-19 重庆邮电大学 一种基于直方图概率修正的图像梯度场双区间均衡化算法
US11100373B1 (en) 2020-11-02 2021-08-24 DOCBOT, Inc. Autonomous and continuously self-improving learning system

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU1837495A (en) * 1994-10-13 1996-05-06 Horus Therapeutics, Inc. Computer assisted methods for diagnosing diseases
US7877130B2 (en) * 2002-10-03 2011-01-25 Siemens Medical Solutions Usa, Inc. System and method for using delayed enhancement magnetic resonance imaging and artificial intelligence to identify non-viable myocardial tissue
US7912528B2 (en) * 2003-06-25 2011-03-22 Siemens Medical Solutions Usa, Inc. Systems and methods for automated diagnosis and decision support for heart related diseases and conditions
EP2158575B1 (en) 2007-06-20 2018-04-11 Koninklijke Philips N.V. Detecting haemorrhagic stroke in ct image data
EP2232443A4 (en) * 2008-01-10 2012-07-04 Agency Science Tech & Res DIFFERENTIATION OF ARTEFACTS INFARCTUS IN IRM SCAN DATA
US8150119B2 (en) * 2008-01-24 2012-04-03 Siemens Aktiengesellschaft Method and system for left ventricle endocardium surface segmentation using constrained optimal mesh smoothing
US8139838B2 (en) * 2008-05-22 2012-03-20 Siemens Aktiengesellschaft System and method for generating MR myocardial perfusion maps without user interaction
CN102047295B (zh) * 2008-06-04 2014-01-08 皇家飞利浦电子股份有限公司 用于测量示踪剂摄取和再分布的动态心脏spect的重建

Also Published As

Publication number Publication date
CN102939616A (zh) 2013-02-20
US8781192B2 (en) 2014-07-15
EP2580737A1 (en) 2013-04-17
US20130077838A1 (en) 2013-03-28
CN102939616B (zh) 2015-11-25
WO2011158135A1 (en) 2011-12-22

Similar Documents

Publication Publication Date Title
EP2580737B1 (en) Tissue classification
AU2018225145B2 (en) Detection of prostate cancer in multi-parametric MRI using random forest with instance weighting and MR prostate segmentation by deep learning with holistically-nested networks
Mukhopadhyay A segmentation framework of pulmonary nodules in lung CT images
Huang et al. Liver tumor detection and segmentation using kernel-based extreme learning machine
Foruzan et al. Improved segmentation of low-contrast lesions using sigmoid edge model
Baâzaoui et al. Semi-automated segmentation of single and multiple tumors in liver CT images using entropy-based fuzzy region growing
US8724866B2 (en) Multi-level contextual learning of data
US8958618B2 (en) Method and system for identification of calcification in imaged blood vessels
Mouton et al. Materials-based 3D segmentation of unknown objects from dual-energy computed tomography imagery in baggage security screening
US9230320B2 (en) Computer aided diagnostic system incorporating shape analysis for diagnosing malignant lung nodules
Wan et al. Automated coronary artery tree segmentation in X-ray angiography using improved Hessian based enhancement and statistical region merging
EP1934940B1 (en) Image processing method for boundary extraction between at least two tissues, the boundary being found as the cost minimum path connecting a start point and an end point using the fast marching algorithm with the cost function decreases the more likely a pixel is not on the boundary
US20070248254A1 (en) System and Method for Automatic Detection of Internal Structures in Medical Images
JP2011526508A (ja) 医療用画像の領域分割
Van Dongen et al. Automatic segmentation of pulmonary vasculature in thoracic CT scans with local thresholding and airway wall removal
JP2010207572A (ja) 障害のコンピュータ支援検出
Zambal et al. Shape and appearance models for automatic coronary artery tracking
Zinoveva et al. A texture-based probabilistic approach for lung nodule segmentation
Shahzad et al. A patient-specific coronary density estimate
Van Velzen et al. Generative models for reproducible coronary calcium scoring
Jalab et al. Fractional Renyi entropy image enhancement for deep segmentation of kidney MRI
Moltz et al. A general framework for automatic detection of matching lesions in follow-up CT
Zuluaga et al. Feature selection for SVM-based vascular anomaly detection
Anwar et al. Segmentation of liver tumor for computer aided diagnosis
Theresa et al. CAD for lung nodule detection in chest radiography using complex wavelet transform and shearlet transform features

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20130114

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAX Request for extension of the european patent (deleted)
RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: KONINKLIJKE PHILIPS N.V.

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

INTG Intention to grant announced

Effective date: 20150320

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 745580

Country of ref document: AT

Kind code of ref document: T

Effective date: 20150915

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602011019148

Country of ref document: DE

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 745580

Country of ref document: AT

Kind code of ref document: T

Effective date: 20150826

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20151127

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20151126

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20150826

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20151228

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20151226

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 6

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602011019148

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20160530

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160430

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20160427

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

Ref country code: LU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160427

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160430

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160430

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160427

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 7

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160427

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 8

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20110427

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160430

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

REG Reference to a national code

Ref country code: DE

Ref legal event code: R084

Ref document number: 602011019148

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150826

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20230421

Year of fee payment: 13

Ref country code: DE

Payment date: 20220628

Year of fee payment: 13