EP2589340B1 - Appareil, procédé et programme de traitement d'image médicale - Google Patents

Appareil, procédé et programme de traitement d'image médicale Download PDF

Info

Publication number
EP2589340B1
EP2589340B1 EP12774407.6A EP12774407A EP2589340B1 EP 2589340 B1 EP2589340 B1 EP 2589340B1 EP 12774407 A EP12774407 A EP 12774407A EP 2589340 B1 EP2589340 B1 EP 2589340B1
Authority
EP
European Patent Office
Prior art keywords
bronchi
representing
medical image
node
continuous region
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Not-in-force
Application number
EP12774407.6A
Other languages
German (de)
English (en)
Other versions
EP2589340A4 (fr
EP2589340A1 (fr
Inventor
Tsutomu Inoue
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fujifilm Corp
Original Assignee
Fujifilm Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fujifilm Corp filed Critical Fujifilm Corp
Publication of EP2589340A1 publication Critical patent/EP2589340A1/fr
Publication of EP2589340A4 publication Critical patent/EP2589340A4/fr
Application granted granted Critical
Publication of EP2589340B1 publication Critical patent/EP2589340B1/fr
Not-in-force legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/06—Topological mapping of higher dimensional structures onto lower dimensional surfaces
    • G06T3/067—Reshaping or unfolding 3D tree structures onto 2D planes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection

Definitions

  • the present invention relates to a technique for constructing a graph structure representing a tubular structure in a three-dimensional medical image.
  • Lung cancers are a disease in which a survival rate sharply drops as the stage of the disease progresses. Therefore, early detection and early treatment of lung cancers are extremely important. For example, when a shadow of a suspicious tumor or the like is found in a simple X-ray image of a chest, a CT image or the like, precise examinations are necessary to judge whether the suspicious tumor is a tumor. If the suspicious tumor is a tumor, it is necessary to judge whether the tumor is benign or malignant. In judging whether the tumor is benign or malignant, a pathology examination is performed by removing a part of the tumor by a bronchial endoscope. In this examination, it is important to more speedily and more accurately move the endoscope to the position of the tumor. For that purpose, it is effective to recognize the shape (a branching pattern or the like) of bronchi and a path to the tumor by using a CT image obtained by imaging before the examination.
  • a technique using a Hessian matrix as an image recognition technique for extracting linear structures, such as bronchi, from a three-dimensional medical image obtained by CT, or the like. Specifically, first, multi-resolution transformation is performed on the three-dimensional medical image. After then, eigenvalue analysis of Hessian matrix is performed on the image of each resolution to extract a linear structure element.
  • the linear structure element has a characteristic that only one of three eigenvalues obtained by eigenvalue analysis is close to 0.
  • linear structure elements blood vessels in various sizes are extracted from the three-dimensional medical image by combining results of analyzing images of respective resolutions.
  • data of a tree structure representing tubular structures in the three-dimensional medical image are obtained by connecting the extracted linear structure elements to each other by using a minimum spanning tree algorithm.
  • a cost function based on a positional relationship between linear structure elements or the principal axis direction of each linear structure element represented by an eigenvector corresponding to the aforementioned eigenvalue close to 0 is used (Japanese Unexamined Patent Publication No. 2010-220742 (Patent Document 1).
  • Kitasaka et al. (Bronchial region extraction from 3D chest CT image by voxel classification based on local intensity structure; 2010) presents a method for extracting bronchial regions by voxel classification based on local intensity structure. Hessian matrix analysis is used as of tube enhancement filter.
  • Bauer et al. 2010 Stementation of interwoven 3d tubular tree structures utilizing shape priors and graph cuts; 2010 ) present a method for segmenting tree structures; the method comprises two main steps: 1) the tree structures are identified and corresponding shape priors are generated by using a bottom-up multi-scale identification of tubular objects combined with a top-down grouping of these objects into complete tree structures, this allows us to separate interwoven trees and to handle local disturbances; 2) the generated shape priors are utilized for the intrinsic segmentation of the different tubular systems to avoid leakage or under segmentation in disturbed regions.
  • bronchi have a tree structure in which a diameter gradually becomes smaller from 20 mm or larger to 0.5 mm or less while repeating branching from a trachea, and exhibit different anatomical and image characteristics depending on the diameter.
  • the angle of branching is different.
  • the angle of branching may be an obtuse angle at a large diameter portion of the bronchi, but the angle of branching is an acute angle at a small diameter portion of the bronchi.
  • the bronchi are interrupted. Specifically, the bronchi in an image are not interrupted at a large diameter portion of the bronchi as long as a stenosis caused by a disease or the like is not present.
  • the bronchi in the image may be represented as if the bronchi are interrupted because of a noise in the image, which is caused by a partial volume effect and a motion artifact of a heart.
  • the thickness of the wall of the bronchi is different. Specifically, since the wall of the bronchi is thick at a large diameter portion of the bronchi, a difference in CT values between the wall of the bronchi and an air region in the bronchi is prominent.
  • a medical image processing apparatus of the present invention is a medical image processing apparatus comprising:
  • a medical image processing method of the present invention is a medical image processing method comprising the steps of:
  • a medical image processing program of the present invention is a program for causing a computer to execute the medical image processing method.
  • the principal axis direction of the linear structure may be calculated, and the cost function in the second segment may be defined in such a manner that the second segment is more likely to be connected as a change in the principal axis directions at nodes located at both ends of the second segment is smaller.
  • a plurality of images having different resolutions from each other may be generated from the three-dimensional medical image. Then, a lower resolution image of the generated plurality of images may be received, and the continuous region may be extracted from the lower resolution image. Further, a higher resolution image of the generated plurality of images may be received, and extraction of the linear structure and calculation of the principal axis direction may be performed on the higher resolution image.
  • the continuous region when a continuous region having a voxel value corresponding to an air region in the bronchi is extracted, the continuous region may be extracted by connecting neighboring voxels the voxel values of which are close to each other in such an extent to satisfy a predetermined condition.
  • a graph-cut method, a region growing method, a level set method, or the like may be used.
  • Hessian matrix may be used to extract a linear structure and to calculate a principal axis direction.
  • a minimum spanning tree algorithm may be used to construct a graph structure.
  • a continuous region representing the inside of bronchi is extracted from a three-dimensional medical image.
  • a large diameter portion of the bronchi in the image is represented, as a region without interruption. Therefore, even if a point branching at an obtuse angle is present, the continuous region of the bronchi is extracted at high accuracy. Therefore, a graph structure generated based on the continuous region represents the bronchi also at high accuracy.
  • plural linear structures representing fragments of bronchi are extracted by analyzing a local density structure in a neighborhood of each point in the three-dimensional medical image. As described above, a small diameter portion of the bronchi in the image may be interrupted. However, since this second extraction technique can handle the bronchi as a set of fragmentary linear structures, highly accurate extraction is possible.
  • different cost functions are used for a first segment connecting the node constituting the graph structure representing the continuous region of the bronchi and the node representing the linear structures representing fragments of the bronchi and a second segment connecting the nodes representing the linear structures representing fragments of the bronchi to each other.
  • the cost function in the first segment is defined in such a manner that the first segment is more likely to be connected as a change in voxel values is smaller among first segments.
  • a graph structure of the bronchi is reconstructed by using this cost function. Therefore, it is possible to appropriately connect a large diameter portion of the bronchi extracted by using the first extraction method and a small diameter portion of the bronchi extracted by using the second extraction method.
  • a graph structure of bronchi in a three-dimensional medical image is constructed at higher accuracy.
  • FIG. 1 is a schematic diagram illustrating the hardware configuration of this medical image diagnosis system.
  • a modality 1 an image storage server 2, and an image processing workstation 3 are connected to each other through a network 9 in such a manner that they can communicate with each other, as illustrated in Figure 1 .
  • the modality 1 includes an apparatus that generates image data of a three-dimensional medical image representing an examination target region of a subject to be examined by imaging the region.
  • the apparatus attaches supplementary information defined by DICOM (Digital Imaging and Communications in Medicine) standard to the image data, and outputs the image data, as image information.
  • DICOM Digital Imaging and Communications in Medicine
  • a CT is used as the modality 1, and a case in which three-dimensional image data are generated by scanning the chest of a human body, which is a subject to be examined, in the direction of the body axis by the CT will be described.
  • the image storage server 2 is a computer that stores, in an image database, medical image data obtained by the modality 1 and image data of medical images generated by image processing at the image processing workstation 3, and that manages the data.
  • the image storage server 2 includes a large capacity external storage device, software for database management (for example, ORDB (Object Relational Database) management software).
  • the image storage server 2 searches the image database based on a search request from the image processing workstation 3, and sends extracted image data to the image processing workstation 3.
  • the image processing workstation 3 is a computer that performs image processing (including image analysis) on medical image data obtained from the modality 1 and the image storage server 2 based on a request from a radiologist who performs image reading, and displays a generated image.
  • the image processing workstation 3 includes known hardware elements, such as a CPU, a main storage device, an auxiliary storage device, an input output interface, a communication interface, an input device (a mouse, a keyboard, or the like), a display device (a display monitor), and a data bus. Further, a known operating system, a medical image processing program, which will be described later, or the like has been installed in the image processing workstation 3.
  • Medical image processing performed at the image processing workstation 3 is realized by executing a medical image processing program installed from a recording medium, such as a CD-ROM.
  • This program may be downloaded from a storage device of a server connected through a network, such as the Internet, and installed.
  • the storage format of image data and communication between devices through the network 9 are based on a protocol, such as DICOM.
  • a simulation and navigation function for a surgery or an endoscopic examination of bronchi or lungs is packaged in the image processing workstation 3.
  • this function after bronchi are automatically extracted from a three-dimensional medical image of a chest obtained by CT, virtual endoscope images of the bronchi are generated along an insertion path of a real endoscope, and sequentially displayed.
  • Medical image processing of the present invention for constructing a graph structure (hereinafter, a tree structure is used as an example) representing the bronchi is packaged as processing for automatically extracting the bronchi.
  • the image processing workstation 3 functions as a medical image processing apparatus of the present invention.
  • FIG. 2 is a schematic diagram illustrating processing blocks related to bronchial region extraction processing according to an embodiment of the present invention among functions of the image processing workstation 3, and a flow of data and processing.
  • the medical image processing of the present invention is also included in this bronchial region extraction processing.
  • the bronchial region extraction processing in the present embodiment is realized by a multi-resolution image generation unit 31, a continuous region extraction unit 32, a tree structure construction unit 33, a linear structure extraction unit 34, a tree structure reconstruction unit 35, and a bronchial region extraction unit 36.
  • three-dimensional medical image V which is data input to the multi-resolution image generation unit 31, is imaged by the modality 1 (CT), and stored in the image storage server 2.
  • the three-dimensional medical image V is extracted by database search by the image storage server 2 based on a search request from the image processing workstation 3, and sent to the image processing workstation 3.
  • the multi-resolution image generation unit 31 reads this three-dimensional medical image V from a memory area, and performs multi-resolution transformation to generate low resolution image VL and high resolution image VH.
  • the low resolution image VL and the high resolution image VH are stored in a predetermined memory area.
  • the continuous region extraction unit 32 reads the low resolution image VL from the memory area, and extracts continuous region RL having a voxel value corresponding to an air region in the bronchi.
  • the continuous region RL is stored in a predetermined memory area.
  • the continuous region RL is a region representing the bronchi extracted from the low resolution image VL. Therefore, the continuous region RL represents a portion of the bronchi having a larger diameter than that of a region representing the bronchi extracted from the high resolution image VH, which will be described later.
  • the tree structure construction unit 33 reads the continuous region RL representing large bronchi from the memory area, and constructs a tree structure representing the large bronchi.
  • the tree structure construction unit 33 stores node information NL n of the tree structure in a predetermined memory area.
  • the linear structure extraction unit 34 reads high resolution image VH from the memory area, and extracts plural linear structures EH m representing fragments of small bronchi from the high resolution image VH by analyzing a local density structure in a neighborhood of each point in the high resolution image VH.
  • the linear structure extraction unit 34 stores the plural linear structures EH m in a predetermined memory area.
  • the tree structure reconstruction unit 35 reads information NL n about each node of the tree structure representing large bronchi and each linear structure EH m representing small bronchi from the memory area.
  • the tree structure reconstruction unit 35 reconstructs a tree structure representing the bronchi by connecting node NL n of a tree structure representing large bronchi and node EH m representing a linear structure of small bronchi to each other, and nodes EH m representing a linear structure of small bronchi to each other by using a predetermined cost function, which will be described later.
  • the tree structure reconstruction unit 35 stores tree structure data T in a predetermined memory area.
  • the bronchial region extraction unit 36 reads the tree structure data T from the memory area, and extracts whole bronchial region R.
  • the bronchial region extraction unit 36 stores the whole bronchial region R in a predetermined memory area.
  • the multi-resolution image generation unit 31 generates a Gaussian pyramid by performing known multi-resolution transformation processing on three-dimensional medical image V. Images generated by this processing are low resolution image VL and high resolution image VH. Three or more images may be generated. In such a case, the generated images should be classified into a low resolution image group and a high resolution image group based on a standard, such as a resolution, which has been set in advance.
  • threshold processing based on a CT value corresponding to each component of a human body and labeling processing may be performed as preprocessing before performing multi-resolution transformation processing.
  • a connected region representing the outside of the human body, which is a subject to be examined, soft tissue in the body surface, bones and the like may be deleted by the preprocessing to extract a lung field region, which remains after the preprocessing.
  • Figure 6 is a schematic diagram illustrating a lung field region, which is a target of the following processing. Some other known techniques may be used to extract the lung field region.
  • the continuous region extraction unit 32 includes a Hessian matrix eigenvalue analysis unit 32a and a graph-cut processing unit 32b.
  • the Hessian matrix eigenvalue analysis unit 32a extracts a linear structure representing bronchi by performing eigenvalue analysis of Hessian matrix.
  • Figure 7A is a schematic diagram illustrating a set of extracted linear structures of bronchi.
  • Figure 7B is a schematic diagram illustrating a part of the lung field region illustrated in Figure 6 , and the part excluding the set of linear structures of bronchi illustrated in Figure 7A .
  • the Hessian matrix is a matrix composed of second order partial differential coefficients.
  • Three eigenvalues ⁇ 0 , ⁇ 1 , ⁇ 2 of this Hessian matrix exhibit a characteristic that two of the three eigenvalues are large and the remaining one eigenvalue is close to 0 at a point representing a linear structure in the image, as follows: ⁇ 0 ⁇ 0 ⁇ 1 ⁇ 0 ⁇ 2 ⁇ 0
  • the multi-resolution image generation unit 31 When the multi-resolution image generation unit 31 generates images at three or more resolutions, eigenvalue analysis of Hessian matrix is performed on each of plural low resolution images belonging to a low resolution image group. At this time, even if ⁇ corresponds to one kind of filter size, it is possible to extract plural sizes of linear structure, because the resolutions of processing target images are different. Finally, it is possible to extract various sizes of linear structure in the low-resolution image group by combining analysis results of images at respective resolutions (please refer to Y. Sato et al., "Three-dimensional multi-scale line filter for segmentation and visualization of curvilinear structures in medical images", Medical Image Analysis, Vol. 2, No. 2, pp. 143-168, June 1998 , and the like).
  • the graph-cut processing unit 32b extracts continuous region RL representing large bronchi, and which is a region of a set of points representing extracted linear structures (please refer to Figure 7A ) by using a known graph-cut method (please refer, for example, to Japanese Unexamined Patent Publication No. 2009-211138 , and the like).
  • the graph-cut processing unit 32b extracts the continuous region RL by dividing the set of points representing extracted linear structures and a set of points (please refer to Figure 7B ) other the set of points representing the extracted linear structures into an extraction target region to which the former points belong and a background region to which the latter points belong.
  • Figure 7C is a schematic diagram illustrating the extracted continuous region RL representing large bronchi.
  • the continuous region extraction unit 32 may extract continuous region RL representing large bronchi by using some other known techniques, such as a region growing method and a level set method, as long as a continuous region having a voxel value corresponding to an air region in the bronchi is extracted (particularly, a technique of connecting neighboring voxels the voxel values of which are close to each other in such an extent to satisfy a predetermined condition).
  • some other known techniques such as a region growing method and a level set method
  • the tree structure construction unit 33 includes a thinning processing unit 33a and a minimum spanning tree processing unit 33b.
  • the thinning processing unit 33a performs three-dimensional thinning processing on the continuous region RL representing large bronchi (for details, please refer, for example, to Y. Kawata et al., "Processing Algorithm of Three-Dimensional Blood Vessel Image of Cone Beam CT", Transactions of the Institute of Electronics, Information and Communication Engineers D-II, Vol. J79-D-II, No.6, pp. 1134-1145, June 1996 , and the like).
  • Figure 8 is a schematic diagram illustrating a continuous region representing thinned large bronchi.
  • the minimum spanning tree processing unit 33b sets, as a node point, each voxel obtained by thinning the continuous region, and determines one of these node points that is closest to the trachea, as a root of the tree structure.
  • the point closest to the trachea may be determined by using a classifier obtained by known machine learning, or the like.
  • a point in the thinned continuous region, and the point being closest to the head in the body axis direction may be determined as the point closest to the trachea, or the point closest to the trachea may be specified by a manual operation by a user.
  • the minimum spanning tree processing unit 33b calculates a cost for connecting each node point based on position information about plural node points, in other words, a distance between node points. Further, the minimum spanning tree processing unit 33b generates tree structure data NL n representing large bronchi by connecting the node points by using a minimum spanning tree algorithm.
  • the tree structure construction unit 33 may construct a tree structure by searching the thinned continuous region from the determined root toward the neighborhood of the root.
  • the linear structure extraction unit 34 includes a Hessian matrix eigenvalue analysis unit 34a and a principal axis direction vector specification unit 34b.
  • the Hessian matrix eigenvalue analysis unit 34a analyzes a local density structure in a neighborhood of each point in the high resolution image VH by performing eigenvalue analysis of Hessian matrix in a manner similar to the Hessian matrix eigenvalue analysis unit 32a in the continuous region extraction unit 32. Accordingly, the Hessian matrix eigenvalue analysis unit 34a extracts linear structure EH m representing small bronchi.
  • Figure 9 is a schematic diagram illustrating the extracted linear structure EH m .
  • the Hessian matrix eigenvalue analysis unit 34a does not extract linear structure EH m from the continuous region RL extracted by the continuous region extraction unit 32 to prevent extraction of bronchi (a continuous region, a linear structure) from the same region in three-dimensional medical image V by both of the continuous region extraction unit 32 and the linear structure extraction unit 34.
  • the tree structure reconstruction unit 35 calculates cost ⁇ i , i+1 (i, i+1) between nodes by using at least one of plural characteristics at node point P i and node points P i-1 , P i+1 , which are before and after the node point P i .
  • node points P i-1 , P i , P i+1 are represented without distinguishing node point NL n representing large bronchi and node point EH m representing small bronchi from each other.
  • a second specific example is a method for calculating a cost representing a change in a direction in which the bronchi run at each node point.
  • the cost is calculated from the sum of angle ⁇ between a vector of the direction in which bronchi run at node point P i and straight line P i P i+1 and angle ⁇ between a vector of the direction in which bronchi run at node point P i+1 and the straight line P i P i+1 .
  • the calculation is represented by the following formula.
  • an eigenvector corresponding to one of three eigenvalues that is closest to 0, which have been calculated by the Hessian matrix eigenvalue analysis units 32a, 34a may be used.
  • the principal axis direction vector of a linear structure extracted by the Hessian matrix eigenvalue analysis units 32a, 34a may be used.
  • the principal axis direction vector should be stored in a predetermined memory area by the Hessian matrix eigenvalue analysis units 32a, 34a together with position information or the like of the point in the linear structure in such a manner to be related to each node point NL n of a tree structure representing large bronchi, or to each node point EH m representing a linear structure of small bronchi.
  • a third specific example is also a method for calculating a cost representing a change in the direction in which bronchi run at each node point.
  • a cost is calculated based on angle ⁇ between a vector from the node point P i-1 to the node point P i and a vector from the node point P i to the node point P i+1 .
  • a fourth specific example is a method for calculating a cost representing a change in density values on a straight line connecting node point P i and node point P i+1 .
  • the tree structure reconstruction unit 35 performs weighting on three elements, i.e. a positional relationship between two node points to be connected to each other (a distance between node points), a change in vectors of the directions in which bronchi run at two node points, and a change in density values between two node points to be connected to each other, based on the kinds of node points to be connected. Further, the tree structure reconstruction unit 35 calculates a connection cost when two node points are connected to each other.
  • connection cost is calculated by increasing the degree of weighting on a positional relationship between two node points.
  • connection cost is calculated by increasing the degree of weighting on a change in the vectors of the directions in which bronchi run at the two node points (for details, please refer to Patent Document 1).
  • connection cost is calculated by increasing the degree of weighting on a change in voxel values between the two node points.
  • Figure 13 is a schematic diagram illustrating the vicinity of a boundary between each node point NL n representing large bronchi and node point EH m representing small bronchi.
  • the mark o represents the former node point
  • the mark x represents the latter node point.
  • a real wall of the bronchi is illustrated.
  • connection from node point NL 100 through node point NL 104 is connection between node points representing large bronchi. Therefore, adjacent node points are connected to each other based on a connection cost obtained by increasing the degree of weighting on a positional relationship between each node point.
  • connection from node point EH 101 through node point EH 109 is connection between node points representing small bronchi.
  • node points EH 101 through EH 104 which run toward a lower right direction, are connected to each other based on a connection cost obtained by increasing the degree of weighting on a change, between nodes, in the vectors of the directions in which bronchi run.
  • the order of connection from the node point EH 101 to the node point EH 104 is based on a cost of the positional relationship of each node, and adjacent node points are connected to each other.
  • node points EH 105 through EH 109 which run toward the right side, are also sequentially connected.
  • node point EH 100 is not connected to these other node points EH 101 , ..., EH 109 .
  • a connection cost is calculated by increasing the degree of weighting on a change in density values between two node points to be connected to each other. Since the wall of the bronchi is not present between the node point NL 104 representing large bronchi and the node point EH 101 representing small bronchi, a change in density values between the node points is small. Therefore, the two nodes are connected to each other. Similarly, since the wall of the bronchi is not present between the node point NL 104 representing large bronchi and the node point EH 105 representing small bronchi, a change in density values between the node points is small.
  • the two node points are connected to each other.
  • the wall of the bronchi is present between the node point NL 104 representing large bronchi and the node point EH 100 representing small bronchi. Since a change in density values between the node points is large, the two node points are not connected to each other.
  • node point NL 104 representing large bronchi and node point EH 102 representing small bronchi are not connected to each other, because a positional relationship between the node point NL 104 and the node point EH 102 is far, compared with a positional relationship between the node point NL 104 and the node point EH 101 , and consequently, a connection cost of the former is larger than a connection cost of the latter.
  • node points NL 100 through NL 104 representing large bronchi, node point NL 104 representing large bronchi and node point EH 101 representing small bronchi, node point NL 104 representing large bronchi and node point EH 105 representing small bronchi, node points EH 101 through EH 104 representing small bronchi, and node points EH 105 through EH 109 representing small bronchi are correctly connected.
  • node point EH 100 is regarded as an erroneous extraction point, and not connected to any node point.
  • the tree structure reconstruction unit 35 may perform reconnection processing on all of node points NL n representing large bronchi and node points EH m representing small bronchi by using a minimum spanning tree algorithm.
  • the tree structure reconstruction unit 35 may directly use the result of connection by the tree structure construction unit 33 for the node points NL n representing large bronchi. Consequently, it is possible to omit the repeated process, i.e., the process of connecting the nodes NL n representing large bronchi. Therefore, the processing efficiency is improved.
  • the bronchial region extraction unit 36 obtains a cross section orthogonal to the direction in which the bronchi run at each node of tree structure data T representing the reconstructed whole bronchi. Further, the bronchial region extraction unit 36 recognizes the contour of the bronchi at each cross section by using a known segmentation method, such as a graph-cut method, which has been described already, and relates information representing the contour to each node of the tree structure data T. Accordingly, bronchial region R is extracted.
  • a known segmentation method such as a graph-cut method
  • the continuous region extraction unit 32 and the tree structure construction unit 33 extract large bronchi by using a technique appropriate for the anatomical and image characteristics of the large diameter portion of the bronchi. Further, the linear structure extraction unit 34 extracts small bronchi by using a technique appropriate for the anatomical and image characteristics of the small diameter portion of the bronchi.
  • the tree structure reconstruction unit 35 connects node points representing large bronchi to each other, a connection cost is calculated based on a cost function that is defined in such a manner that the degree of weighting on a positional relationship between two node points is increased.
  • a connection cost is calculated based on a cost function that is defined in such a manner that the degree of weighting on a change in vectors of the directions in which bronchi run at two node points is increased. Accordingly, each node is appropriately connected based on a cost function calculated based on a cost function appropriate for an anatomical and image characteristic of each of a large diameter portion of the bronchi and a small diameter portion of the bronchi. Further, when a node point representing large bronchi and a node point representing a small node point are connected to each other, a connection cost is calculated by increasing the degree of weighting on a change in density values between the node points. Accordingly, it is possible to consider whether the wall of the bronchi is present between these node points. Therefore, large bronchi and small bronchi that have been extracted by using different techniques from each other are appropriately connected to each other.
  • the bronchi of the human body are an extraction target.
  • the present invention may be applied to extraction of a tubular structure having an anatomical and image characteristic similar to that of the bronchi.
  • the bronchi extraction technique of the present invention is packaged as a part of simulation and navigation functions of a surgery and an examination using a virtual endoscope image of bronchi.
  • the present invention may be packaged as a bronchi analysis function for diagnosing a disease of lungs or bronchi, such as chronic obstructive pulmonary disease (COPD), based on an image.
  • COPD chronic obstructive pulmonary disease
  • the bronchi analysis function can generate a CPR image of a part of bronchi specified by a user after the bronchi are automatically extracted from a three-dimensional medical image of the chest obtained by CT. Further, the bronchi analysis function can measure the diameter of the bronchi and the thickness of the wall of the bronchi.
  • the multi-resolution image generation unit 31 is provided to reduce a process load and to increase the processing speed by extracting large bronchi from a low resolution image.
  • large bronchi and small bronchi may be extracted from the original three-dimensional medical image without providing the multi-resolution image generation unit 31.
  • different sizes of filters should be used by the continuous region extraction unit 32 and by the linear structure extraction unit 34 when eigenvalue analysis of Hessian matrix is performed.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Quality & Reliability (AREA)
  • Radiology & Medical Imaging (AREA)
  • Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Apparatus For Radiation Diagnosis (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)
  • Image Generation (AREA)
  • Magnetic Resonance Imaging Apparatus (AREA)
  • Measuring And Recording Apparatus For Diagnosis (AREA)

Claims (8)

  1. Appareil de traitement d'image médicale comprenant :
    des moyens d'extraction de région continue (32) qui reçoivent une image médicale tridimensionnelle représentant des bronches et qui extraient une région continue ayant une valeur de voxel correspondant à une région aérienne des bronches ;
    des moyens de construction de structure de graphe (33) qui construisent une structure de graphe correspondant à la région continue ;
    des moyens d'extraction de structure linéaire (34) qui extraient une pluralité de structures linéaires représentant des fragments des bronches par réception de l'image médicale tridimensionnelle, et par analyse d'une structure de densité locale au voisinage de chaque point de l'image médicale tridimensionnelle ; et
    des moyens de reconstitution de structure de graphe (35) qui reconstituent une structure de graphe représentant les bronches en reliant un noeud constituant la structure de graphe correspondant à la région continue et un noeud représentant les structures linéaires l'un à l'autre, et les noeuds représentant les structures linéaires les uns aux autres en utilisant une fonction de coût prédéterminée,
    dans lequel différentes fonctions de coût sont définies pour un premier segment reliant le noeud constituant la structure de graphe correspondant à la région continue et le noeud représentant les structures linéaires et un second segment reliant les noeuds représentant les structures linéaires les uns aux autres, et
    dans lequel la fonction de coût du premier segment est définie de telle manière que le premier segment est plus susceptible d'être relié lorsqu'un changement des valeurs de voxel est plus faible parmi les premiers segments.
  2. Appareil de traitement d'image médicale, selon la revendication 1, dans lequel les moyens d'extraction de structure linéaire (34) extraient les structures linéaires, et calculent les directions d'axe principal des structures linéaires, et
    dans lequel la fonction de coût du second segment est définie de telle manière que le second segment est plus susceptible d'être relié lorsqu'un changement des directions d'axe principal au niveau des noeuds situés aux deux extrémités du second segment est plus faible.
  3. Appareil de traitement d'image médicale, selon la revendication 1 ou 2, l'appareil comprenant en outre :
    des moyens de génération d'image à résolution multiple (31) qui génèrent une pluralité d'images ayant des résolutions différentes les unes des autres à partir de de l'image médicale tridimensionnelle,
    dans lequel les moyens d'extraction de région continue (32) reçoivent une image de résolution inférieure de la pluralité d'images générée, et extraient la région continue de l'image de résolution inférieure, et
    dans lequel les moyens d'extraction de structure linéaire (34) reçoivent une image de résolution supérieure de la pluralité d'images générée, et réalisent l'extraction de la structure linéaire et le calcul de la direction d'axe principal sur l'image de résolution supérieure.
  4. Appareil de traitement d'image médicale, selon l'une quelconque des revendications 1 à 3, dans lequel les moyens d'extraction de région continue (32) extraient la région continue en utilisant l'un d'entre un procédé de découpage graphique, un procédé de fusion des régions et un procédé des lignes de niveau.
  5. Appareil de traitement d'image médicale, selon l'une quelconque des revendications 1 à 4, dans lequel les moyens d'extraction de structure linéaire (34) réalisent l'analyse des valeurs propres de matrice hessienne par rapport à chaque point de l'image médicale tridimensionnelle.
  6. Appareil de traitement d'image médicale, selon l'une quelconque des revendications 1 à 5, dans lequel les moyens de construction de structure de graphe (33) et les moyens de reconstitution de structure de graphe (35) construisent des structures de graphe en utilisant un algorithme de l'arbre maximal.
  7. Procédé de traitement d'image médicale, comprenant les étapes suivantes :
    la réception d'une image médicale tridimensionnelle représentant les bronches et extraction d'une région continue ayant une valeur de voxel correspondant à une région aérienne des bronches ;
    la construction d'une structure de graphe correspondant à la région continue ;
    l'extraction d'une pluralité de structures linéaires représentant des fragments des bronches, par réception de l'image médicale tridimensionnelle et par analyse d'une structure de densité locale au voisinage de chaque point de l'image médicale tridimensionnelle ; et
    la reconstitution d'une structure de graphe représentant les bronches en reliant un noeud constituant la structure de graphe correspondant à la région continue et un noeud représentant les structures linéaires l'un à l'autre, et les noeuds représentant les structures linéaires les uns aux autres en utilisant une fonction de coût prédéterminée,
    dans lequel différentes fonctions de coût sont définies pour un premier segment reliant le noeud constituant la structure de graphe correspondant à la région continue et le noeud représentant les structures linéaires et un second segment reliant les noeuds représentant les structures linéaires les uns aux autres, et
    dans lequel la fonction de coût du premier segment est définie de telle manière que le premier segment soit plus susceptible d'être relié lorsqu'un changement des valeurs de voxel est plus faible parmi les premiers segments.
  8. Support d'enregistrement lisible par ordinateur stockant un programme de traitement d'image médicale pour faire exécuter à un ordinateur les étapes suivantes :
    la réception d'une image médicale tridimensionnelle représentant les bronches, et l'extraction d'une région continue ayant une valeur de voxel correspondant à une région aérienne des bronches ;
    la construction d'une structure de graphe correspondant à la région continue ;
    l'extraction d'une pluralité de structures linéaires représentant des fragments des bronches par réception de l'image médicale tridimensionnelle et par analyse d'une structure de densité locale au voisinage de chaque point de l'image médicale tridimensionnelle ; et
    la reconstitution d'une structure de graphe représentant les bronches en reliant un noeud constituant la structure de graphe correspondant à la région continue et un noeud représentant les structures linéaires l'un à l'autre, et les noeuds représentant les structures linéaires les uns aux autres en utilisant une fonction de coût prédéterminée,
    dans lequel différentes fonctions de coût sont définies pour un premier segment reliant le noeud constituant la structure de graphe correspondant à la région continue et le noeud représentant les structures linéaires et un second segment reliant les noeuds représentant les structures linéaires les uns aux autres, et
    dans lequel la fonction de coût du premier segment est définie de telle manière que le premier segment est plus susceptible d'être relié lorsqu'un changement des valeurs de voxel est plus faible parmi les premiers segments.
EP12774407.6A 2011-04-19 2012-04-12 Appareil, procédé et programme de traitement d'image médicale Not-in-force EP2589340B1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2011092723A JP5701138B2 (ja) 2011-04-19 2011-04-19 医用画像処理装置および方法、並びにプログラム
PCT/JP2012/002536 WO2012144167A1 (fr) 2011-04-19 2012-04-12 Appareil, procédé et programme de traitement d'image médicale

Publications (3)

Publication Number Publication Date
EP2589340A1 EP2589340A1 (fr) 2013-05-08
EP2589340A4 EP2589340A4 (fr) 2013-07-24
EP2589340B1 true EP2589340B1 (fr) 2014-06-18

Family

ID=47041301

Family Applications (1)

Application Number Title Priority Date Filing Date
EP12774407.6A Not-in-force EP2589340B1 (fr) 2011-04-19 2012-04-12 Appareil, procédé et programme de traitement d'image médicale

Country Status (4)

Country Link
US (1) US8634628B2 (fr)
EP (1) EP2589340B1 (fr)
JP (1) JP5701138B2 (fr)
WO (1) WO2012144167A1 (fr)

Families Citing this family (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012078636A1 (fr) 2010-12-07 2012-06-14 University Of Iowa Research Foundation Séparation optimale et conviviale d'arrière-plan d'objet
JP6005663B2 (ja) 2011-01-20 2016-10-12 ユニバーシティ オブ アイオワ リサーチ ファウンデーション 血管画像における動静脈比の自動測定
JP2012223338A (ja) 2011-04-19 2012-11-15 Fujifilm Corp 木構造作成装置および方法ならびにプログラム
JP5391229B2 (ja) * 2011-04-27 2014-01-15 富士フイルム株式会社 木構造抽出装置および方法ならびにプログラム
JP5748693B2 (ja) * 2012-03-13 2015-07-15 富士フイルム株式会社 画像処理装置および方法並びにプログラム
WO2013165614A1 (fr) 2012-05-04 2013-11-07 University Of Iowa Research Foundation Évaluation automatisée de la perte glaucomateuse par tomographie par cohérence optique
WO2014002148A1 (fr) * 2012-06-29 2014-01-03 株式会社島津製作所 Dispositif de traitement d'image
JP5836908B2 (ja) * 2012-09-27 2015-12-24 富士フイルム株式会社 画像処理装置および方法並びにプログラム
JP2014127011A (ja) * 2012-12-26 2014-07-07 Sony Corp 情報処理装置、情報処理方法、およびプログラム
US9672640B2 (en) * 2013-01-24 2017-06-06 Varian Medical Systems International Ag Method for interactive manual matching and real-time projection calculation in imaging
US10360672B2 (en) * 2013-03-15 2019-07-23 University Of Iowa Research Foundation Automated separation of binary overlapping trees
KR101482247B1 (ko) * 2013-08-01 2015-01-14 서울대학교산학협력단 기도 추출 방법 및 그 장치
US9996919B2 (en) 2013-08-01 2018-06-12 Seoul National University R&Db Foundation Method for extracting airways and pulmonary lobes and apparatus therefor
US10410355B2 (en) 2014-03-21 2019-09-10 U.S. Department Of Veterans Affairs Methods and systems for image analysis using non-euclidean deformed graphs
US10152651B2 (en) * 2014-10-31 2018-12-11 Toshiba Medical Systems Corporation Medical image processing apparatus and medical image processing method
US9773325B2 (en) 2015-04-02 2017-09-26 Toshiba Medical Systems Corporation Medical imaging data processing apparatus and method
US10115194B2 (en) 2015-04-06 2018-10-30 IDx, LLC Systems and methods for feature detection in retinal images
JP6867117B2 (ja) * 2016-07-08 2021-04-28 株式会社トプコン 医用画像処理方法及び医用画像処理装置
KR102177951B1 (ko) * 2020-06-16 2020-11-12 주식회사 딥바이오 슬라이드 이미지에 포함된 생체 조직의 길이를 측정하는 방법 및 이를 수행하는 컴퓨팅 시스템
JP2021007017A (ja) * 2020-09-15 2021-01-21 株式会社トプコン 医用画像処理方法及び医用画像処理装置

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030099390A1 (en) * 2001-11-23 2003-05-29 Xiaolan Zeng Lung field segmentation from CT thoracic images
JP4964171B2 (ja) 2008-02-29 2012-06-27 富士フイルム株式会社 対象領域抽出方法および装置ならびにプログラム
EP2279491B1 (fr) * 2008-04-18 2012-02-01 Koninklijke Philips Electronics N.V. Segmentation d'artères pulmonaires
BRPI1004212A2 (pt) 2009-03-03 2016-02-23 Fujifilm Corp aparelho de processamento de imagem, método de processamento de imagem, e programa de processamento de imagem
JP4717935B2 (ja) * 2009-03-23 2011-07-06 富士フイルム株式会社 画像処理装置および方法並びにプログラム

Also Published As

Publication number Publication date
EP2589340A4 (fr) 2013-07-24
EP2589340A1 (fr) 2013-05-08
JP5701138B2 (ja) 2015-04-15
US8634628B2 (en) 2014-01-21
JP2012223315A (ja) 2012-11-15
US20130108133A1 (en) 2013-05-02
WO2012144167A1 (fr) 2012-10-26

Similar Documents

Publication Publication Date Title
EP2589340B1 (fr) Appareil, procédé et programme de traitement d'image médicale
EP2194505B1 (fr) Procédé et dispositif pour segmenter la colonne vertébrale et de l'aorte dans les données d'imagerie médicale en fonction d'un atlas squelettique
JP4708362B2 (ja) コンピュータで実施される方法
EP4014201A1 (fr) Segmentation d'objet tridimensionnelle d'images médicales localisées avec détection d'objet
US8605978B2 (en) Medical image processing apparatus and method, and computer readable recording medium on which is recorded program for the same
US8244012B2 (en) Computer aided detection of pulmonary embolism with local characteristic features in CT angiography
US6766043B2 (en) Pleural nodule detection from CT thoracic images
EP2810598A1 (fr) Dispositif de support chirurgical, procédé de support chirurgical et programme de support chirurgical
EP3444781B1 (fr) Appareil et procédé de traitement d'images
WO2007044508A2 (fr) Systeme et procede de detection de reperes du corps entier et de segmentation et quantification des variations dans des images numeriques
JP2010207572A (ja) 障害のコンピュータ支援検出
US20230005140A1 (en) Automated detection of tumors based on image processing
US20150065868A1 (en) System, method, and computer accessible medium for volumetric texture analysis for computer aided detection and diagnosis of polyps
Ramakrishnan et al. Automatic three-dimensional rib centerline extraction from CT scans for enhanced visualization and anatomical context
Alomari et al. Vertebral column localization, labeling, and segmentation
WO2017011532A1 (fr) Traitement des anomalies candidates dans une imagerie médicale sur la base d'une classification hiérarchique
JP2012504003A (ja) コンピュータを用いて実行される障害検出方法及び装置
Fiori et al. Automatic colon polyp flagging via geometric and texture features
JP2014161388A (ja) 画像処理装置、画像処理方法、画像処理装置の制御プログラム、記録媒体
Lv et al. Centerline extraction based on Hessian matrix and scale space analysis
KR20220115757A (ko) 약지도 딥러닝 인공 지능을 이용한 유방 초음파 진단 방법 및 시스템
Martınez et al. Automatic calculation of pelvis morphology from CT images
Guijarro Martínez et al. Automatic calculation of pelvis morphology from CT images
CN115631149A (zh) 乳房医学图像处理设备和方法
Mendonça et al. Model-Based Methods for Detection of Pulmonary Nodules

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20121227

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

A4 Supplementary search report drawn up and despatched

Effective date: 20130624

RIC1 Information provided on ipc code assigned before grant

Ipc: A61B 6/03 20060101AFI20130618BHEP

Ipc: G06T 7/00 20060101ALI20130618BHEP

Ipc: G06T 1/00 20060101ALI20130618BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

RIC1 Information provided on ipc code assigned before grant

Ipc: G06T 7/00 20060101ALI20131213BHEP

Ipc: G06T 3/00 20060101ALI20131213BHEP

Ipc: G06T 1/00 20060101ALI20131213BHEP

Ipc: A61B 6/03 20060101AFI20131213BHEP

DAX Request for extension of the european patent (deleted)
INTG Intention to grant announced

Effective date: 20140117

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 672880

Country of ref document: AT

Kind code of ref document: T

Effective date: 20140715

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602012002162

Country of ref document: DE

Effective date: 20140807

REG Reference to a national code

Ref country code: NL

Ref legal event code: T3

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140918

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140919

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 672880

Country of ref document: AT

Kind code of ref document: T

Effective date: 20140618

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20141020

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20141018

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602012002162

Country of ref document: DE

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 4

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20150309

Year of fee payment: 4

26N No opposition filed

Effective date: 20150319

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20150408

Year of fee payment: 4

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: LU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150412

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20150430

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20150430

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20150412

REG Reference to a national code

Ref country code: NL

Ref legal event code: MM

Effective date: 20160501

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20160412

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

REG Reference to a national code

Ref country code: FR

Ref legal event code: ST

Effective date: 20161230

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160412

Ref country code: NL

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160501

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160502

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20120412

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

REG Reference to a national code

Ref country code: DE

Ref legal event code: R082

Ref document number: 602012002162

Country of ref document: DE

Representative=s name: KLUNKER IP PATENTANWAELTE PARTG MBB, DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20190402

Year of fee payment: 8

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 602012002162

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20201103