EP2662830A1 - Image processing apparatus, image processing method and program - Google Patents

Image processing apparatus, image processing method and program Download PDF

Info

Publication number
EP2662830A1
EP2662830A1 EP13163985.8A EP13163985A EP2662830A1 EP 2662830 A1 EP2662830 A1 EP 2662830A1 EP 13163985 A EP13163985 A EP 13163985A EP 2662830 A1 EP2662830 A1 EP 2662830A1
Authority
EP
European Patent Office
Prior art keywords
image
images
image processing
processing apparatus
evaluation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP13163985.8A
Other languages
German (de)
French (fr)
Other versions
EP2662830B1 (en
Inventor
Hideaki Miyamoto
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Canon Inc
Original Assignee
Canon Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Canon Inc filed Critical Canon Inc
Publication of EP2662830A1 publication Critical patent/EP2662830A1/en
Application granted granted Critical
Publication of EP2662830B1 publication Critical patent/EP2662830B1/en
Not-in-force legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/0002Inspection of images, e.g. flaw detection
    • G06T7/0012Biomedical image inspection
    • G06T7/0014Biomedical image inspection using an image reference approach
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/0002Inspection of images, e.g. flaw detection
    • G06T7/0012Biomedical image inspection
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B6/00Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
    • A61B6/52Devices using data or image processing specially adapted for radiation diagnosis
    • A61B6/5211Devices using data or image processing specially adapted for radiation diagnosis involving processing of medical diagnostic data
    • A61B6/5229Devices using data or image processing specially adapted for radiation diagnosis involving processing of medical diagnostic data combining image data of a patient, e.g. combining a functional image with an anatomical image
    • A61B6/5235Devices using data or image processing specially adapted for radiation diagnosis involving processing of medical diagnostic data combining image data of a patient, e.g. combining a functional image with an anatomical image combining images from the same or different ionising radiation imaging techniques, e.g. PET and CT
    • A61B6/5241Devices using data or image processing specially adapted for radiation diagnosis involving processing of medical diagnostic data combining image data of a patient, e.g. combining a functional image with an anatomical image combining images from the same or different ionising radiation imaging techniques, e.g. PET and CT combining overlapping images of the same imaging modality, e.g. by stitching
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B6/00Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
    • A61B6/48Diagnostic techniques
    • A61B6/481Diagnostic techniques involving the use of contrast agents
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B6/00Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
    • A61B6/50Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment specially adapted for specific body parts; specially adapted for specific clinical applications
    • A61B6/504Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment specially adapted for specific body parts; specially adapted for specific clinical applications for diagnosis of blood vessels, e.g. by angiography
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10116X-ray image
    • G06T2207/10121Fluoroscopy
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30004Biomedical image processing
    • G06T2207/30021Catheter; Guide wire
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30004Biomedical image processing
    • G06T2207/30101Blood vessel; Artery; Vein; Vascular
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30168Image quality inspection

Definitions

  • the present invention relates to an image processing apparatus, an image processing method and a program.
  • the present invention relates to an image processing technique for selecting an image suitable for use in diagnosis and surgery from a plurality of X-ray digital subtraction angiography images.
  • DSA digital subtraction angiography
  • a DSA image is a subtraction image acquired as follows. Images are acquired before and after a radiopaque dye is injected into an object. The result of subtracting the image acquired before the injection of the radiopaque dye (hereinafter, "mask image”) from the image acquired after the injection of the radiopaque dye (hereinafter, "live image”) is a subtraction image.
  • a DSA image not only keeps a blood vessel region, which is a region of interest used in diagnosis, as a region exhibiting a difference between the images acquired before and after the injection of the radiopaque dye, but also removes all unnecessary region other than the region exhibiting the difference as a background region and shows the background region in a uniform fashion. That is to say, in a DSA image, a region of blood vessels filled with the radiopaque dye, which becomes less visible in a live image due to obstruction by the background region, clearly appears as a subtraction image.
  • Such a DSA image is commonly used in vascular interventional radiology (IVR).
  • Vascular IVR is a technique to insert a catheter into a target organ in a body while checking X-ray fluoroscopy images.
  • Japanese Patent Laid-Open No. 1-221142 discloses the invention of an X-ray image processing apparatus suitable for acquiring orientation images that make it easy to understand how the blood vessels are arranged during surgery and the like.
  • X-ray fluoroscopy images cannot show blood vessels with high X-ray transmittance.
  • a radiopaque dye is administered.
  • vascular IVR if the radiopaque dye is injected continuously until a catheter reaches a target organ, then a large dosage of radiopaque dye would be ultimately administered, which increases the burden on a patient.
  • a predetermined dosage of radiopaque dye is administered prior to fluoroscopy, and images are captured successively thereafter. By subtracting a pre-angiography image from successive post-angiography images, subtraction images are acquired as moving images that serve as DSA moving images.
  • reference image a frame that clearly shows blood vessels through which the catheter should be advanced is selected (hereinafter, "reference image”).
  • the reference image is displayed on a second monitor that is different from a main monitor for displaying fluoroscopic images.
  • the reference image is used as a guide to advance the catheter during fluoroscopy.
  • vascular IVR In general vascular IVR, the following operations are carried out repeatedly until the catheter reaches a target organ: a catheter operation via fluoroscopy, successive imaging under administration of a radiopaque dye, generation of DSA moving images from acquired images, and selection of a frame used as a reference image.
  • the administered dosage of radiopaque dye can be suppressed by using a reference image acquired prior to fluoroscopy instead of operating the catheter under administration of the radiopaque dye during fluoroscopy.
  • a technician searches for an optimal frame through visual examination by, for example, reproducing, frame-forwarding and frame-reversing the DSA moving images.
  • the catheter does not reach a target organ by a catheter operation involving a single generation of DSA moving images and a single fluoroscopy. That is to say, generation of DSA moving images and selection of a reference image are repeated multiple times in accordance with the state of catheter insertion. Therefore, selection of a reference image is a very troublesome operation for the technician and could increase the amount of time required for the treatment.
  • the present invention provides an image processing technique that enables reduction in the effort required to select a reference image to be displayed on a second monitor during vascular IVR.
  • the present invention in its first aspect provides an image processing apparatus as specified in claims 1 to 13.
  • the present invention in its second aspect provides an image processing method as specified in claim 14.
  • the present invention in its third aspect provides a program as specified in claim 15.
  • the present invention enables reduction in the effort required to select a reference image to be displayed on a second monitor during vascular IVR.
  • FIG. 1 illustrates a configuration of an image processing apparatus according to an embodiment.
  • FIG. 2 illustrates a flow of an image processing method according to an embodiment.
  • FIG. 3 schematically illustrates a subtraction image.
  • FIG. 4 schematically illustrates a minimum-value projection image.
  • FIG. 5 illustrates a configuration of an information processing apparatus according to an embodiment.
  • the X-ray image processing apparatus 100 includes an X-ray generation unit 101, a two-dimensional X-ray sensor 104, a preprocessing unit 105, an image storage unit 106, an inter-image subtraction unit 107, an evaluation unit 108, a reference image selection unit 109, and an image display unit 110.
  • the X-ray generation unit 101 can generate X-ray pulses (X-rays) at 3 to 30 pulses per second.
  • the two-dimensional X-ray sensor 104 can capture moving images. More specifically, the two-dimensional X-ray sensor 104 receives X-rays 103 that have been transmitted through an object 102, and captures moving images synchronized with the X-ray pulses.
  • the two-dimensional X-ray sensor 104 also functions as an imaging unit that captures moving images of the object 102 illuminated with the X-rays.
  • the preprocessing unit 105 preprocesses frames of the moving images output from the two-dimensional X-ray sensor 104.
  • the image storage unit 106 stores at least one of the frames of the moving images preprocessed by the preprocessing unit 105 as a mask image acquired before injecting a radiopaque dye.
  • a frame stored as a mask image is, for example, a frame that is acquired immediately after an operation of capturing the moving images is started, or a frame that is automatically acquired immediately before the injection of a radiopaque dye by detecting the injection of the radiopaque dye from the moving images.
  • a frame stored as a mask image may be a frame that is selected by an operator designating a storage timing when starting the injection of the radiopaque dye.
  • a frame used as a mask image may be selected as appropriate from a plurality of frames stored in the image storage unit 106, or may be acquired by compositing a plurality of frames.
  • the inter-image subtraction unit 107 generates images composed of a plurality of frames from X-ray images of the object acquired before and after the injection of the radiopaque dye. More specifically, the inter-image subtraction unit 107 generates subtraction images by subtracting the mask image stored in the image storage unit 106 from live images, which are images (frames) that are acquired after the injection of the radiopaque dye and are output from the preprocessing unit 105, and outputs the subtraction images as frames of DSA moving images.
  • the evaluation unit 108 calculates, from the frames of the DSA moving images output from the inter-image subtraction unit 107, evaluation values indicating the states of the injection of the radiopaque dye for the frames, and outputs the evaluation values.
  • the reference image selection unit 109 selects at least one of the frames of the DSA moving images based on the evaluation values for the frames output from the evaluation unit 108, and outputs the selected frame as a reference image.
  • the image display unit 110 displays the reference image output from the reference image selection unit 109. In the case where a plurality of frames have been selected as reference images, the image display unit 110 can reproduce the reference images one by one in a loop at a predetermined frame rate.
  • the reference image selection unit 109 may output and store a reference image to a storage unit not shown in the figures. In this case, the image display unit 110 can display the reference image stored in the storage unit.
  • the following describes the operations of the evaluation unit 108 and the image display unit 110 that represent characteristic configurations of the present embodiment in connection with the X-ray image processing apparatus 100 configured in the above manner.
  • step S201 the inter-image subtraction unit 107 outputs the frames of the DSA moving images to the evaluation unit 108, and the evaluation unit 108 outputs the evaluation values calculated by evaluating the states of the injection of the radiopaque dye for the frames of the DSA moving images.
  • the states of the injection of the radiopaque dye may be evaluated using various methods. Below, an evaluation method based on the area of a blood vessel region in each frame (DSA image) of the DSA moving images is explained as one example.
  • FIG. 3 is a schematic diagram illustrating a subtraction image acquired by subtracting a mask image from a live image.
  • the subtraction image illustrated in FIG. 3 has pixel values of -1.0 to 1.0. Pixel values close to 1.0 are illustrated in white, while pixel values close to -1.0 are illustrated in black.
  • DSA images are subtraction images acquired by subtracting a mask image which is generated from an X-ray image of an object acquired before the injection of a radiopaque dye, from live images which are X-ray images constituted by a plurality of frames acquired after the injection of the radiopaque dye.
  • a background region 302 that is acquired by removing a blood vessel region 301 into which the radiopaque dye has been introduced (injected) is cancelled out by subtraction processing, and therefore has a pixel value of 0.
  • the blood vessel region 301 mostly has a pixel value that is far from the pixel value of the background region 302 (a negative pixel value in the example of FIG. 3 ).
  • the number of pixels with negative pixel values is counted.
  • DSA moving images with the following conditions: a total number of frames is T, the number of pixel rows in each frame is H, the number of pixel columns in each frame is W, and the t th frame of the DSA moving images is a subtraction image It (where 0 ⁇ t ⁇ T).
  • a pixel (x, y) in the subtraction image It (where 0 ⁇ x ⁇ W, 0 ⁇ y ⁇ H) has a pixel value It (x, y).
  • an evaluation value Sv t for the t th frame of the DSA moving images can be expressed by the following equation.
  • step S202 the evaluation unit 108 outputs the evaluation values to the reference image selection unit 109, and the reference image selection unit 109 selects at least one frame that is suitable as a reference image from the DSA images based on the evaluation values output from the evaluation unit 108.
  • the evaluation unit 108 evaluates the states of the injection of the radiopaque dye based on the area of a blood vessel region in the DSA images, and calculates an evaluation value Sv t as an evaluation value for the t th frame of the DSA moving images (where 0 ⁇ t ⁇ T).
  • the frame t ref selected by the reference image selection unit 109 has the largest evaluation value Sv of all the frames output from the evaluation unit 108.
  • the frame t ref can be expressed by the following equation.
  • the above equation indicates that the t th frame with the largest evaluation value Sv t is selected by the reference image selection unit 109 as the reference image (frame t ref ).
  • the above equation is applicable under the assumption that the reference image selection unit 109 selects one frame, i.e. the t th frame with the largest evaluation value Sv t as the reference image (where 0 ⁇ t ⁇ T). It should be noted, however, that the reference image selection unit 109 is not limited to selecting one frame image as the reference image based on the largest evaluation value Sv. Alternatively, the reference image selection unit 109 may select a plurality of frame images using the evaluation values Sv.
  • a plurality of frames that precede and succeed a frame with the largest evaluation value Sv may be selected.
  • a plurality of frames may be selected in the order of the largest evaluation value Sv to the smallest evaluation value Sv.
  • the evaluation unit 108 evaluates the frames (DSA images) of the DSA moving images based on the area of the blood vessel region in the DSA images, and the reference image selection unit 109 selects (outputs) a frame including a blood vessel region of the largest area as a reference image.
  • the reference image selection unit 109 selects (outputs) a frame including a blood vessel region of the largest area as a reference image.
  • pixel values of pixels included in the frames are compared with a threshold, and evaluation values are calculated using the number of pixels with pixel values within a predetermined range.
  • DSA images include minute noise that has pixel values resembling blood vessels filled with a radiopaque dye.
  • the evaluation unit 108 may calculate evaluation values in consideration of the continuity of pixels in addition to the number of pixels with pixel values within a predetermined range. For example, the evaluation unit 108 may calculate the number of pixels that have pixel values falling within a predetermined range with respect to a threshold and that are continuously arranged as an evaluation value.
  • the use of evaluation values calculated in the above manner can further improve the accuracy of selection of a reference image.
  • the area of the blood vessel region 301 in a DSA image is used as the evaluation value for an ideal subtraction image.
  • a motion artifact 303 could appear due to the movement of the object and the movement of the inserted catheter.
  • the motion artifact 303 has both positive and negative pixel values as illustrated in FIG. 3 , and thus has a large influence on the image quality of a DSA image. It is possible that the motion artifact 303 in a reference image be small.
  • values associated with the motion artifact 303 may be added to the evaluation values calculated by the evaluation unit 108 in step S201.
  • the motion artifact 303 could have both positive and negative pixel values as illustrated in FIG. 3 .
  • the motion artifact 303 could include a region with pixel values close to 1.0 (a white region) and a region with pixel values close to -1.0 (a black region). The following describes a method whereby the number of pixels with positive pixel values is counted as a simple example.
  • an evaluation value Sm t based on the motion artifact 303 can be expressed by the following equation, using the same signs as those used in Expression 1.
  • Sm + Sv an evaluation value S calculated as a sum of an evaluation value Sv and an evaluation value Sm is expressed by the following equation, using a coefficient a greater than 0 as a weight for the evaluation value Sv, and a coefficient b smaller than 0 as a weight for the evaluation value Sm.
  • an evaluation value S t is an evaluation value for the t th frame of the DSA moving images (where 0 ⁇ t ⁇ T)
  • the frame t ref selected by the reference image selection unit 109 in step S202 can be expressed by the following equation.
  • the above equation indicates that the t th frame with the largest evaluation value S t is selected by the reference image selection unit 109 as the reference image (t ref ). That is to say, the reference image selection unit 109 selects a frame including a blood vessel region of a large area and a small motion artifact as a reference image.
  • the number of pixels at the boundary between the radiopaque dye and the background region may be used to calculate an evaluation value.
  • the boundary between the radiopaque dye and the background region can be determined, for example, by applying edge detection processing to a subtraction image using the Canny edge detector and the like and finding out pixel values at the detected edge positions.
  • the evaluation unit 108 can calculate evaluation values based on an imaging portion for which the DSA moving images have been captured, imaging technique information, imaging conditions, and a time period that has elapsed since the injection of the radiopaque dye.
  • a DSA image acquired at the timing when the radiopaque dye has spread throughout the object is selected as a reference image. Therefore, there is a high possibility that a DSA image acquired immediately after the injection of the radiopaque dye, or acquired after a long time period has elapsed since the injection of the radiopaque dye, cannot be used as a reference image.
  • the administered dosage of radiopaque dye determined by the imaging portion and the imaging technique information is Q
  • the velocity of blood flow in the imaging portion is D
  • a frame rate determined by the imaging conditions is R.
  • a range of frames [t start , t end ] that can be used as a reference image when the above conditions are changed is experimentally acquired and stored as a table in advance.
  • the evaluation unit 108 acquires a frame range [t start , t end ] determined by the imaging conditions from this table, and calculates evaluation values S t within this frame range.
  • the reference image selection unit 109 selects the frame t ref as a reference image based on the evaluation values calculated by the evaluation unit 108.
  • data stored in the table is not limited to the range of frames that can be used as a reference image.
  • the table may store values from 0.0 to 1.0 as weights w t corresponding to different sets of the administered dosage Q of radiopaque dye, the velocity D of blood flow in the imaging portion and the frame rate R determined by the imaging conditions. For example, once the administered dosage Q of radiopaque dye, the velocity D of blood flow and the frame rate R have been determined, a corresponding weight w t can be acquired as shown in the following equation.
  • the frame t ref selected by the reference image selection unit 109 in step S202 can be expressed by the following equation.
  • a weight is applied to the evaluation value S t for each frame in correspondence with the administered dosage Q of radiopaque dye, the velocity D of blood flow and the frame rate R, and therefore a reference image can be selected at higher accuracy.
  • An evaluation value may be the result of comparing a minimum-value projection image M generated from the DSA moving images with the frames of the DSA moving images. Provided that each frame of the DSA moving images is It and a value of each pixel therein (x, y) is It (x, y), the minimum-value projection image M is expressed by the following equation.
  • FIG. 4 schematically illustrates a minimum-value projection image 400 generated from DSA images 401, 402 and 403. As illustrated in FIG. 4 , the position where the region of blood vessels filled with the radiopaque dye is shown changes over time in the DSA images 401, 402 and 403 that are included among the DSA moving images as frames. Therefore, in any of these DSA images (frames), only a part of the entire blood vessel region is shown.
  • the minimum-value projection image 400 constituted by these frames is generated by acquiring pixels that each have the smallest pixel value among corresponding pixels in all the frames.
  • any of the frames constituting the DSA moving images includes the region of blood vessels filled with the radiopaque dye, pixel values of this region of blood vessels filled with the radiopaque dye are acquired as the minimum pixel values. Therefore, the minimum-value projection image 400 can show the entire blood vessel region. Evaluation values Sc t based on comparison between this minimum-value projection image M and the frames of the DSA moving images can be calculated using the following equation.
  • the smallest evaluation value Sc t is acquired from a frame of the DSA moving images that is closest to the minimum-value projection image M, that is to say, a frame of the DSA moving images in which the radiopaque dye has spread over a wide range.
  • the evaluation unit 108 may calculate evaluation values based on the addition of a plurality of feature amounts calculated from the frames of the subtraction images. For example, provided that an evaluation value S is acquired by calculating n types of feature amounts (S 1 , S 2 , ... , S n ) from an image I and adding the feature amounts multiplied by weight coefficients (k 1 , k 2 , ... , k n ) through linear combination, the evaluation value S can be expressed by the following equation.
  • each weight coefficient k is, for example, larger than 0 when the corresponding feature amount is large and represents a reference image (S > 0), and smaller than 0 when the corresponding feature amount is small and represents a reference image (S ⁇ 0).
  • the reference image selection unit 109 can select a reference image based on a plurality of feature amounts by selecting a frame in accordance with the evaluation values S (e.g. a frame with the largest evaluation value S).
  • the components illustrated in FIG. 1 may be configured by special-purpose hardware. Functional configurations of the hardware may be realized by software. In this case, the functions of the components illustrated in FIG. 1 may be realized by installing software on an information processing apparatus and implementing an image processing method through the execution of the software using the arithmetic functions of the information processing apparatus. For example, through the execution of the software, a subtraction step, an evaluation step and a selection step are executed for the frames of the moving images output from the two-dimensional X-ray sensor 104.
  • FIG. 5 is a block diagram illustrating a hardware configuration of an information processing apparatus and configurations of peripheral apparatuses therefor.
  • An information processing apparatus 1000 and an image capturing apparatus 2000 are connected to each other and are configured to perform data communication with each other.
  • a CPU 1010 can control the entirety of the information processing apparatus 1000 using programs and data stored in a RAM 1020 and a ROM 1030.
  • the CPU 1010 can also execute arithmetic processing related to predetermined image processing by executing programs.
  • the RAM 1020 includes an area for temporarily storing programs and data loaded from a magneto-optical disc 1060 and a hard disk drive 1050.
  • the RAM 1020 also includes an area for temporarily storing data of X-ray fluoroscopy moving images acquired from the image capturing apparatus 2000.
  • the RAM 1020 also includes a work area used when the CPU 1010 executes various types of processing.
  • the ROM 1030 stores data of the settings of the information processing apparatus 1000, boot programs, and the like.
  • the hard disk drive 1050 is one example of an information storage medium, and stores programs and data for causing the CPU 1010 to execute processing of an operating system (OS) and the components illustrated in FIG. 1 . They are loaded to the RAM 1020 as appropriate in accordance with control by the CPU 1010 and are executed by the CPU 1010. Data of the X-ray fluoroscopy moving images may also be stored in the hard disk drive 1050.
  • OS operating system
  • Data of the X-ray fluoroscopy moving images may also be stored in the hard disk drive 1050.
  • the magneto-optical disc 1060 is one example of an information storage medium, and can store a part or all of the programs and data stored in the hard disk drive 1050.
  • a mouse 1070 and a keyboard 1080 can input various types of instructions to the CPU 1010 by being operated by an operator of the information processing apparatus 1000.
  • a printer 1090 can print/output images displayed on the image display unit 110 on/to a recording medium.
  • a display apparatus 1100 is constituted by a CRT, a liquid crystal display, or the like, and can display the result of processing of the CPU 1010 using images and text.
  • the display apparatus 1100 can display images that have been processed by the components illustrated in FIG. 1 and ultimately output from the image display unit 110.
  • the image display unit 110 functions as a display control unit for displaying images on the display apparatus 1100.
  • a bus 1040 which connects between the components of the information processing apparatus 1000, enables them to exchange data with one another.
  • the image capturing apparatus 2000 is an apparatus that can capture moving images, such as an X-ray fluoroscope.
  • the image capturing apparatus 2000 transmits captured image data to the information processing apparatus 1000.
  • the image capturing apparatus 2000 may transmit a plurality of pieces of image data collectively to the information processing apparatus 1000, or may transmit a piece of image data each time it captures an image.
  • aspects of the present invention can also be realized by a computer of a system or apparatus (or devices such as a CPU or MPU) that reads out and executes a program recorded on a memory device to perform the functions of the above-described embodiment(s), and by a method, the steps of which are performed by a computer of a system or apparatus by, for example, reading out and executing a program recorded on a memory device to perform the functions of the above-described embodiment(s).
  • the program is provided to the computer for example via a network or from a recording medium of various types serving as the memory device (e.g., computer-readable medium).

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Biophysics (AREA)
  • High Energy & Nuclear Physics (AREA)
  • Optics & Photonics (AREA)
  • Pathology (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Molecular Biology (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Apparatus For Radiation Diagnosis (AREA)

Abstract

An image processing apparatus includes: subtraction means that generates subtraction images constituted by a plurality of frames from X-ray images of an object acquired before and after injecting a radiopaque dye; evaluation means that calculates evaluation values indicating states of the injection of the radiopaque dye for the frames constituting the subtraction images; and selection means that selects at least one of the frames constituting the subtraction images using the evaluation values and outputs the selected frame as a reference image.

Description

    BACKGROUND OF THE INVENTION Field of the Invention
  • The present invention relates to an image processing apparatus, an image processing method and a program. In particular, the present invention relates to an image processing technique for selecting an image suitable for use in diagnosis and surgery from a plurality of X-ray digital subtraction angiography images.
  • Description of the Related Art
  • Recent advancement in digital technology has made it common to apply digital processing to images in the medical field. In place of the conventional X-ray imaging that uses films for X-ray diagnosis, two-dimensional X-ray sensors that output X-ray images as digital images have become widespread. The application of digital image processing to digital images output from the two-dimensional X-ray sensors comes in a broad range.
  • One example of suitable application of such digital image processing is digital subtraction angiography (DSA) processing for acquiring a DSA image. A DSA image is a subtraction image acquired as follows. Images are acquired before and after a radiopaque dye is injected into an object. The result of subtracting the image acquired before the injection of the radiopaque dye (hereinafter, "mask image") from the image acquired after the injection of the radiopaque dye (hereinafter, "live image") is a subtraction image. A DSA image not only keeps a blood vessel region, which is a region of interest used in diagnosis, as a region exhibiting a difference between the images acquired before and after the injection of the radiopaque dye, but also removes all unnecessary region other than the region exhibiting the difference as a background region and shows the background region in a uniform fashion. That is to say, in a DSA image, a region of blood vessels filled with the radiopaque dye, which becomes less visible in a live image due to obstruction by the background region, clearly appears as a subtraction image. Such a DSA image is commonly used in vascular interventional radiology (IVR). Vascular IVR is a technique to insert a catheter into a target organ in a body while checking X-ray fluoroscopy images.
  • Regarding DSA processing, for example, Japanese Patent Laid-Open No. 1-221142 discloses the invention of an X-ray image processing apparatus suitable for acquiring orientation images that make it easy to understand how the blood vessels are arranged during surgery and the like.
  • Normally, X-ray fluoroscopy images cannot show blood vessels with high X-ray transmittance. To show such blood vessels, a radiopaque dye is administered. However, in vascular IVR, if the radiopaque dye is injected continuously until a catheter reaches a target organ, then a large dosage of radiopaque dye would be ultimately administered, which increases the burden on a patient. In view of this, in general vascular IVR, a predetermined dosage of radiopaque dye is administered prior to fluoroscopy, and images are captured successively thereafter. By subtracting a pre-angiography image from successive post-angiography images, subtraction images are acquired as moving images that serve as DSA moving images. From the DSA moving images, a frame that clearly shows blood vessels through which the catheter should be advanced is selected (hereinafter, "reference image"). The reference image is displayed on a second monitor that is different from a main monitor for displaying fluoroscopic images. The reference image is used as a guide to advance the catheter during fluoroscopy.
  • In general vascular IVR, the following operations are carried out repeatedly until the catheter reaches a target organ: a catheter operation via fluoroscopy, successive imaging under administration of a radiopaque dye, generation of DSA moving images from acquired images, and selection of a frame used as a reference image. The administered dosage of radiopaque dye can be suppressed by using a reference image acquired prior to fluoroscopy instead of operating the catheter under administration of the radiopaque dye during fluoroscopy.
  • Conventionally, in order to select a reference image from DSA moving images, a technician searches for an optimal frame through visual examination by, for example, reproducing, frame-forwarding and frame-reversing the DSA moving images. In general vascular IVR, the catheter does not reach a target organ by a catheter operation involving a single generation of DSA moving images and a single fluoroscopy. That is to say, generation of DSA moving images and selection of a reference image are repeated multiple times in accordance with the state of catheter insertion. Therefore, selection of a reference image is a very troublesome operation for the technician and could increase the amount of time required for the treatment.
  • In view of the above problems, the present invention provides an image processing technique that enables reduction in the effort required to select a reference image to be displayed on a second monitor during vascular IVR.
  • SUMMARY OF THE INVENTION
  • The present invention in its first aspect provides an image processing apparatus as specified in claims 1 to 13.
  • The present invention in its second aspect provides an image processing method as specified in claim 14.
  • The present invention in its third aspect provides a program as specified in claim 15.
  • The present invention enables reduction in the effort required to select a reference image to be displayed on a second monitor during vascular IVR.
  • Further features of the present invention will become apparent from the following description of embodiments (with reference to the attached drawings). Each of the embodiments of the present invention described below can be implemented solely or as a combination of a plurality of the embodiments or features thereof where necessary or where the combination of elements or features from individual embodiments in a single embodiment is beneficial.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 illustrates a configuration of an image processing apparatus according to an embodiment.
  • FIG. 2 illustrates a flow of an image processing method according to an embodiment.
  • FIG. 3 schematically illustrates a subtraction image.
  • FIG. 4 schematically illustrates a minimum-value projection image.
  • FIG. 5 illustrates a configuration of an information processing apparatus according to an embodiment.
  • DESCRIPTION OF THE EMBODIMENTS
  • The following describes one example of embodiments of the present invention with reference to the drawings.
  • (First Embodiment)
  • A description is now given of an image processing apparatus according to the first embodiment of the present invention (hereinafter, "X-ray image processing apparatus 100") with reference to FIG. 1. The X-ray image processing apparatus 100 includes an X-ray generation unit 101, a two-dimensional X-ray sensor 104, a preprocessing unit 105, an image storage unit 106, an inter-image subtraction unit 107, an evaluation unit 108, a reference image selection unit 109, and an image display unit 110.
  • The X-ray generation unit 101 can generate X-ray pulses (X-rays) at 3 to 30 pulses per second. The two-dimensional X-ray sensor 104 can capture moving images. More specifically, the two-dimensional X-ray sensor 104 receives X-rays 103 that have been transmitted through an object 102, and captures moving images synchronized with the X-ray pulses. The two-dimensional X-ray sensor 104 also functions as an imaging unit that captures moving images of the object 102 illuminated with the X-rays.
  • The preprocessing unit 105 preprocesses frames of the moving images output from the two-dimensional X-ray sensor 104. The image storage unit 106 stores at least one of the frames of the moving images preprocessed by the preprocessing unit 105 as a mask image acquired before injecting a radiopaque dye. Note that a frame stored as a mask image is, for example, a frame that is acquired immediately after an operation of capturing the moving images is started, or a frame that is automatically acquired immediately before the injection of a radiopaque dye by detecting the injection of the radiopaque dye from the moving images. Alternatively, a frame stored as a mask image may be a frame that is selected by an operator designating a storage timing when starting the injection of the radiopaque dye. Alternatively, a frame used as a mask image may be selected as appropriate from a plurality of frames stored in the image storage unit 106, or may be acquired by compositing a plurality of frames.
  • The inter-image subtraction unit 107 generates images composed of a plurality of frames from X-ray images of the object acquired before and after the injection of the radiopaque dye. More specifically, the inter-image subtraction unit 107 generates subtraction images by subtracting the mask image stored in the image storage unit 106 from live images, which are images (frames) that are acquired after the injection of the radiopaque dye and are output from the preprocessing unit 105, and outputs the subtraction images as frames of DSA moving images.
  • The evaluation unit 108 calculates, from the frames of the DSA moving images output from the inter-image subtraction unit 107, evaluation values indicating the states of the injection of the radiopaque dye for the frames, and outputs the evaluation values.
  • The reference image selection unit 109 selects at least one of the frames of the DSA moving images based on the evaluation values for the frames output from the evaluation unit 108, and outputs the selected frame as a reference image.
  • The image display unit 110 displays the reference image output from the reference image selection unit 109. In the case where a plurality of frames have been selected as reference images, the image display unit 110 can reproduce the reference images one by one in a loop at a predetermined frame rate. The reference image selection unit 109 may output and store a reference image to a storage unit not shown in the figures. In this case, the image display unit 110 can display the reference image stored in the storage unit.
  • With reference to a flowchart of FIG. 2, the following describes the operations of the evaluation unit 108 and the image display unit 110 that represent characteristic configurations of the present embodiment in connection with the X-ray image processing apparatus 100 configured in the above manner.
  • In step S201, the inter-image subtraction unit 107 outputs the frames of the DSA moving images to the evaluation unit 108, and the evaluation unit 108 outputs the evaluation values calculated by evaluating the states of the injection of the radiopaque dye for the frames of the DSA moving images. The states of the injection of the radiopaque dye may be evaluated using various methods. Below, an evaluation method based on the area of a blood vessel region in each frame (DSA image) of the DSA moving images is explained as one example.
  • FIG. 3 is a schematic diagram illustrating a subtraction image acquired by subtracting a mask image from a live image. The subtraction image illustrated in FIG. 3 has pixel values of -1.0 to 1.0. Pixel values close to 1.0 are illustrated in white, while pixel values close to -1.0 are illustrated in black. DSA images are subtraction images acquired by subtracting a mask image which is generated from an X-ray image of an object acquired before the injection of a radiopaque dye, from live images which are X-ray images constituted by a plurality of frames acquired after the injection of the radiopaque dye. Therefore, in an ideal subtraction image, a background region 302 that is acquired by removing a blood vessel region 301 into which the radiopaque dye has been introduced (injected) is cancelled out by subtraction processing, and therefore has a pixel value of 0. The blood vessel region 301 mostly has a pixel value that is far from the pixel value of the background region 302 (a negative pixel value in the example of FIG. 3).
  • Therefore, with a method of evaluating the states of the injection of the radiopaque dye based on the area of the blood vessel region 301 in DSA images, the number of pixels with negative pixel values is counted. Assume there are DSA moving images with the following conditions: a total number of frames is T, the number of pixel rows in each frame is H, the number of pixel columns in each frame is W, and the tth frame of the DSA moving images is a subtraction image It (where 0 ≤ t < T). Also assume that a pixel (x, y) in the subtraction image It (where 0 ≤ x < W, 0 ≤ y < H) has a pixel value It (x, y). Also assume that a threshold to separate the background region 302 with a pixel value of 0 from the blood vessel region 301 is generalized as thresh. Under the above relationships, an evaluation value Svt for the tth frame of the DSA moving images can be expressed by the following equation.
  • Sv t = y = 0 y < H x = 0 x < W f t x y H × W f t x y = { 1 I t x y < thresh 0 I t x y thresh
    Figure imgb0001
  • In step S202, the evaluation unit 108 outputs the evaluation values to the reference image selection unit 109, and the reference image selection unit 109 selects at least one frame that is suitable as a reference image from the DSA images based on the evaluation values output from the evaluation unit 108.
  • According to the above-described example, in step S201, the evaluation unit 108 evaluates the states of the injection of the radiopaque dye based on the area of a blood vessel region in the DSA images, and calculates an evaluation value Svt as an evaluation value for the tth frame of the DSA moving images (where 0 ≤ t < T).
  • An image showing the state where the radiopaque dye is widely spread in a blood vessel region is suitable as a reference image. Therefore, the frame tref selected by the reference image selection unit 109 has the largest evaluation value Sv of all the frames output from the evaluation unit 108. Using an evaluation value output from the evaluation unit 108, the frame tref can be expressed by the following equation.
  • t ref = t 0 t < T : Sv t = max Sv t 0 t < T
    Figure imgb0002
  • The above equation indicates that the tth frame with the largest evaluation value Svt is selected by the reference image selection unit 109 as the reference image (frame tref). The above equation is applicable under the assumption that the reference image selection unit 109 selects one frame, i.e. the tth frame with the largest evaluation value Svt as the reference image (where 0 ≤ t < T). It should be noted, however, that the reference image selection unit 109 is not limited to selecting one frame image as the reference image based on the largest evaluation value Sv. Alternatively, the reference image selection unit 109 may select a plurality of frame images using the evaluation values Sv. For example, in the case where two or more frames are selected as reference images and reproduced one by one in a loop at a predetermined frame rate, a plurality of frames that precede and succeed a frame with the largest evaluation value Sv may be selected. Alternatively, a plurality of frames may be selected in the order of the largest evaluation value Sv to the smallest evaluation value Sv.
  • In the present embodiment, the evaluation unit 108 evaluates the frames (DSA images) of the DSA moving images based on the area of the blood vessel region in the DSA images, and the reference image selection unit 109 selects (outputs) a frame including a blood vessel region of the largest area as a reference image. By outputting this reference image to the image display unit 110, the problem of frame selection can be alleviated for the operator of the X-ray image processing apparatus.
  • In the above-described example, pixel values of pixels included in the frames (DSA images) are compared with a threshold, and evaluation values are calculated using the number of pixels with pixel values within a predetermined range. It should be noted, however, that DSA images include minute noise that has pixel values resembling blood vessels filled with a radiopaque dye. In view of this, the evaluation unit 108 may calculate evaluation values in consideration of the continuity of pixels in addition to the number of pixels with pixel values within a predetermined range. For example, the evaluation unit 108 may calculate the number of pixels that have pixel values falling within a predetermined range with respect to a threshold and that are continuously arranged as an evaluation value. As a region of blood vessels filled with a radiopaque dye can be thought as a region that is continuous with the tip of the catheter, the use of evaluation values calculated in the above manner can further improve the accuracy of selection of a reference image.
  • Regarding the examples of the blood vessel region 301 and the background region 302 illustrated in FIG. 3, the area of the blood vessel region 301 in a DSA image is used as the evaluation value for an ideal subtraction image. However, in an actual subtraction image, a motion artifact 303 could appear due to the movement of the object and the movement of the inserted catheter. For example, the motion artifact 303 has both positive and negative pixel values as illustrated in FIG. 3, and thus has a large influence on the image quality of a DSA image. It is possible that the motion artifact 303 in a reference image be small. In order to select a reference image in consideration of the existence of the motion artifact 303, values associated with the motion artifact 303 may be added to the evaluation values calculated by the evaluation unit 108 in step S201.
  • There are various methods of calculating evaluation values that take the motion artifact 303 into consideration. Unlike the blood vessel region 301 that mostly has negative pixel values, the motion artifact 303 could have both positive and negative pixel values as illustrated in FIG. 3. For example, the motion artifact 303 could include a region with pixel values close to 1.0 (a white region) and a region with pixel values close to -1.0 (a black region). The following describes a method whereby the number of pixels with positive pixel values is counted as a simple example. Provided that a threshold for separating the background region 302 from the motion artifact 303 is thresh2, an evaluation value Smt based on the motion artifact 303 can be expressed by the following equation, using the same signs as those used in Expression 1.
  • Sm t = y = 0 y < H x = 0 x < W f t x y H × W f t x y = { 1 I t x y > thresh 2 0 I t x y thresh 2
    Figure imgb0003
  • The result of adding this evaluation value Sm based on the motion artifact 303 and an evaluation value Sv based on the area of the blood vessel region 301 is an evaluation value S (= Sm + Sv) to be ultimately calculated. When an evaluation value Sm based on the motion artifact 303 is used, it is possible that the motion artifact 303 in a reference image be small. In view of this, an evaluation value S calculated as a sum of an evaluation value Sv and an evaluation value Sm is expressed by the following equation, using a coefficient a greater than 0 as a weight for the evaluation value Sv, and a coefficient b smaller than 0 as a weight for the evaluation value Sm.
  • S = a × Sv + b × Sm a > 0 , b < 0
    Figure imgb0004
  • Provided that an evaluation value St is an evaluation value for the tth frame of the DSA moving images (where 0 ≤ t < T), the frame tref selected by the reference image selection unit 109 in step S202 can be expressed by the following equation.
  • t ref = t 0 t < T : S t = max S t 0 t < T
    Figure imgb0005
  • Similarly to Expression 2, the above equation indicates that the tth frame with the largest evaluation value St is selected by the reference image selection unit 109 as the reference image (tref). That is to say, the reference image selection unit 109 selects a frame including a blood vessel region of a large area and a small motion artifact as a reference image.
  • The number of pixels at the boundary between the radiopaque dye and the background region may be used to calculate an evaluation value. The boundary between the radiopaque dye and the background region can be determined, for example, by applying edge detection processing to a subtraction image using the Canny edge detector and the like and finding out pixel values at the detected edge positions. By calculating an evaluation value using the number of pixels at the boundary, the complexity of structures of fine blood vessels that cannot be evaluated based only on the aforementioned area of the blood vessel region can be added to the evaluation value, thereby enabling selection of a more suitable reference image.
  • Furthermore, the evaluation unit 108 can calculate evaluation values based on an imaging portion for which the DSA moving images have been captured, imaging technique information, imaging conditions, and a time period that has elapsed since the injection of the radiopaque dye. A DSA image acquired at the timing when the radiopaque dye has spread throughout the object is selected as a reference image. Therefore, there is a high possibility that a DSA image acquired immediately after the injection of the radiopaque dye, or acquired after a long time period has elapsed since the injection of the radiopaque dye, cannot be used as a reference image. For example, assume that the administered dosage of radiopaque dye determined by the imaging portion and the imaging technique information is Q, the velocity of blood flow in the imaging portion is D, and a frame rate determined by the imaging conditions is R. A range of frames [tstart, tend] that can be used as a reference image when the above conditions are changed is experimentally acquired and stored as a table in advance.
  • table Q D R = t start t end
    Figure imgb0006
  • During the actual imaging, the evaluation unit 108 acquires a frame range [tstart, tend] determined by the imaging conditions from this table, and calculates evaluation values St within this frame range. The reference image selection unit 109 then selects the frame tref as a reference image based on the evaluation values calculated by the evaluation unit 108. With this configuration, a range of frame search can be restricted, and therefore the speed of processing can be accelerated. Note that data stored in the table is not limited to the range of frames that can be used as a reference image. For example, the table may store values from 0.0 to 1.0 as weights wt corresponding to different sets of the administered dosage Q of radiopaque dye, the velocity D of blood flow in the imaging portion and the frame rate R determined by the imaging conditions. For example, once the administered dosage Q of radiopaque dye, the velocity D of blood flow and the frame rate R have been determined, a corresponding weight wt can be acquired as shown in the following equation.
  • table Q D R = w t
    Figure imgb0007
  • Using this weight wt, the frame tref selected by the reference image selection unit 109 in step S202 can be expressed by the following equation.
  • t ref = t 0 t < T : w t × S t = max 0 t < T w t × S t
    Figure imgb0008
  • With the above configuration, a weight is applied to the evaluation value St for each frame in correspondence with the administered dosage Q of radiopaque dye, the velocity D of blood flow and the frame rate R, and therefore a reference image can be selected at higher accuracy.
  • An evaluation value may be the result of comparing a minimum-value projection image M generated from the DSA moving images with the frames of the DSA moving images. Provided that each frame of the DSA moving images is It and a value of each pixel therein (x, y) is It (x, y), the minimum-value projection image M is expressed by the following equation.
  • M x y min 0 t < T I t x y
    Figure imgb0009
  • The pixel value M (x, y) of each pixel (x, y) in the minimum-value projection image M is the smallest of pixel values of corresponding pixels in all the frames of the DSA moving images. FIG. 4 schematically illustrates a minimum-value projection image 400 generated from DSA images 401, 402 and 403.
    As illustrated in FIG. 4, the position where the region of blood vessels filled with the radiopaque dye is shown changes over time in the DSA images 401, 402 and 403 that are included among the DSA moving images as frames. Therefore, in any of these DSA images (frames), only a part of the entire blood vessel region is shown. The minimum-value projection image 400 constituted by these frames is generated by acquiring pixels that each have the smallest pixel value among corresponding pixels in all the frames. When any of the frames constituting the DSA moving images includes the region of blood vessels filled with the radiopaque dye, pixel values of this region of blood vessels filled with the radiopaque dye are acquired as the minimum pixel values. Therefore, the minimum-value projection image 400 can show the entire blood vessel region. Evaluation values Sct based on comparison between this minimum-value projection image M and the frames of the DSA moving images can be calculated using the following equation.
  • Sc t = y = 0 y < H x = 0 x < W I t x y - M x y
    Figure imgb0010
  • The smallest evaluation value Sct is acquired from a frame of the DSA moving images that is closest to the minimum-value projection image M, that is to say, a frame of the DSA moving images in which the radiopaque dye has spread over a wide range. In view of this, by adding the result of multiplication by a predetermined coefficient smaller than 0 to the evaluation value in a manner similar to the evaluation value Sm based on the motion artifact, a more suitable reference image can be selected.
  • Although several types of calculation of evaluation values have been described above as examples, evaluation values that can be used in the present invention are not limited to the above examples. For example, the evaluation unit 108 may calculate evaluation values based on the addition of a plurality of feature amounts calculated from the frames of the subtraction images. For example, provided that an evaluation value S is acquired by calculating n types of feature amounts (S1, S2, ... , Sn) from an image I and adding the feature amounts multiplied by weight coefficients (k1, k2, ... , kn) through linear combination, the evaluation value S can be expressed by the following equation.
  • S = k 1 × S 1 + k 2 × S 2 + + k n × S n
    Figure imgb0011
  • The value of each weight coefficient k is, for example, larger than 0 when the corresponding feature amount is large and represents a reference image (S > 0), and smaller than 0 when the corresponding feature amount is small and represents a reference image (S < 0).
  • In this case, the reference image selection unit 109 can select a reference image based on a plurality of feature amounts by selecting a frame in accordance with the evaluation values S (e.g. a frame with the largest evaluation value S).
  • The components illustrated in FIG. 1 may be configured by special-purpose hardware. Functional configurations of the hardware may be realized by software. In this case, the functions of the components illustrated in FIG. 1 may be realized by installing software on an information processing apparatus and implementing an image processing method through the execution of the software using the arithmetic functions of the information processing apparatus. For example, through the execution of the software, a subtraction step, an evaluation step and a selection step are executed for the frames of the moving images output from the two-dimensional X-ray sensor 104.
  • FIG. 5 is a block diagram illustrating a hardware configuration of an information processing apparatus and configurations of peripheral apparatuses therefor. An information processing apparatus 1000 and an image capturing apparatus 2000 are connected to each other and are configured to perform data communication with each other.
  • (Information Processing Apparatus)
  • A CPU 1010 can control the entirety of the information processing apparatus 1000 using programs and data stored in a RAM 1020 and a ROM 1030. The CPU 1010 can also execute arithmetic processing related to predetermined image processing by executing programs.
  • The RAM 1020 includes an area for temporarily storing programs and data loaded from a magneto-optical disc 1060 and a hard disk drive 1050. The RAM 1020 also includes an area for temporarily storing data of X-ray fluoroscopy moving images acquired from the image capturing apparatus 2000. The RAM 1020 also includes a work area used when the CPU 1010 executes various types of processing.
  • The ROM 1030 stores data of the settings of the information processing apparatus 1000, boot programs, and the like.
  • The hard disk drive 1050 is one example of an information storage medium, and stores programs and data for causing the CPU 1010 to execute processing of an operating system (OS) and the components illustrated in FIG. 1. They are loaded to the RAM 1020 as appropriate in accordance with control by the CPU 1010 and are executed by the CPU 1010. Data of the X-ray fluoroscopy moving images may also be stored in the hard disk drive 1050.
  • The magneto-optical disc 1060 is one example of an information storage medium, and can store a part or all of the programs and data stored in the hard disk drive 1050.
  • A mouse 1070 and a keyboard 1080 can input various types of instructions to the CPU 1010 by being operated by an operator of the information processing apparatus 1000.
  • A printer 1090 can print/output images displayed on the image display unit 110 on/to a recording medium.
  • A display apparatus 1100 is constituted by a CRT, a liquid crystal display, or the like, and can display the result of processing of the CPU 1010 using images and text. For example, the display apparatus 1100 can display images that have been processed by the components illustrated in FIG. 1 and ultimately output from the image display unit 110. In this case, the image display unit 110 functions as a display control unit for displaying images on the display apparatus 1100. A bus 1040, which connects between the components of the information processing apparatus 1000, enables them to exchange data with one another.
  • (Image Capturing Apparatus)
  • A description is now given of the image capturing apparatus 2000. The image capturing apparatus 2000 is an apparatus that can capture moving images, such as an X-ray fluoroscope. The image capturing apparatus 2000 transmits captured image data to the information processing apparatus 1000. Note that the image capturing apparatus 2000 may transmit a plurality of pieces of image data collectively to the information processing apparatus 1000, or may transmit a piece of image data each time it captures an image.
  • Other Embodiments
  • Aspects of the present invention can also be realized by a computer of a system or apparatus (or devices such as a CPU or MPU) that reads out and executes a program recorded on a memory device to perform the functions of the above-described embodiment(s), and by a method, the steps of which are performed by a computer of a system or apparatus by, for example, reading out and executing a program recorded on a memory device to perform the functions of the above-described embodiment(s). For this purpose, the program is provided to the computer for example via a network or from a recording medium of various types serving as the memory device (e.g., computer-readable medium).
  • While the present invention has been described with reference to embodiments, it is to be understood that the invention is not limited to the disclosed embodiments.

Claims (15)

  1. An image processing apparatus comprising:
    subtraction means (107) arranged to generate digital subtraction angiography images constituted by a plurality of frames from X-ray images of an object acquired before and after injecting a radiopaque dye into the object;
    evaluation means (108) arranged to calculate evaluation values indicating the state of the injection of the radiopaque dye for each of the frames constituting the digital subtraction angiography images; and
    selection means (109) arranged to select at least one of digital subtraction angiography image using the evaluation values and to output the selected, at least one, digital subtraction angiography image as a reference image.
  2. The image processing apparatus according to Claim 1,
    wherein the subtraction means (107) is arranged to generate each of the digital subtraction angiography images by subtracting a mask image from live images, the mask image being generated from an X-ray image of the object acquired before injecting the radiopaque dye, and the live images being X-ray images constituted by a plurality of frames acquired after injection of the radiopaque dye.
  3. The image processing apparatus according to Claim 1 or 2,
    wherein the evaluation means (108) is arranged to calculate the evaluation values from the frames constituting the digital subtraction angiography images using the number of pixels that have pixel values falling within a predetermined range with respect to a threshold.
  4. The image processing apparatus according to Claim 3,
    wherein the evaluation means (108) is arranged to calculate the evaluation values from the frames constituting the digital subtraction angiography images using the number of pixels that have pixel values falling within a predetermined range with respect to a threshold and that are continuously arranged.
  5. The image processing apparatus according to Claim 1 or 2,
    wherein the evaluation means (108) is arranged to calculate the evaluation values from the frames constituting the digital subtraction angiography images using the number of pixels at a boundary between the radiopaque dye and a background region.
  6. The image processing apparatus according to Claim 1 or 2,
    wherein the evaluation means (108) is arranged to calculate the evaluation values based on an imaging portion for which the digital subtraction angiography images have been generated, imaging technique information, imaging conditions, and a time period that has elapsed since the injection of the radiopaque dye.
  7. The image processing apparatus according to Claim 6,
    wherein the evaluation means (108) is arranged to acquire a frame range for the digital subtraction angiography images based on the imaging conditions and to calculate the evaluation values within the frame range.
  8. The image processing apparatus according to Claim 1 or 2,
    wherein the evaluation means (108) is arranged to calculate the evaluation values by comparing a minimum-value projection image with the digital subtraction angiography images, the minimum-value projection image being constituted by pixels that each have the smallest pixel value among corresponding pixels in all the frames constituting the digital subtraction angiography images.
  9. The image processing apparatus according to Claim 1 or 2,
    wherein the evaluation means (108) is arranged to calculate a plurality of feature amounts from the frames constituting the digital subtraction angiography images, and to calculate the evaluation values based on addition of the calculated plurality of feature amounts.
  10. The image processing apparatus according to Claim 9,
    wherein the evaluation means (108) is arranged to calculate the evaluation values by adding values acquired by multiplying the plurality of feature amounts calculated from the frames of the digital subtraction angiography images by weight coefficients through linear combination.
  11. The image processing apparatus according to any of Claims 1 to 10, further comprising
    image display means (110) arranged to display the reference image.
  12. The image processing apparatus according to Claim 11,
    wherein when a plurality of frames have been selected as reference images, the image display means (110) being arranged to display the reference images one by one at a predetermined frame rate.
  13. The image processing apparatus according to any of Claims 1 to 12, further comprising
    storage means (106) arranged to store the reference image.
  14. An image processing method for an image processing apparatus, said method comprising:
    a subtraction step of generating digital subtraction angiography images constituted by a plurality of frames from X-ray images of an object acquired before and after injecting a radiopaque dye into the object;
    an evaluation step of calculating evaluation values indicating the states of the injection of the radiopaque dye for each of the frames constituting the digital subtraction angiography images; and
    a selection step of selecting at least one digital subtraction angiography image using the evaluation values and outputting the selected, at least one, digital subtraction angiography image as a reference image.
  15. A program that, when executed by an image processing apparatus, causes the image processing apparatus to perform an image processing method according to claim 14.
EP13163985.8A 2012-05-07 2013-04-16 Selection of a reference image in interventional vascular radiology Not-in-force EP2662830B1 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP2012106311A JP6021420B2 (en) 2012-05-07 2012-05-07 Image processing apparatus, image processing method, and program

Publications (2)

Publication Number Publication Date
EP2662830A1 true EP2662830A1 (en) 2013-11-13
EP2662830B1 EP2662830B1 (en) 2018-10-10

Family

ID=48184034

Family Applications (1)

Application Number Title Priority Date Filing Date
EP13163985.8A Not-in-force EP2662830B1 (en) 2012-05-07 2013-04-16 Selection of a reference image in interventional vascular radiology

Country Status (4)

Country Link
US (1) US9396535B2 (en)
EP (1) EP2662830B1 (en)
JP (1) JP6021420B2 (en)
CN (1) CN103385692B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10499864B2 (en) 2014-09-22 2019-12-10 Koninklijke Philips N.V. Contrast arrival detection

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5981220B2 (en) * 2012-05-21 2016-08-31 東芝メディカルシステムズ株式会社 Medical image processing apparatus and X-ray imaging apparatus
JP6071444B2 (en) 2012-11-07 2017-02-01 キヤノン株式会社 Image processing apparatus, operation method thereof, and program
JP6371515B2 (en) 2013-11-13 2018-08-08 キヤノン株式会社 X-ray image processing apparatus, X-ray image processing method, and program
JP6305747B2 (en) * 2013-12-11 2018-04-04 キヤノンメディカルシステムズ株式会社 Medical image processing apparatus and X-ray diagnostic apparatus
CN104156931B (en) * 2014-09-04 2017-03-29 成都金盘电子科大多媒体技术有限公司 A kind of digital subtraction angiography method
JP6664873B2 (en) 2014-11-21 2020-03-13 キヤノンメディカルシステムズ株式会社 Image processing apparatus, X-ray diagnostic apparatus, and image processing program
JP6706339B2 (en) * 2016-10-27 2020-06-03 富士フイルム株式会社 Endoscope system
US11191503B2 (en) 2018-07-17 2021-12-07 International Business Machines Corporation Fluid-injector for a simultaneous anatomical and fluid dynamic analysis in coronary angiography
CN109816650B (en) * 2019-01-24 2022-11-25 强联智创(北京)科技有限公司 Target area identification method and system based on two-dimensional DSA image
JP7442976B2 (en) * 2019-04-10 2024-03-05 株式会社島津製作所 X-ray imaging device
US10996306B2 (en) * 2019-04-25 2021-05-04 General Electric Company MRI system and method using neural network for detection of patient motion
CN112554853A (en) * 2020-12-11 2021-03-26 成都北方石油勘探开发技术有限公司 Method and system for controlling water injection or gas injection
CN113989171A (en) * 2021-09-06 2022-01-28 北京东软医疗设备有限公司 Subtraction map generation method and device, storage medium and computer equipment

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH01221142A (en) 1988-02-29 1989-09-04 Shimadzu Corp X-ray image treating device
US20050089143A1 (en) * 2003-09-19 2005-04-28 Kabushiki Kaisha Toshiba X-ray diagnosis apparatus and method for creating image data

Family Cites Families (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0368348A (en) * 1989-08-08 1991-03-25 Toshiba Corp X-ray image display device
US6195450B1 (en) * 1997-09-18 2001-02-27 Siemens Corporate Research, Inc. Methods and apparatus for controlling X-ray angiographic image acquisition
US7013035B2 (en) * 1998-09-25 2006-03-14 Canon Kabushiki Kaisha Image processing method for setting an extraction area, and apparatus and recording medium
DE10064768B4 (en) 2000-12-22 2006-12-07 Siemens Ag Method for examining a living being by means of an imaging method
US6754522B2 (en) * 2001-09-05 2004-06-22 Medimag C.V.I., Inc. Imaging methods and apparatus particularly useful for two and three-dimensional angiography
JP4505805B2 (en) * 2004-08-02 2010-07-21 横河電機株式会社 Region extraction method and apparatus
JP4150005B2 (en) * 2005-01-31 2008-09-17 株式会社東芝 Image diagnostic apparatus, image processing method, image processing apparatus, and storage medium
JP4936929B2 (en) * 2007-02-28 2012-05-23 株式会社東芝 X-ray diagnostic apparatus and image display method
JP4979125B2 (en) * 2007-05-18 2012-07-18 富士フイルム株式会社 Radiation imaging system and program
JP4981578B2 (en) * 2007-08-14 2012-07-25 キヤノン株式会社 Radiation imaging apparatus and control method thereof
JP5197140B2 (en) 2008-05-07 2013-05-15 キヤノン株式会社 X-ray fluoroscopic apparatus, moving image processing method, program, and storage medium
JP5366618B2 (en) * 2009-04-06 2013-12-11 株式会社東芝 X-ray diagnostic imaging equipment
JP2011036433A (en) * 2009-08-11 2011-02-24 Toshiba Corp X-ray diagnostic apparatus and method for generating x-rays diagnostic image
JP5610761B2 (en) 2009-12-16 2014-10-22 キヤノン株式会社 X-ray image processing apparatus, X-ray image processing system, X-ray image processing method, and computer program
JP5645399B2 (en) 2009-12-18 2014-12-24 キヤノン株式会社 X-ray image processing apparatus, X-ray image processing method, and computer program
JP5631030B2 (en) 2010-03-16 2014-11-26 キヤノン株式会社 Radiation image processing apparatus, image processing method, radiation imaging system, and program
JP5537262B2 (en) 2010-05-28 2014-07-02 株式会社東芝 X-ray diagnostic imaging equipment
JP5680654B2 (en) 2010-08-31 2015-03-04 株式会社日立メディコ Ultrasonic diagnostic apparatus and ultrasonic image display method
JP5691341B2 (en) * 2010-09-22 2015-04-01 株式会社島津製作所 Radiation imaging apparatus and image reproduction apparatus

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH01221142A (en) 1988-02-29 1989-09-04 Shimadzu Corp X-ray image treating device
US20050089143A1 (en) * 2003-09-19 2005-04-28 Kabushiki Kaisha Toshiba X-ray diagnosis apparatus and method for creating image data

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
"SPECTUM E-5000 Cardiac Catheterization Lab", WWW.ARCHIVE.ORG, 13 February 2011 (2011-02-13), pages 1 - 2, XP055073858, Retrieved from the Internet <URL:http://web.archive.org/web/20110213065536/http://www.emdmedical.com/i_pro_cardiology.html> [retrieved on 20130801] *
ROSE JOHNSON ET AL: "Exploring the potential for touchless interaction in image-guided interventional radiology", CHI 2011, 7 May 2011 (2011-05-07), pages 3323 - 3332, XP055073761, ISBN: 978-1-45-030228-9, DOI: 10.1145/1978942.1979436 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10499864B2 (en) 2014-09-22 2019-12-10 Koninklijke Philips N.V. Contrast arrival detection

Also Published As

Publication number Publication date
US9396535B2 (en) 2016-07-19
EP2662830B1 (en) 2018-10-10
US20130294674A1 (en) 2013-11-07
CN103385692A (en) 2013-11-13
CN103385692B (en) 2015-09-02
JP6021420B2 (en) 2016-11-09
JP2013233227A (en) 2013-11-21

Similar Documents

Publication Publication Date Title
US9396535B2 (en) Image processing apparatus, image processing method and storage medium to generate subtraction image
EP1820450B1 (en) X-ray diagnostic apparatus, image processing apparatus, and program
CN108703764B (en) Angiography method, device, system, equipment and storage medium
EP1604612B1 (en) X-ray CT apparatus for myocardial perfusion image generation
JP5610761B2 (en) X-ray image processing apparatus, X-ray image processing system, X-ray image processing method, and computer program
US20130066198A1 (en) Contrast Agent Perfusion Adaptive Imaging System
US8633945B2 (en) X-ray diagnostic apparatus and image processing apparatus
US20090185730A1 (en) Automated Image Data Subtraction System Suitable for Use in Angiography
US9275439B2 (en) Image processing apparatus, control method thereof and computer-readable storage medium
US20140193082A1 (en) Image processing apparatus and method
EP1987771A1 (en) Radiation image pickup device and method of processing radiation detecting signal
US11179122B2 (en) Bolus imaging
JP2004112469A (en) Image diagnostic apparatus, image processing method, image processor, and storage medium
JPH05192319A (en) X-ray diagnostic device
JP2005198330A (en) Image diagnosis apparatus, image processing method, image processing apparatus and storage medium
EP3975850B1 (en) X-ray imaging system
JP6605220B2 (en) X-ray imaging device
JP5229496B2 (en) Image processing device
US20130094744A1 (en) Image processing device
CN111603189B (en) Method for operating an X-ray device during the execution of an X-ray examination and X-ray device
JP5591963B2 (en) X-ray diagnostic equipment
JP7503996B2 (en) Nuclear medicine diagnostic device and medical information processing device
JP5488003B2 (en) X-ray imaging method and X-ray imaging apparatus
JP2010022667A (en) Cerebral blood flow analyzer
JPS5917330A (en) X-ray diagnostic apparatus

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

17P Request for examination filed

Effective date: 20140318

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20180601

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

Ref country code: AT

Ref legal event code: REF

Ref document number: 1052129

Country of ref document: AT

Kind code of ref document: T

Effective date: 20181015

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

Ref country code: DE

Ref legal event code: R096

Ref document number: 602013044734

Country of ref document: DE

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20181010

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1052129

Country of ref document: AT

Kind code of ref document: T

Effective date: 20181010

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190210

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190110

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190110

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190210

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190111

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602013044734

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

26N No opposition filed

Effective date: 20190711

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20190430

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190416

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190430

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190430

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190430

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190416

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20130416

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20220323

Year of fee payment: 10

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20220323

Year of fee payment: 10

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20220322

Year of fee payment: 10

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 602013044734

Country of ref document: DE

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20230416

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230416

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230416

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230430

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231103