EP2662830A1 - Image processing apparatus, image processing method and program - Google Patents
Image processing apparatus, image processing method and program Download PDFInfo
- Publication number
- EP2662830A1 EP2662830A1 EP13163985.8A EP13163985A EP2662830A1 EP 2662830 A1 EP2662830 A1 EP 2662830A1 EP 13163985 A EP13163985 A EP 13163985A EP 2662830 A1 EP2662830 A1 EP 2662830A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- image
- images
- image processing
- processing apparatus
- evaluation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012545 processing Methods 0.000 title claims abstract description 42
- 238000003672 processing method Methods 0.000 title claims description 7
- 238000011156 evaluation Methods 0.000 claims abstract description 103
- 238000002347 injection Methods 0.000 claims abstract description 23
- 239000007924 injection Substances 0.000 claims abstract description 23
- 238000002583 angiography Methods 0.000 claims description 22
- 238000003384 imaging method Methods 0.000 claims description 18
- 238000000034 method Methods 0.000 claims description 9
- 210000004204 blood vessel Anatomy 0.000 description 29
- 230000010365 information processing Effects 0.000 description 12
- 238000002594 fluoroscopy Methods 0.000 description 10
- 238000002697 interventional radiology Methods 0.000 description 9
- 230000002792 vascular Effects 0.000 description 8
- 230000017531 blood circulation Effects 0.000 description 4
- 210000000056 organ Anatomy 0.000 description 4
- 238000007781 pre-processing Methods 0.000 description 4
- 238000003745 diagnosis Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000001747 exhibiting effect Effects 0.000 description 2
- 238000001356 surgical procedure Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000002834 transmittance Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
- G06T7/0014—Biomedical image inspection using an image reference approach
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
- A61B6/52—Devices using data or image processing specially adapted for radiation diagnosis
- A61B6/5211—Devices using data or image processing specially adapted for radiation diagnosis involving processing of medical diagnostic data
- A61B6/5229—Devices using data or image processing specially adapted for radiation diagnosis involving processing of medical diagnostic data combining image data of a patient, e.g. combining a functional image with an anatomical image
- A61B6/5235—Devices using data or image processing specially adapted for radiation diagnosis involving processing of medical diagnostic data combining image data of a patient, e.g. combining a functional image with an anatomical image combining images from the same or different ionising radiation imaging techniques, e.g. PET and CT
- A61B6/5241—Devices using data or image processing specially adapted for radiation diagnosis involving processing of medical diagnostic data combining image data of a patient, e.g. combining a functional image with an anatomical image combining images from the same or different ionising radiation imaging techniques, e.g. PET and CT combining overlapping images of the same imaging modality, e.g. by stitching
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
- A61B6/48—Diagnostic techniques
- A61B6/481—Diagnostic techniques involving the use of contrast agents
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
- A61B6/50—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment specially adapted for specific body parts; specially adapted for specific clinical applications
- A61B6/504—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment specially adapted for specific body parts; specially adapted for specific clinical applications for diagnosis of blood vessels, e.g. by angiography
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10116—X-ray image
- G06T2207/10121—Fluoroscopy
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30021—Catheter; Guide wire
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30101—Blood vessel; Artery; Vein; Vascular
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30168—Image quality inspection
Definitions
- the present invention relates to an image processing apparatus, an image processing method and a program.
- the present invention relates to an image processing technique for selecting an image suitable for use in diagnosis and surgery from a plurality of X-ray digital subtraction angiography images.
- DSA digital subtraction angiography
- a DSA image is a subtraction image acquired as follows. Images are acquired before and after a radiopaque dye is injected into an object. The result of subtracting the image acquired before the injection of the radiopaque dye (hereinafter, "mask image”) from the image acquired after the injection of the radiopaque dye (hereinafter, "live image”) is a subtraction image.
- a DSA image not only keeps a blood vessel region, which is a region of interest used in diagnosis, as a region exhibiting a difference between the images acquired before and after the injection of the radiopaque dye, but also removes all unnecessary region other than the region exhibiting the difference as a background region and shows the background region in a uniform fashion. That is to say, in a DSA image, a region of blood vessels filled with the radiopaque dye, which becomes less visible in a live image due to obstruction by the background region, clearly appears as a subtraction image.
- Such a DSA image is commonly used in vascular interventional radiology (IVR).
- Vascular IVR is a technique to insert a catheter into a target organ in a body while checking X-ray fluoroscopy images.
- Japanese Patent Laid-Open No. 1-221142 discloses the invention of an X-ray image processing apparatus suitable for acquiring orientation images that make it easy to understand how the blood vessels are arranged during surgery and the like.
- X-ray fluoroscopy images cannot show blood vessels with high X-ray transmittance.
- a radiopaque dye is administered.
- vascular IVR if the radiopaque dye is injected continuously until a catheter reaches a target organ, then a large dosage of radiopaque dye would be ultimately administered, which increases the burden on a patient.
- a predetermined dosage of radiopaque dye is administered prior to fluoroscopy, and images are captured successively thereafter. By subtracting a pre-angiography image from successive post-angiography images, subtraction images are acquired as moving images that serve as DSA moving images.
- reference image a frame that clearly shows blood vessels through which the catheter should be advanced is selected (hereinafter, "reference image”).
- the reference image is displayed on a second monitor that is different from a main monitor for displaying fluoroscopic images.
- the reference image is used as a guide to advance the catheter during fluoroscopy.
- vascular IVR In general vascular IVR, the following operations are carried out repeatedly until the catheter reaches a target organ: a catheter operation via fluoroscopy, successive imaging under administration of a radiopaque dye, generation of DSA moving images from acquired images, and selection of a frame used as a reference image.
- the administered dosage of radiopaque dye can be suppressed by using a reference image acquired prior to fluoroscopy instead of operating the catheter under administration of the radiopaque dye during fluoroscopy.
- a technician searches for an optimal frame through visual examination by, for example, reproducing, frame-forwarding and frame-reversing the DSA moving images.
- the catheter does not reach a target organ by a catheter operation involving a single generation of DSA moving images and a single fluoroscopy. That is to say, generation of DSA moving images and selection of a reference image are repeated multiple times in accordance with the state of catheter insertion. Therefore, selection of a reference image is a very troublesome operation for the technician and could increase the amount of time required for the treatment.
- the present invention provides an image processing technique that enables reduction in the effort required to select a reference image to be displayed on a second monitor during vascular IVR.
- the present invention in its first aspect provides an image processing apparatus as specified in claims 1 to 13.
- the present invention in its second aspect provides an image processing method as specified in claim 14.
- the present invention in its third aspect provides a program as specified in claim 15.
- the present invention enables reduction in the effort required to select a reference image to be displayed on a second monitor during vascular IVR.
- FIG. 1 illustrates a configuration of an image processing apparatus according to an embodiment.
- FIG. 2 illustrates a flow of an image processing method according to an embodiment.
- FIG. 3 schematically illustrates a subtraction image.
- FIG. 4 schematically illustrates a minimum-value projection image.
- FIG. 5 illustrates a configuration of an information processing apparatus according to an embodiment.
- the X-ray image processing apparatus 100 includes an X-ray generation unit 101, a two-dimensional X-ray sensor 104, a preprocessing unit 105, an image storage unit 106, an inter-image subtraction unit 107, an evaluation unit 108, a reference image selection unit 109, and an image display unit 110.
- the X-ray generation unit 101 can generate X-ray pulses (X-rays) at 3 to 30 pulses per second.
- the two-dimensional X-ray sensor 104 can capture moving images. More specifically, the two-dimensional X-ray sensor 104 receives X-rays 103 that have been transmitted through an object 102, and captures moving images synchronized with the X-ray pulses.
- the two-dimensional X-ray sensor 104 also functions as an imaging unit that captures moving images of the object 102 illuminated with the X-rays.
- the preprocessing unit 105 preprocesses frames of the moving images output from the two-dimensional X-ray sensor 104.
- the image storage unit 106 stores at least one of the frames of the moving images preprocessed by the preprocessing unit 105 as a mask image acquired before injecting a radiopaque dye.
- a frame stored as a mask image is, for example, a frame that is acquired immediately after an operation of capturing the moving images is started, or a frame that is automatically acquired immediately before the injection of a radiopaque dye by detecting the injection of the radiopaque dye from the moving images.
- a frame stored as a mask image may be a frame that is selected by an operator designating a storage timing when starting the injection of the radiopaque dye.
- a frame used as a mask image may be selected as appropriate from a plurality of frames stored in the image storage unit 106, or may be acquired by compositing a plurality of frames.
- the inter-image subtraction unit 107 generates images composed of a plurality of frames from X-ray images of the object acquired before and after the injection of the radiopaque dye. More specifically, the inter-image subtraction unit 107 generates subtraction images by subtracting the mask image stored in the image storage unit 106 from live images, which are images (frames) that are acquired after the injection of the radiopaque dye and are output from the preprocessing unit 105, and outputs the subtraction images as frames of DSA moving images.
- the evaluation unit 108 calculates, from the frames of the DSA moving images output from the inter-image subtraction unit 107, evaluation values indicating the states of the injection of the radiopaque dye for the frames, and outputs the evaluation values.
- the reference image selection unit 109 selects at least one of the frames of the DSA moving images based on the evaluation values for the frames output from the evaluation unit 108, and outputs the selected frame as a reference image.
- the image display unit 110 displays the reference image output from the reference image selection unit 109. In the case where a plurality of frames have been selected as reference images, the image display unit 110 can reproduce the reference images one by one in a loop at a predetermined frame rate.
- the reference image selection unit 109 may output and store a reference image to a storage unit not shown in the figures. In this case, the image display unit 110 can display the reference image stored in the storage unit.
- the following describes the operations of the evaluation unit 108 and the image display unit 110 that represent characteristic configurations of the present embodiment in connection with the X-ray image processing apparatus 100 configured in the above manner.
- step S201 the inter-image subtraction unit 107 outputs the frames of the DSA moving images to the evaluation unit 108, and the evaluation unit 108 outputs the evaluation values calculated by evaluating the states of the injection of the radiopaque dye for the frames of the DSA moving images.
- the states of the injection of the radiopaque dye may be evaluated using various methods. Below, an evaluation method based on the area of a blood vessel region in each frame (DSA image) of the DSA moving images is explained as one example.
- FIG. 3 is a schematic diagram illustrating a subtraction image acquired by subtracting a mask image from a live image.
- the subtraction image illustrated in FIG. 3 has pixel values of -1.0 to 1.0. Pixel values close to 1.0 are illustrated in white, while pixel values close to -1.0 are illustrated in black.
- DSA images are subtraction images acquired by subtracting a mask image which is generated from an X-ray image of an object acquired before the injection of a radiopaque dye, from live images which are X-ray images constituted by a plurality of frames acquired after the injection of the radiopaque dye.
- a background region 302 that is acquired by removing a blood vessel region 301 into which the radiopaque dye has been introduced (injected) is cancelled out by subtraction processing, and therefore has a pixel value of 0.
- the blood vessel region 301 mostly has a pixel value that is far from the pixel value of the background region 302 (a negative pixel value in the example of FIG. 3 ).
- the number of pixels with negative pixel values is counted.
- DSA moving images with the following conditions: a total number of frames is T, the number of pixel rows in each frame is H, the number of pixel columns in each frame is W, and the t th frame of the DSA moving images is a subtraction image It (where 0 ⁇ t ⁇ T).
- a pixel (x, y) in the subtraction image It (where 0 ⁇ x ⁇ W, 0 ⁇ y ⁇ H) has a pixel value It (x, y).
- an evaluation value Sv t for the t th frame of the DSA moving images can be expressed by the following equation.
- step S202 the evaluation unit 108 outputs the evaluation values to the reference image selection unit 109, and the reference image selection unit 109 selects at least one frame that is suitable as a reference image from the DSA images based on the evaluation values output from the evaluation unit 108.
- the evaluation unit 108 evaluates the states of the injection of the radiopaque dye based on the area of a blood vessel region in the DSA images, and calculates an evaluation value Sv t as an evaluation value for the t th frame of the DSA moving images (where 0 ⁇ t ⁇ T).
- the frame t ref selected by the reference image selection unit 109 has the largest evaluation value Sv of all the frames output from the evaluation unit 108.
- the frame t ref can be expressed by the following equation.
- the above equation indicates that the t th frame with the largest evaluation value Sv t is selected by the reference image selection unit 109 as the reference image (frame t ref ).
- the above equation is applicable under the assumption that the reference image selection unit 109 selects one frame, i.e. the t th frame with the largest evaluation value Sv t as the reference image (where 0 ⁇ t ⁇ T). It should be noted, however, that the reference image selection unit 109 is not limited to selecting one frame image as the reference image based on the largest evaluation value Sv. Alternatively, the reference image selection unit 109 may select a plurality of frame images using the evaluation values Sv.
- a plurality of frames that precede and succeed a frame with the largest evaluation value Sv may be selected.
- a plurality of frames may be selected in the order of the largest evaluation value Sv to the smallest evaluation value Sv.
- the evaluation unit 108 evaluates the frames (DSA images) of the DSA moving images based on the area of the blood vessel region in the DSA images, and the reference image selection unit 109 selects (outputs) a frame including a blood vessel region of the largest area as a reference image.
- the reference image selection unit 109 selects (outputs) a frame including a blood vessel region of the largest area as a reference image.
- pixel values of pixels included in the frames are compared with a threshold, and evaluation values are calculated using the number of pixels with pixel values within a predetermined range.
- DSA images include minute noise that has pixel values resembling blood vessels filled with a radiopaque dye.
- the evaluation unit 108 may calculate evaluation values in consideration of the continuity of pixels in addition to the number of pixels with pixel values within a predetermined range. For example, the evaluation unit 108 may calculate the number of pixels that have pixel values falling within a predetermined range with respect to a threshold and that are continuously arranged as an evaluation value.
- the use of evaluation values calculated in the above manner can further improve the accuracy of selection of a reference image.
- the area of the blood vessel region 301 in a DSA image is used as the evaluation value for an ideal subtraction image.
- a motion artifact 303 could appear due to the movement of the object and the movement of the inserted catheter.
- the motion artifact 303 has both positive and negative pixel values as illustrated in FIG. 3 , and thus has a large influence on the image quality of a DSA image. It is possible that the motion artifact 303 in a reference image be small.
- values associated with the motion artifact 303 may be added to the evaluation values calculated by the evaluation unit 108 in step S201.
- the motion artifact 303 could have both positive and negative pixel values as illustrated in FIG. 3 .
- the motion artifact 303 could include a region with pixel values close to 1.0 (a white region) and a region with pixel values close to -1.0 (a black region). The following describes a method whereby the number of pixels with positive pixel values is counted as a simple example.
- an evaluation value Sm t based on the motion artifact 303 can be expressed by the following equation, using the same signs as those used in Expression 1.
- Sm + Sv an evaluation value S calculated as a sum of an evaluation value Sv and an evaluation value Sm is expressed by the following equation, using a coefficient a greater than 0 as a weight for the evaluation value Sv, and a coefficient b smaller than 0 as a weight for the evaluation value Sm.
- an evaluation value S t is an evaluation value for the t th frame of the DSA moving images (where 0 ⁇ t ⁇ T)
- the frame t ref selected by the reference image selection unit 109 in step S202 can be expressed by the following equation.
- the above equation indicates that the t th frame with the largest evaluation value S t is selected by the reference image selection unit 109 as the reference image (t ref ). That is to say, the reference image selection unit 109 selects a frame including a blood vessel region of a large area and a small motion artifact as a reference image.
- the number of pixels at the boundary between the radiopaque dye and the background region may be used to calculate an evaluation value.
- the boundary between the radiopaque dye and the background region can be determined, for example, by applying edge detection processing to a subtraction image using the Canny edge detector and the like and finding out pixel values at the detected edge positions.
- the evaluation unit 108 can calculate evaluation values based on an imaging portion for which the DSA moving images have been captured, imaging technique information, imaging conditions, and a time period that has elapsed since the injection of the radiopaque dye.
- a DSA image acquired at the timing when the radiopaque dye has spread throughout the object is selected as a reference image. Therefore, there is a high possibility that a DSA image acquired immediately after the injection of the radiopaque dye, or acquired after a long time period has elapsed since the injection of the radiopaque dye, cannot be used as a reference image.
- the administered dosage of radiopaque dye determined by the imaging portion and the imaging technique information is Q
- the velocity of blood flow in the imaging portion is D
- a frame rate determined by the imaging conditions is R.
- a range of frames [t start , t end ] that can be used as a reference image when the above conditions are changed is experimentally acquired and stored as a table in advance.
- the evaluation unit 108 acquires a frame range [t start , t end ] determined by the imaging conditions from this table, and calculates evaluation values S t within this frame range.
- the reference image selection unit 109 selects the frame t ref as a reference image based on the evaluation values calculated by the evaluation unit 108.
- data stored in the table is not limited to the range of frames that can be used as a reference image.
- the table may store values from 0.0 to 1.0 as weights w t corresponding to different sets of the administered dosage Q of radiopaque dye, the velocity D of blood flow in the imaging portion and the frame rate R determined by the imaging conditions. For example, once the administered dosage Q of radiopaque dye, the velocity D of blood flow and the frame rate R have been determined, a corresponding weight w t can be acquired as shown in the following equation.
- the frame t ref selected by the reference image selection unit 109 in step S202 can be expressed by the following equation.
- a weight is applied to the evaluation value S t for each frame in correspondence with the administered dosage Q of radiopaque dye, the velocity D of blood flow and the frame rate R, and therefore a reference image can be selected at higher accuracy.
- An evaluation value may be the result of comparing a minimum-value projection image M generated from the DSA moving images with the frames of the DSA moving images. Provided that each frame of the DSA moving images is It and a value of each pixel therein (x, y) is It (x, y), the minimum-value projection image M is expressed by the following equation.
- FIG. 4 schematically illustrates a minimum-value projection image 400 generated from DSA images 401, 402 and 403. As illustrated in FIG. 4 , the position where the region of blood vessels filled with the radiopaque dye is shown changes over time in the DSA images 401, 402 and 403 that are included among the DSA moving images as frames. Therefore, in any of these DSA images (frames), only a part of the entire blood vessel region is shown.
- the minimum-value projection image 400 constituted by these frames is generated by acquiring pixels that each have the smallest pixel value among corresponding pixels in all the frames.
- any of the frames constituting the DSA moving images includes the region of blood vessels filled with the radiopaque dye, pixel values of this region of blood vessels filled with the radiopaque dye are acquired as the minimum pixel values. Therefore, the minimum-value projection image 400 can show the entire blood vessel region. Evaluation values Sc t based on comparison between this minimum-value projection image M and the frames of the DSA moving images can be calculated using the following equation.
- the smallest evaluation value Sc t is acquired from a frame of the DSA moving images that is closest to the minimum-value projection image M, that is to say, a frame of the DSA moving images in which the radiopaque dye has spread over a wide range.
- the evaluation unit 108 may calculate evaluation values based on the addition of a plurality of feature amounts calculated from the frames of the subtraction images. For example, provided that an evaluation value S is acquired by calculating n types of feature amounts (S 1 , S 2 , ... , S n ) from an image I and adding the feature amounts multiplied by weight coefficients (k 1 , k 2 , ... , k n ) through linear combination, the evaluation value S can be expressed by the following equation.
- each weight coefficient k is, for example, larger than 0 when the corresponding feature amount is large and represents a reference image (S > 0), and smaller than 0 when the corresponding feature amount is small and represents a reference image (S ⁇ 0).
- the reference image selection unit 109 can select a reference image based on a plurality of feature amounts by selecting a frame in accordance with the evaluation values S (e.g. a frame with the largest evaluation value S).
- the components illustrated in FIG. 1 may be configured by special-purpose hardware. Functional configurations of the hardware may be realized by software. In this case, the functions of the components illustrated in FIG. 1 may be realized by installing software on an information processing apparatus and implementing an image processing method through the execution of the software using the arithmetic functions of the information processing apparatus. For example, through the execution of the software, a subtraction step, an evaluation step and a selection step are executed for the frames of the moving images output from the two-dimensional X-ray sensor 104.
- FIG. 5 is a block diagram illustrating a hardware configuration of an information processing apparatus and configurations of peripheral apparatuses therefor.
- An information processing apparatus 1000 and an image capturing apparatus 2000 are connected to each other and are configured to perform data communication with each other.
- a CPU 1010 can control the entirety of the information processing apparatus 1000 using programs and data stored in a RAM 1020 and a ROM 1030.
- the CPU 1010 can also execute arithmetic processing related to predetermined image processing by executing programs.
- the RAM 1020 includes an area for temporarily storing programs and data loaded from a magneto-optical disc 1060 and a hard disk drive 1050.
- the RAM 1020 also includes an area for temporarily storing data of X-ray fluoroscopy moving images acquired from the image capturing apparatus 2000.
- the RAM 1020 also includes a work area used when the CPU 1010 executes various types of processing.
- the ROM 1030 stores data of the settings of the information processing apparatus 1000, boot programs, and the like.
- the hard disk drive 1050 is one example of an information storage medium, and stores programs and data for causing the CPU 1010 to execute processing of an operating system (OS) and the components illustrated in FIG. 1 . They are loaded to the RAM 1020 as appropriate in accordance with control by the CPU 1010 and are executed by the CPU 1010. Data of the X-ray fluoroscopy moving images may also be stored in the hard disk drive 1050.
- OS operating system
- Data of the X-ray fluoroscopy moving images may also be stored in the hard disk drive 1050.
- the magneto-optical disc 1060 is one example of an information storage medium, and can store a part or all of the programs and data stored in the hard disk drive 1050.
- a mouse 1070 and a keyboard 1080 can input various types of instructions to the CPU 1010 by being operated by an operator of the information processing apparatus 1000.
- a printer 1090 can print/output images displayed on the image display unit 110 on/to a recording medium.
- a display apparatus 1100 is constituted by a CRT, a liquid crystal display, or the like, and can display the result of processing of the CPU 1010 using images and text.
- the display apparatus 1100 can display images that have been processed by the components illustrated in FIG. 1 and ultimately output from the image display unit 110.
- the image display unit 110 functions as a display control unit for displaying images on the display apparatus 1100.
- a bus 1040 which connects between the components of the information processing apparatus 1000, enables them to exchange data with one another.
- the image capturing apparatus 2000 is an apparatus that can capture moving images, such as an X-ray fluoroscope.
- the image capturing apparatus 2000 transmits captured image data to the information processing apparatus 1000.
- the image capturing apparatus 2000 may transmit a plurality of pieces of image data collectively to the information processing apparatus 1000, or may transmit a piece of image data each time it captures an image.
- aspects of the present invention can also be realized by a computer of a system or apparatus (or devices such as a CPU or MPU) that reads out and executes a program recorded on a memory device to perform the functions of the above-described embodiment(s), and by a method, the steps of which are performed by a computer of a system or apparatus by, for example, reading out and executing a program recorded on a memory device to perform the functions of the above-described embodiment(s).
- the program is provided to the computer for example via a network or from a recording medium of various types serving as the memory device (e.g., computer-readable medium).
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Biophysics (AREA)
- High Energy & Nuclear Physics (AREA)
- Optics & Photonics (AREA)
- Pathology (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Molecular Biology (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Apparatus For Radiation Diagnosis (AREA)
Abstract
Description
- The present invention relates to an image processing apparatus, an image processing method and a program. In particular, the present invention relates to an image processing technique for selecting an image suitable for use in diagnosis and surgery from a plurality of X-ray digital subtraction angiography images.
- Recent advancement in digital technology has made it common to apply digital processing to images in the medical field. In place of the conventional X-ray imaging that uses films for X-ray diagnosis, two-dimensional X-ray sensors that output X-ray images as digital images have become widespread. The application of digital image processing to digital images output from the two-dimensional X-ray sensors comes in a broad range.
- One example of suitable application of such digital image processing is digital subtraction angiography (DSA) processing for acquiring a DSA image. A DSA image is a subtraction image acquired as follows. Images are acquired before and after a radiopaque dye is injected into an object. The result of subtracting the image acquired before the injection of the radiopaque dye (hereinafter, "mask image") from the image acquired after the injection of the radiopaque dye (hereinafter, "live image") is a subtraction image. A DSA image not only keeps a blood vessel region, which is a region of interest used in diagnosis, as a region exhibiting a difference between the images acquired before and after the injection of the radiopaque dye, but also removes all unnecessary region other than the region exhibiting the difference as a background region and shows the background region in a uniform fashion. That is to say, in a DSA image, a region of blood vessels filled with the radiopaque dye, which becomes less visible in a live image due to obstruction by the background region, clearly appears as a subtraction image. Such a DSA image is commonly used in vascular interventional radiology (IVR). Vascular IVR is a technique to insert a catheter into a target organ in a body while checking X-ray fluoroscopy images.
- Regarding DSA processing, for example, Japanese Patent Laid-Open No.
1-221142 - Normally, X-ray fluoroscopy images cannot show blood vessels with high X-ray transmittance. To show such blood vessels, a radiopaque dye is administered. However, in vascular IVR, if the radiopaque dye is injected continuously until a catheter reaches a target organ, then a large dosage of radiopaque dye would be ultimately administered, which increases the burden on a patient. In view of this, in general vascular IVR, a predetermined dosage of radiopaque dye is administered prior to fluoroscopy, and images are captured successively thereafter. By subtracting a pre-angiography image from successive post-angiography images, subtraction images are acquired as moving images that serve as DSA moving images. From the DSA moving images, a frame that clearly shows blood vessels through which the catheter should be advanced is selected (hereinafter, "reference image"). The reference image is displayed on a second monitor that is different from a main monitor for displaying fluoroscopic images. The reference image is used as a guide to advance the catheter during fluoroscopy.
- In general vascular IVR, the following operations are carried out repeatedly until the catheter reaches a target organ: a catheter operation via fluoroscopy, successive imaging under administration of a radiopaque dye, generation of DSA moving images from acquired images, and selection of a frame used as a reference image. The administered dosage of radiopaque dye can be suppressed by using a reference image acquired prior to fluoroscopy instead of operating the catheter under administration of the radiopaque dye during fluoroscopy.
- Conventionally, in order to select a reference image from DSA moving images, a technician searches for an optimal frame through visual examination by, for example, reproducing, frame-forwarding and frame-reversing the DSA moving images. In general vascular IVR, the catheter does not reach a target organ by a catheter operation involving a single generation of DSA moving images and a single fluoroscopy. That is to say, generation of DSA moving images and selection of a reference image are repeated multiple times in accordance with the state of catheter insertion. Therefore, selection of a reference image is a very troublesome operation for the technician and could increase the amount of time required for the treatment.
- In view of the above problems, the present invention provides an image processing technique that enables reduction in the effort required to select a reference image to be displayed on a second monitor during vascular IVR.
- The present invention in its first aspect provides an image processing apparatus as specified in
claims 1 to 13. - The present invention in its second aspect provides an image processing method as specified in claim 14.
- The present invention in its third aspect provides a program as specified in claim 15.
- The present invention enables reduction in the effort required to select a reference image to be displayed on a second monitor during vascular IVR.
- Further features of the present invention will become apparent from the following description of embodiments (with reference to the attached drawings). Each of the embodiments of the present invention described below can be implemented solely or as a combination of a plurality of the embodiments or features thereof where necessary or where the combination of elements or features from individual embodiments in a single embodiment is beneficial.
-
FIG. 1 illustrates a configuration of an image processing apparatus according to an embodiment. -
FIG. 2 illustrates a flow of an image processing method according to an embodiment. -
FIG. 3 schematically illustrates a subtraction image. -
FIG. 4 schematically illustrates a minimum-value projection image. -
FIG. 5 illustrates a configuration of an information processing apparatus according to an embodiment. - The following describes one example of embodiments of the present invention with reference to the drawings.
- A description is now given of an image processing apparatus according to the first embodiment of the present invention (hereinafter, "X-ray
image processing apparatus 100") with reference toFIG. 1 . The X-rayimage processing apparatus 100 includes anX-ray generation unit 101, a two-dimensional X-ray sensor 104, apreprocessing unit 105, animage storage unit 106, aninter-image subtraction unit 107, anevaluation unit 108, a referenceimage selection unit 109, and animage display unit 110. - The
X-ray generation unit 101 can generate X-ray pulses (X-rays) at 3 to 30 pulses per second. The two-dimensional X-ray sensor 104 can capture moving images. More specifically, the two-dimensional X-ray sensor 104 receivesX-rays 103 that have been transmitted through anobject 102, and captures moving images synchronized with the X-ray pulses. The two-dimensional X-ray sensor 104 also functions as an imaging unit that captures moving images of theobject 102 illuminated with the X-rays. - The preprocessing
unit 105 preprocesses frames of the moving images output from the two-dimensional X-ray sensor 104. Theimage storage unit 106 stores at least one of the frames of the moving images preprocessed by the preprocessingunit 105 as a mask image acquired before injecting a radiopaque dye. Note that a frame stored as a mask image is, for example, a frame that is acquired immediately after an operation of capturing the moving images is started, or a frame that is automatically acquired immediately before the injection of a radiopaque dye by detecting the injection of the radiopaque dye from the moving images. Alternatively, a frame stored as a mask image may be a frame that is selected by an operator designating a storage timing when starting the injection of the radiopaque dye. Alternatively, a frame used as a mask image may be selected as appropriate from a plurality of frames stored in theimage storage unit 106, or may be acquired by compositing a plurality of frames. - The
inter-image subtraction unit 107 generates images composed of a plurality of frames from X-ray images of the object acquired before and after the injection of the radiopaque dye. More specifically, theinter-image subtraction unit 107 generates subtraction images by subtracting the mask image stored in theimage storage unit 106 from live images, which are images (frames) that are acquired after the injection of the radiopaque dye and are output from the preprocessingunit 105, and outputs the subtraction images as frames of DSA moving images. - The
evaluation unit 108 calculates, from the frames of the DSA moving images output from theinter-image subtraction unit 107, evaluation values indicating the states of the injection of the radiopaque dye for the frames, and outputs the evaluation values. - The reference
image selection unit 109 selects at least one of the frames of the DSA moving images based on the evaluation values for the frames output from theevaluation unit 108, and outputs the selected frame as a reference image. - The
image display unit 110 displays the reference image output from the referenceimage selection unit 109. In the case where a plurality of frames have been selected as reference images, theimage display unit 110 can reproduce the reference images one by one in a loop at a predetermined frame rate. The referenceimage selection unit 109 may output and store a reference image to a storage unit not shown in the figures. In this case, theimage display unit 110 can display the reference image stored in the storage unit. - With reference to a flowchart of
FIG. 2 , the following describes the operations of theevaluation unit 108 and theimage display unit 110 that represent characteristic configurations of the present embodiment in connection with the X-rayimage processing apparatus 100 configured in the above manner. - In step S201, the
inter-image subtraction unit 107 outputs the frames of the DSA moving images to theevaluation unit 108, and theevaluation unit 108 outputs the evaluation values calculated by evaluating the states of the injection of the radiopaque dye for the frames of the DSA moving images. The states of the injection of the radiopaque dye may be evaluated using various methods. Below, an evaluation method based on the area of a blood vessel region in each frame (DSA image) of the DSA moving images is explained as one example. -
FIG. 3 is a schematic diagram illustrating a subtraction image acquired by subtracting a mask image from a live image. The subtraction image illustrated inFIG. 3 has pixel values of -1.0 to 1.0. Pixel values close to 1.0 are illustrated in white, while pixel values close to -1.0 are illustrated in black. DSA images are subtraction images acquired by subtracting a mask image which is generated from an X-ray image of an object acquired before the injection of a radiopaque dye, from live images which are X-ray images constituted by a plurality of frames acquired after the injection of the radiopaque dye. Therefore, in an ideal subtraction image, abackground region 302 that is acquired by removing ablood vessel region 301 into which the radiopaque dye has been introduced (injected) is cancelled out by subtraction processing, and therefore has a pixel value of 0. Theblood vessel region 301 mostly has a pixel value that is far from the pixel value of the background region 302 (a negative pixel value in the example ofFIG. 3 ). - Therefore, with a method of evaluating the states of the injection of the radiopaque dye based on the area of the
blood vessel region 301 in DSA images, the number of pixels with negative pixel values is counted. Assume there are DSA moving images with the following conditions: a total number of frames is T, the number of pixel rows in each frame is H, the number of pixel columns in each frame is W, and the tth frame of the DSA moving images is a subtraction image It (where 0 ≤ t < T). Also assume that a pixel (x, y) in the subtraction image It (where 0 ≤ x < W, 0 ≤ y < H) has a pixel value It (x, y). Also assume that a threshold to separate thebackground region 302 with a pixel value of 0 from theblood vessel region 301 is generalized as thresh. Under the above relationships, an evaluation value Svt for the tth frame of the DSA moving images can be expressed by the following equation. -
- In step S202, the
evaluation unit 108 outputs the evaluation values to the referenceimage selection unit 109, and the referenceimage selection unit 109 selects at least one frame that is suitable as a reference image from the DSA images based on the evaluation values output from theevaluation unit 108. - According to the above-described example, in step S201, the
evaluation unit 108 evaluates the states of the injection of the radiopaque dye based on the area of a blood vessel region in the DSA images, and calculates an evaluation value Svt as an evaluation value for the tth frame of the DSA moving images (where 0 ≤ t < T). - An image showing the state where the radiopaque dye is widely spread in a blood vessel region is suitable as a reference image. Therefore, the frame tref selected by the reference
image selection unit 109 has the largest evaluation value Sv of all the frames output from theevaluation unit 108. Using an evaluation value output from theevaluation unit 108, the frame tref can be expressed by the following equation. -
- The above equation indicates that the tth frame with the largest evaluation value Svt is selected by the reference
image selection unit 109 as the reference image (frame tref). The above equation is applicable under the assumption that the referenceimage selection unit 109 selects one frame, i.e. the tth frame with the largest evaluation value Svt as the reference image (where 0 ≤ t < T). It should be noted, however, that the referenceimage selection unit 109 is not limited to selecting one frame image as the reference image based on the largest evaluation value Sv. Alternatively, the referenceimage selection unit 109 may select a plurality of frame images using the evaluation values Sv. For example, in the case where two or more frames are selected as reference images and reproduced one by one in a loop at a predetermined frame rate, a plurality of frames that precede and succeed a frame with the largest evaluation value Sv may be selected. Alternatively, a plurality of frames may be selected in the order of the largest evaluation value Sv to the smallest evaluation value Sv. - In the present embodiment, the
evaluation unit 108 evaluates the frames (DSA images) of the DSA moving images based on the area of the blood vessel region in the DSA images, and the referenceimage selection unit 109 selects (outputs) a frame including a blood vessel region of the largest area as a reference image. By outputting this reference image to theimage display unit 110, the problem of frame selection can be alleviated for the operator of the X-ray image processing apparatus. - In the above-described example, pixel values of pixels included in the frames (DSA images) are compared with a threshold, and evaluation values are calculated using the number of pixels with pixel values within a predetermined range. It should be noted, however, that DSA images include minute noise that has pixel values resembling blood vessels filled with a radiopaque dye. In view of this, the
evaluation unit 108 may calculate evaluation values in consideration of the continuity of pixels in addition to the number of pixels with pixel values within a predetermined range. For example, theevaluation unit 108 may calculate the number of pixels that have pixel values falling within a predetermined range with respect to a threshold and that are continuously arranged as an evaluation value. As a region of blood vessels filled with a radiopaque dye can be thought as a region that is continuous with the tip of the catheter, the use of evaluation values calculated in the above manner can further improve the accuracy of selection of a reference image. - Regarding the examples of the
blood vessel region 301 and thebackground region 302 illustrated inFIG. 3 , the area of theblood vessel region 301 in a DSA image is used as the evaluation value for an ideal subtraction image. However, in an actual subtraction image, amotion artifact 303 could appear due to the movement of the object and the movement of the inserted catheter. For example, themotion artifact 303 has both positive and negative pixel values as illustrated inFIG. 3 , and thus has a large influence on the image quality of a DSA image. It is possible that themotion artifact 303 in a reference image be small. In order to select a reference image in consideration of the existence of themotion artifact 303, values associated with themotion artifact 303 may be added to the evaluation values calculated by theevaluation unit 108 in step S201. - There are various methods of calculating evaluation values that take the
motion artifact 303 into consideration. Unlike theblood vessel region 301 that mostly has negative pixel values, themotion artifact 303 could have both positive and negative pixel values as illustrated inFIG. 3 . For example, themotion artifact 303 could include a region with pixel values close to 1.0 (a white region) and a region with pixel values close to -1.0 (a black region). The following describes a method whereby the number of pixels with positive pixel values is counted as a simple example. Provided that a threshold for separating thebackground region 302 from themotion artifact 303 is thresh2, an evaluation value Smt based on themotion artifact 303 can be expressed by the following equation, using the same signs as those used inExpression 1. -
- The result of adding this evaluation value Sm based on the
motion artifact 303 and an evaluation value Sv based on the area of theblood vessel region 301 is an evaluation value S (= Sm + Sv) to be ultimately calculated. When an evaluation value Sm based on themotion artifact 303 is used, it is possible that themotion artifact 303 in a reference image be small. In view of this, an evaluation value S calculated as a sum of an evaluation value Sv and an evaluation value Sm is expressed by the following equation, using a coefficient a greater than 0 as a weight for the evaluation value Sv, and a coefficient b smaller than 0 as a weight for the evaluation value Sm. -
- Provided that an evaluation value St is an evaluation value for the tth frame of the DSA moving images (where 0 ≤ t < T), the frame tref selected by the reference
image selection unit 109 in step S202 can be expressed by the following equation. -
- Similarly to Expression 2, the above equation indicates that the tth frame with the largest evaluation value St is selected by the reference
image selection unit 109 as the reference image (tref). That is to say, the referenceimage selection unit 109 selects a frame including a blood vessel region of a large area and a small motion artifact as a reference image. - The number of pixels at the boundary between the radiopaque dye and the background region may be used to calculate an evaluation value. The boundary between the radiopaque dye and the background region can be determined, for example, by applying edge detection processing to a subtraction image using the Canny edge detector and the like and finding out pixel values at the detected edge positions. By calculating an evaluation value using the number of pixels at the boundary, the complexity of structures of fine blood vessels that cannot be evaluated based only on the aforementioned area of the blood vessel region can be added to the evaluation value, thereby enabling selection of a more suitable reference image.
- Furthermore, the
evaluation unit 108 can calculate evaluation values based on an imaging portion for which the DSA moving images have been captured, imaging technique information, imaging conditions, and a time period that has elapsed since the injection of the radiopaque dye. A DSA image acquired at the timing when the radiopaque dye has spread throughout the object is selected as a reference image. Therefore, there is a high possibility that a DSA image acquired immediately after the injection of the radiopaque dye, or acquired after a long time period has elapsed since the injection of the radiopaque dye, cannot be used as a reference image. For example, assume that the administered dosage of radiopaque dye determined by the imaging portion and the imaging technique information is Q, the velocity of blood flow in the imaging portion is D, and a frame rate determined by the imaging conditions is R. A range of frames [tstart, tend] that can be used as a reference image when the above conditions are changed is experimentally acquired and stored as a table in advance. -
- During the actual imaging, the
evaluation unit 108 acquires a frame range [tstart, tend] determined by the imaging conditions from this table, and calculates evaluation values St within this frame range. The referenceimage selection unit 109 then selects the frame tref as a reference image based on the evaluation values calculated by theevaluation unit 108. With this configuration, a range of frame search can be restricted, and therefore the speed of processing can be accelerated. Note that data stored in the table is not limited to the range of frames that can be used as a reference image. For example, the table may store values from 0.0 to 1.0 as weights wt corresponding to different sets of the administered dosage Q of radiopaque dye, the velocity D of blood flow in the imaging portion and the frame rate R determined by the imaging conditions. For example, once the administered dosage Q of radiopaque dye, the velocity D of blood flow and the frame rate R have been determined, a corresponding weight wt can be acquired as shown in the following equation. -
- Using this weight wt, the frame tref selected by the reference
image selection unit 109 in step S202 can be expressed by the following equation. -
- With the above configuration, a weight is applied to the evaluation value St for each frame in correspondence with the administered dosage Q of radiopaque dye, the velocity D of blood flow and the frame rate R, and therefore a reference image can be selected at higher accuracy.
- An evaluation value may be the result of comparing a minimum-value projection image M generated from the DSA moving images with the frames of the DSA moving images. Provided that each frame of the DSA moving images is It and a value of each pixel therein (x, y) is It (x, y), the minimum-value projection image M is expressed by the following equation.
-
- The pixel value M (x, y) of each pixel (x, y) in the minimum-value projection image M is the smallest of pixel values of corresponding pixels in all the frames of the DSA moving images.
FIG. 4 schematically illustrates a minimum-value projection image 400 generated fromDSA images
As illustrated inFIG. 4 , the position where the region of blood vessels filled with the radiopaque dye is shown changes over time in theDSA images value projection image 400 constituted by these frames is generated by acquiring pixels that each have the smallest pixel value among corresponding pixels in all the frames. When any of the frames constituting the DSA moving images includes the region of blood vessels filled with the radiopaque dye, pixel values of this region of blood vessels filled with the radiopaque dye are acquired as the minimum pixel values. Therefore, the minimum-value projection image 400 can show the entire blood vessel region. Evaluation values Sct based on comparison between this minimum-value projection image M and the frames of the DSA moving images can be calculated using the following equation. -
- The smallest evaluation value Sct is acquired from a frame of the DSA moving images that is closest to the minimum-value projection image M, that is to say, a frame of the DSA moving images in which the radiopaque dye has spread over a wide range. In view of this, by adding the result of multiplication by a predetermined coefficient smaller than 0 to the evaluation value in a manner similar to the evaluation value Sm based on the motion artifact, a more suitable reference image can be selected.
- Although several types of calculation of evaluation values have been described above as examples, evaluation values that can be used in the present invention are not limited to the above examples. For example, the
evaluation unit 108 may calculate evaluation values based on the addition of a plurality of feature amounts calculated from the frames of the subtraction images. For example, provided that an evaluation value S is acquired by calculating n types of feature amounts (S1, S2, ... , Sn) from an image I and adding the feature amounts multiplied by weight coefficients (k1, k2, ... , kn) through linear combination, the evaluation value S can be expressed by the following equation. -
- The value of each weight coefficient k is, for example, larger than 0 when the corresponding feature amount is large and represents a reference image (S > 0), and smaller than 0 when the corresponding feature amount is small and represents a reference image (S < 0).
- In this case, the reference
image selection unit 109 can select a reference image based on a plurality of feature amounts by selecting a frame in accordance with the evaluation values S (e.g. a frame with the largest evaluation value S). - The components illustrated in
FIG. 1 may be configured by special-purpose hardware. Functional configurations of the hardware may be realized by software. In this case, the functions of the components illustrated inFIG. 1 may be realized by installing software on an information processing apparatus and implementing an image processing method through the execution of the software using the arithmetic functions of the information processing apparatus. For example, through the execution of the software, a subtraction step, an evaluation step and a selection step are executed for the frames of the moving images output from the two-dimensional X-ray sensor 104. -
FIG. 5 is a block diagram illustrating a hardware configuration of an information processing apparatus and configurations of peripheral apparatuses therefor. Aninformation processing apparatus 1000 and animage capturing apparatus 2000 are connected to each other and are configured to perform data communication with each other. - A
CPU 1010 can control the entirety of theinformation processing apparatus 1000 using programs and data stored in aRAM 1020 and aROM 1030. TheCPU 1010 can also execute arithmetic processing related to predetermined image processing by executing programs. - The
RAM 1020 includes an area for temporarily storing programs and data loaded from a magneto-optical disc 1060 and ahard disk drive 1050. TheRAM 1020 also includes an area for temporarily storing data of X-ray fluoroscopy moving images acquired from theimage capturing apparatus 2000. TheRAM 1020 also includes a work area used when theCPU 1010 executes various types of processing. - The
ROM 1030 stores data of the settings of theinformation processing apparatus 1000, boot programs, and the like. - The
hard disk drive 1050 is one example of an information storage medium, and stores programs and data for causing theCPU 1010 to execute processing of an operating system (OS) and the components illustrated inFIG. 1 . They are loaded to theRAM 1020 as appropriate in accordance with control by theCPU 1010 and are executed by theCPU 1010. Data of the X-ray fluoroscopy moving images may also be stored in thehard disk drive 1050. - The magneto-
optical disc 1060 is one example of an information storage medium, and can store a part or all of the programs and data stored in thehard disk drive 1050. - A
mouse 1070 and akeyboard 1080 can input various types of instructions to theCPU 1010 by being operated by an operator of theinformation processing apparatus 1000. - A
printer 1090 can print/output images displayed on theimage display unit 110 on/to a recording medium. - A
display apparatus 1100 is constituted by a CRT, a liquid crystal display, or the like, and can display the result of processing of theCPU 1010 using images and text. For example, thedisplay apparatus 1100 can display images that have been processed by the components illustrated inFIG. 1 and ultimately output from theimage display unit 110. In this case, theimage display unit 110 functions as a display control unit for displaying images on thedisplay apparatus 1100. Abus 1040, which connects between the components of theinformation processing apparatus 1000, enables them to exchange data with one another. - A description is now given of the
image capturing apparatus 2000. Theimage capturing apparatus 2000 is an apparatus that can capture moving images, such as an X-ray fluoroscope. Theimage capturing apparatus 2000 transmits captured image data to theinformation processing apparatus 1000. Note that theimage capturing apparatus 2000 may transmit a plurality of pieces of image data collectively to theinformation processing apparatus 1000, or may transmit a piece of image data each time it captures an image. - Aspects of the present invention can also be realized by a computer of a system or apparatus (or devices such as a CPU or MPU) that reads out and executes a program recorded on a memory device to perform the functions of the above-described embodiment(s), and by a method, the steps of which are performed by a computer of a system or apparatus by, for example, reading out and executing a program recorded on a memory device to perform the functions of the above-described embodiment(s). For this purpose, the program is provided to the computer for example via a network or from a recording medium of various types serving as the memory device (e.g., computer-readable medium).
- While the present invention has been described with reference to embodiments, it is to be understood that the invention is not limited to the disclosed embodiments.
Claims (15)
- An image processing apparatus comprising:subtraction means (107) arranged to generate digital subtraction angiography images constituted by a plurality of frames from X-ray images of an object acquired before and after injecting a radiopaque dye into the object;evaluation means (108) arranged to calculate evaluation values indicating the state of the injection of the radiopaque dye for each of the frames constituting the digital subtraction angiography images; andselection means (109) arranged to select at least one of digital subtraction angiography image using the evaluation values and to output the selected, at least one, digital subtraction angiography image as a reference image.
- The image processing apparatus according to Claim 1,
wherein the subtraction means (107) is arranged to generate each of the digital subtraction angiography images by subtracting a mask image from live images, the mask image being generated from an X-ray image of the object acquired before injecting the radiopaque dye, and the live images being X-ray images constituted by a plurality of frames acquired after injection of the radiopaque dye. - The image processing apparatus according to Claim 1 or 2,
wherein the evaluation means (108) is arranged to calculate the evaluation values from the frames constituting the digital subtraction angiography images using the number of pixels that have pixel values falling within a predetermined range with respect to a threshold. - The image processing apparatus according to Claim 3,
wherein the evaluation means (108) is arranged to calculate the evaluation values from the frames constituting the digital subtraction angiography images using the number of pixels that have pixel values falling within a predetermined range with respect to a threshold and that are continuously arranged. - The image processing apparatus according to Claim 1 or 2,
wherein the evaluation means (108) is arranged to calculate the evaluation values from the frames constituting the digital subtraction angiography images using the number of pixels at a boundary between the radiopaque dye and a background region. - The image processing apparatus according to Claim 1 or 2,
wherein the evaluation means (108) is arranged to calculate the evaluation values based on an imaging portion for which the digital subtraction angiography images have been generated, imaging technique information, imaging conditions, and a time period that has elapsed since the injection of the radiopaque dye. - The image processing apparatus according to Claim 6,
wherein the evaluation means (108) is arranged to acquire a frame range for the digital subtraction angiography images based on the imaging conditions and to calculate the evaluation values within the frame range. - The image processing apparatus according to Claim 1 or 2,
wherein the evaluation means (108) is arranged to calculate the evaluation values by comparing a minimum-value projection image with the digital subtraction angiography images, the minimum-value projection image being constituted by pixels that each have the smallest pixel value among corresponding pixels in all the frames constituting the digital subtraction angiography images. - The image processing apparatus according to Claim 1 or 2,
wherein the evaluation means (108) is arranged to calculate a plurality of feature amounts from the frames constituting the digital subtraction angiography images, and to calculate the evaluation values based on addition of the calculated plurality of feature amounts. - The image processing apparatus according to Claim 9,
wherein the evaluation means (108) is arranged to calculate the evaluation values by adding values acquired by multiplying the plurality of feature amounts calculated from the frames of the digital subtraction angiography images by weight coefficients through linear combination. - The image processing apparatus according to any of Claims 1 to 10, further comprising
image display means (110) arranged to display the reference image. - The image processing apparatus according to Claim 11,
wherein when a plurality of frames have been selected as reference images, the image display means (110) being arranged to display the reference images one by one at a predetermined frame rate. - The image processing apparatus according to any of Claims 1 to 12, further comprising
storage means (106) arranged to store the reference image. - An image processing method for an image processing apparatus, said method comprising:a subtraction step of generating digital subtraction angiography images constituted by a plurality of frames from X-ray images of an object acquired before and after injecting a radiopaque dye into the object;an evaluation step of calculating evaluation values indicating the states of the injection of the radiopaque dye for each of the frames constituting the digital subtraction angiography images; anda selection step of selecting at least one digital subtraction angiography image using the evaluation values and outputting the selected, at least one, digital subtraction angiography image as a reference image.
- A program that, when executed by an image processing apparatus, causes the image processing apparatus to perform an image processing method according to claim 14.
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2012106311A JP6021420B2 (en) | 2012-05-07 | 2012-05-07 | Image processing apparatus, image processing method, and program |
Publications (2)
Publication Number | Publication Date |
---|---|
EP2662830A1 true EP2662830A1 (en) | 2013-11-13 |
EP2662830B1 EP2662830B1 (en) | 2018-10-10 |
Family
ID=48184034
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP13163985.8A Not-in-force EP2662830B1 (en) | 2012-05-07 | 2013-04-16 | Selection of a reference image in interventional vascular radiology |
Country Status (4)
Country | Link |
---|---|
US (1) | US9396535B2 (en) |
EP (1) | EP2662830B1 (en) |
JP (1) | JP6021420B2 (en) |
CN (1) | CN103385692B (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10499864B2 (en) | 2014-09-22 | 2019-12-10 | Koninklijke Philips N.V. | Contrast arrival detection |
Families Citing this family (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5981220B2 (en) * | 2012-05-21 | 2016-08-31 | 東芝メディカルシステムズ株式会社 | Medical image processing apparatus and X-ray imaging apparatus |
JP6071444B2 (en) | 2012-11-07 | 2017-02-01 | キヤノン株式会社 | Image processing apparatus, operation method thereof, and program |
JP6371515B2 (en) | 2013-11-13 | 2018-08-08 | キヤノン株式会社 | X-ray image processing apparatus, X-ray image processing method, and program |
JP6305747B2 (en) * | 2013-12-11 | 2018-04-04 | キヤノンメディカルシステムズ株式会社 | Medical image processing apparatus and X-ray diagnostic apparatus |
CN104156931B (en) * | 2014-09-04 | 2017-03-29 | 成都金盘电子科大多媒体技术有限公司 | A kind of digital subtraction angiography method |
JP6664873B2 (en) | 2014-11-21 | 2020-03-13 | キヤノンメディカルシステムズ株式会社 | Image processing apparatus, X-ray diagnostic apparatus, and image processing program |
JP6706339B2 (en) * | 2016-10-27 | 2020-06-03 | 富士フイルム株式会社 | Endoscope system |
US11191503B2 (en) | 2018-07-17 | 2021-12-07 | International Business Machines Corporation | Fluid-injector for a simultaneous anatomical and fluid dynamic analysis in coronary angiography |
CN109816650B (en) * | 2019-01-24 | 2022-11-25 | 强联智创(北京)科技有限公司 | Target area identification method and system based on two-dimensional DSA image |
JP7442976B2 (en) * | 2019-04-10 | 2024-03-05 | 株式会社島津製作所 | X-ray imaging device |
US10996306B2 (en) * | 2019-04-25 | 2021-05-04 | General Electric Company | MRI system and method using neural network for detection of patient motion |
CN112554853A (en) * | 2020-12-11 | 2021-03-26 | 成都北方石油勘探开发技术有限公司 | Method and system for controlling water injection or gas injection |
CN113989171A (en) * | 2021-09-06 | 2022-01-28 | 北京东软医疗设备有限公司 | Subtraction map generation method and device, storage medium and computer equipment |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH01221142A (en) | 1988-02-29 | 1989-09-04 | Shimadzu Corp | X-ray image treating device |
US20050089143A1 (en) * | 2003-09-19 | 2005-04-28 | Kabushiki Kaisha Toshiba | X-ray diagnosis apparatus and method for creating image data |
Family Cites Families (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH0368348A (en) * | 1989-08-08 | 1991-03-25 | Toshiba Corp | X-ray image display device |
US6195450B1 (en) * | 1997-09-18 | 2001-02-27 | Siemens Corporate Research, Inc. | Methods and apparatus for controlling X-ray angiographic image acquisition |
US7013035B2 (en) * | 1998-09-25 | 2006-03-14 | Canon Kabushiki Kaisha | Image processing method for setting an extraction area, and apparatus and recording medium |
DE10064768B4 (en) | 2000-12-22 | 2006-12-07 | Siemens Ag | Method for examining a living being by means of an imaging method |
US6754522B2 (en) * | 2001-09-05 | 2004-06-22 | Medimag C.V.I., Inc. | Imaging methods and apparatus particularly useful for two and three-dimensional angiography |
JP4505805B2 (en) * | 2004-08-02 | 2010-07-21 | 横河電機株式会社 | Region extraction method and apparatus |
JP4150005B2 (en) * | 2005-01-31 | 2008-09-17 | 株式会社東芝 | Image diagnostic apparatus, image processing method, image processing apparatus, and storage medium |
JP4936929B2 (en) * | 2007-02-28 | 2012-05-23 | 株式会社東芝 | X-ray diagnostic apparatus and image display method |
JP4979125B2 (en) * | 2007-05-18 | 2012-07-18 | 富士フイルム株式会社 | Radiation imaging system and program |
JP4981578B2 (en) * | 2007-08-14 | 2012-07-25 | キヤノン株式会社 | Radiation imaging apparatus and control method thereof |
JP5197140B2 (en) | 2008-05-07 | 2013-05-15 | キヤノン株式会社 | X-ray fluoroscopic apparatus, moving image processing method, program, and storage medium |
JP5366618B2 (en) * | 2009-04-06 | 2013-12-11 | 株式会社東芝 | X-ray diagnostic imaging equipment |
JP2011036433A (en) * | 2009-08-11 | 2011-02-24 | Toshiba Corp | X-ray diagnostic apparatus and method for generating x-rays diagnostic image |
JP5610761B2 (en) | 2009-12-16 | 2014-10-22 | キヤノン株式会社 | X-ray image processing apparatus, X-ray image processing system, X-ray image processing method, and computer program |
JP5645399B2 (en) | 2009-12-18 | 2014-12-24 | キヤノン株式会社 | X-ray image processing apparatus, X-ray image processing method, and computer program |
JP5631030B2 (en) | 2010-03-16 | 2014-11-26 | キヤノン株式会社 | Radiation image processing apparatus, image processing method, radiation imaging system, and program |
JP5537262B2 (en) | 2010-05-28 | 2014-07-02 | 株式会社東芝 | X-ray diagnostic imaging equipment |
JP5680654B2 (en) | 2010-08-31 | 2015-03-04 | 株式会社日立メディコ | Ultrasonic diagnostic apparatus and ultrasonic image display method |
JP5691341B2 (en) * | 2010-09-22 | 2015-04-01 | 株式会社島津製作所 | Radiation imaging apparatus and image reproduction apparatus |
-
2012
- 2012-05-07 JP JP2012106311A patent/JP6021420B2/en active Active
-
2013
- 2013-04-16 EP EP13163985.8A patent/EP2662830B1/en not_active Not-in-force
- 2013-04-22 US US13/867,266 patent/US9396535B2/en not_active Expired - Fee Related
- 2013-05-07 CN CN201310163992.5A patent/CN103385692B/en active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH01221142A (en) | 1988-02-29 | 1989-09-04 | Shimadzu Corp | X-ray image treating device |
US20050089143A1 (en) * | 2003-09-19 | 2005-04-28 | Kabushiki Kaisha Toshiba | X-ray diagnosis apparatus and method for creating image data |
Non-Patent Citations (2)
Title |
---|
"SPECTUM E-5000 Cardiac Catheterization Lab", WWW.ARCHIVE.ORG, 13 February 2011 (2011-02-13), pages 1 - 2, XP055073858, Retrieved from the Internet <URL:http://web.archive.org/web/20110213065536/http://www.emdmedical.com/i_pro_cardiology.html> [retrieved on 20130801] * |
ROSE JOHNSON ET AL: "Exploring the potential for touchless interaction in image-guided interventional radiology", CHI 2011, 7 May 2011 (2011-05-07), pages 3323 - 3332, XP055073761, ISBN: 978-1-45-030228-9, DOI: 10.1145/1978942.1979436 * |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10499864B2 (en) | 2014-09-22 | 2019-12-10 | Koninklijke Philips N.V. | Contrast arrival detection |
Also Published As
Publication number | Publication date |
---|---|
US9396535B2 (en) | 2016-07-19 |
EP2662830B1 (en) | 2018-10-10 |
US20130294674A1 (en) | 2013-11-07 |
CN103385692A (en) | 2013-11-13 |
CN103385692B (en) | 2015-09-02 |
JP6021420B2 (en) | 2016-11-09 |
JP2013233227A (en) | 2013-11-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9396535B2 (en) | Image processing apparatus, image processing method and storage medium to generate subtraction image | |
EP1820450B1 (en) | X-ray diagnostic apparatus, image processing apparatus, and program | |
CN108703764B (en) | Angiography method, device, system, equipment and storage medium | |
EP1604612B1 (en) | X-ray CT apparatus for myocardial perfusion image generation | |
JP5610761B2 (en) | X-ray image processing apparatus, X-ray image processing system, X-ray image processing method, and computer program | |
US20130066198A1 (en) | Contrast Agent Perfusion Adaptive Imaging System | |
US8633945B2 (en) | X-ray diagnostic apparatus and image processing apparatus | |
US20090185730A1 (en) | Automated Image Data Subtraction System Suitable for Use in Angiography | |
US9275439B2 (en) | Image processing apparatus, control method thereof and computer-readable storage medium | |
US20140193082A1 (en) | Image processing apparatus and method | |
EP1987771A1 (en) | Radiation image pickup device and method of processing radiation detecting signal | |
US11179122B2 (en) | Bolus imaging | |
JP2004112469A (en) | Image diagnostic apparatus, image processing method, image processor, and storage medium | |
JPH05192319A (en) | X-ray diagnostic device | |
JP2005198330A (en) | Image diagnosis apparatus, image processing method, image processing apparatus and storage medium | |
EP3975850B1 (en) | X-ray imaging system | |
JP6605220B2 (en) | X-ray imaging device | |
JP5229496B2 (en) | Image processing device | |
US20130094744A1 (en) | Image processing device | |
CN111603189B (en) | Method for operating an X-ray device during the execution of an X-ray examination and X-ray device | |
JP5591963B2 (en) | X-ray diagnostic equipment | |
JP7503996B2 (en) | Nuclear medicine diagnostic device and medical information processing device | |
JP5488003B2 (en) | X-ray imaging method and X-ray imaging apparatus | |
JP2010022667A (en) | Cerebral blood flow analyzer | |
JPS5917330A (en) | X-ray diagnostic apparatus |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
17P | Request for examination filed |
Effective date: 20140318 |
|
RBV | Designated contracting states (corrected) |
Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20180601 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP Ref country code: AT Ref legal event code: REF Ref document number: 1052129 Country of ref document: AT Kind code of ref document: T Effective date: 20181015 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D Ref country code: DE Ref legal event code: R096 Ref document number: 602013044734 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20181010 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1052129 Country of ref document: AT Kind code of ref document: T Effective date: 20181010 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190210 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190110 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190110 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190210 Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190111 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602013044734 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 |
|
26N | No opposition filed |
Effective date: 20190711 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20190430 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190416 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190430 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190430 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190430 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190416 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20130416 Ref country code: MT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20220323 Year of fee payment: 10 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20220323 Year of fee payment: 10 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181010 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20220322 Year of fee payment: 10 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R119 Ref document number: 602013044734 Country of ref document: DE |
|
GBPC | Gb: european patent ceased through non-payment of renewal fee |
Effective date: 20230416 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20230416 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20230416 Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20230430 Ref country code: DE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20231103 |