CN114332285B

CN114332285B - Method and device for generating coronary artery path map and readable storage medium

Info

Publication number: CN114332285B
Application number: CN202210234976.XA
Authority: CN
Inventors: 翟光耀; 王建龙; 郭倩云; 刘宇扬; 孙铁男; 陈政; 王琳; 刘春燕; 解菁
Original assignee: Beijing Anzhen Hospital
Current assignee: Beijing Anzhen Hospital
Priority date: 2022-03-11
Filing date: 2022-03-11
Publication date: 2022-06-28
Anticipated expiration: 2042-03-11
Also published as: CN114937100A; CN114332285A

Abstract

The invention provides a method, a device and a readable storage medium for generating a coronary artery path map, wherein the method for generating the coronary artery path map comprises the steps of acquiring a dynamic perspective image of a coronary artery; acquiring a sequence of angiographic images of a coronary artery over a preset time period during which the heart performs a periodic motion; matching an angiographic image corresponding to the phase of the fluoroscopy images in the sequence of angiographic images; the fluoroscopic image is fused with the phase-coincident angiographic image to generate a dynamic coronary roadmap, which provides real-time image guidance, improves accuracy and surgical efficiency, and reduces fluoroscopy time and contrast agent usage.

Description

Method and device for generating coronary artery path map and readable storage medium

Technical Field

The present invention relates to image processing technologies, and in particular, to a method, an apparatus, and a readable storage medium for generating a coronary artery road map.

Background

The amount of contrast agent required for Percutaneous Coronary Intervention (PCI) depends on a number of factors: firstly the experience of the interventionalist is of the utmost importance, secondly the complexity of the anatomy, and secondly the targeted vascular target lesion and the complexity of the surgery. Currently, during PCI, the operation of advancing the guide wire in the coronary artery, advancing and positioning the balloon catheter and related instruments, etc. requires repeated contrast bolus injection in the coronary artery for assistance or confirmation.

In the PCI operation, especially under the conditions of more coronary branches, blood vessel tortuosity, bulkier patient body, poor quality of fluoroscopic images and more overlapped blood vessel images, the identification degree of target lesion of target blood vessels is obviously reduced for an operator, the difficulty of interventional operation is increased, the contrast agent needs to be injected repeatedly for assistance or confirmation, the use amount of the contrast agent is increased, and the operation time is prolonged. It is known that contrast-related complications including contrast-related nephropathy, contrast-related allergy, etc. may be life-threatening in severe cases, and the occurrence of the above complications is positively correlated to the amount of contrast agent used. The traditional PCI operation needs to use a contrast medium for multiple times to determine the positions of a guide wire and a blood vessel, and the traditional path diagram manufacturing method comprises the calculation processes of taking a mask, taking a target, subtracting and superposing, and the like, namely, the traditional path diagram manufacturing method can only guide static blood vessels, and is not suitable for the PCI operation environment with heart beating and real-time dynamic state. Conventional dynamic coronary roadmapping requires a large amount of fluoroscopy time and contrast agent dose. The inventor finds that the existing heart dynamic roadmap brings certain help to a clinician, but the existing heart dynamic roadmap only considers the cardiac cycle (ECG) and does not consider the influence of respiratory motion, and the heart vessel morphology is influenced by the respiratory motion, so that the traditional heart dynamic roadmap is not high in precision, cannot meet the requirement of accurate judgment in clinic, and the clinical use frequency of the dynamic roadmap function is not high. In addition, not only is the conventional vessel segmentation method slow, but also the ECG signal access needs to put higher demands on device compatibility.

Disclosure of Invention

The present invention has been made to solve the above-mentioned problems occurring in the prior art. There is a need for a method, an apparatus, and a readable storage medium for generating a coronary artery road map, which can reduce the amount of contrast agent used and reduce the adverse effect of the contrast agent on the human body. Meanwhile, the marked target containing two motion characteristics of respiration and heartbeat can be segmented by utilizing the deep learning network, and a dynamic coronary artery path diagram with higher accuracy is obtained in an efficient mode so as to meet the requirement of real-time performance.

According to a first aspect of the present invention, a method of generating a coronary artery roadmap comprises acquiring a dynamic fluoroscopic image of a coronary artery; acquiring a sequence of angiographic images of a coronary artery over a preset time period during which the heart performs a periodic motion; matching an angiographic image corresponding to the phase of the fluoroscopic image in the sequence of angiographic images; the fluoroscopic image is fused with the phase-coincident angiographic image to generate a dynamic coronary artery road map.

According to a second aspect of the invention, an apparatus for generating a coronary artery road map comprises a processor configured to perform a method of generating a coronary artery road map according to various embodiments of the invention.

According to a third aspect of the present invention, a computer-readable storage medium has stored thereon computer program instructions which, when executed by a processor, cause the processor to execute a method of generating a coronary roadmap according to various embodiments of the present invention.

Compared with the prior art, the embodiment of the invention has the following beneficial effects:

matching an angiography image corresponding to the phase of the fluoroscopy image in the sequence of angiography images, wherein the matching precision is high, and fusing the fluoroscopy image and the angiography image corresponding to the phase to generate a dynamic coronary artery path map with high accuracy. The method is not only beneficial to reducing the use amount of the contrast agent and reducing the harm of the contrast agent to a human body, but also can meet the requirement of real-time performance, provide automatic and real-time image guidance for a PCI operation, provide continuous and specific position feedback of a guide wire and a blood vessel for a doctor, and greatly reduce the risk caused by blind puncture.

The foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to limit the invention as claimed.

Drawings

In the drawings, which are not necessarily drawn to scale, like reference numerals may describe similar parts throughout the different views. Like reference numerals having letter suffixes or different letter suffixes may represent different examples of similar components. The drawings illustrate generally, by way of example, but not by way of limitation, various embodiments and, together with the description and claims, serve to explain the disclosed embodiments. Such embodiments are illustrative and exemplary and are not intended to be exhaustive or exclusive embodiments of the present method, apparatus, system, or non-transitory computer-readable medium having instructions for carrying out the method.

FIG. 1 shows a flow chart of a method of generating a coronary artery road map according to an embodiment of the invention;

FIG. 2 shows a representation of a perspective image according to an embodiment of the invention;

FIG. 3 shows a flowchart of a method of matching an angiographic image corresponding to a phase of a fluoroscopic image in a sequence of angiographic images according to an embodiment of the invention;

FIG. 4 shows a graphical representation of an angiographic image obtained after image registration in accordance with an embodiment of the invention;

fig. 5 shows a diagram of a coronary artery path map generated by the method of generating a coronary artery path map according to an embodiment of the present invention;

Fig. 6 shows a system for performing a method of generating a coronary roadmap in accordance with an embodiment of the invention.

Detailed Description

In order to make the technical solutions of the present invention better understood, the present invention will be described in detail below with reference to the accompanying drawings and the detailed description. The following detailed description of the embodiments of the present invention is provided in connection with the accompanying drawings and the specific embodiments, but not intended to limit the invention. The order in which the various steps described herein are described as examples should not be construed as a limitation if there is no requirement for a contextual relationship between each other, and one skilled in the art would know that sequential adjustments may be made without destroying the logical relationship between each other, rendering the overall process impractical.

The use of "first," "second," and the like, herein does not denote any order, quantity, or importance, but rather the terms "first," "second," and the like are used to distinguish one element from another. The word "comprising" or "comprises", and the like, means that the element preceding the word comprises the element listed after the word, and does not exclude the possibility that other elements may also be included. "upper", "lower", "left", "right", and the like are used only to indicate relative positional relationships, and when the absolute position of the object being described is changed, the relative positional relationships may also be changed accordingly.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs unless specifically defined otherwise. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein. Techniques, methods, and apparatus known to one of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.

Fig. 1 shows a flow chart of a method of generating a coronary artery road map according to an embodiment of the invention.

As shown in step S101, the method for generating the coronary artery road map includes acquiring a dynamic fluoroscopic image of the coronary artery, where the dynamic fluoroscopic image may be a dynamic fluoroscopic image acquired in real time during an operation, or a dynamic fluoroscopic image extracted from an image database or a dynamic fluoroscopic image acquired based on another method, and is not limited in particular. The acquisition mode of the fluoroscopic image or the photographic image includes, but is not limited to, direct acquisition by various imaging modalities, such as, but not limited to, medical contrast imaging techniques such as CT, MR, radionuclide scan, spiral CT, positron emission tomography, X-ray imaging, fluorescence imaging, and ultrasound imaging, or reconstruction based on an original image acquired by an imaging device. For example, a fluoroscopic scan is performed to obtain a fluoroscopic image with the X-ray imaging aligned with the center on the estimated target position. The technical term "acquisition" refers, inter alia, to any means of direct or indirect acquisition, with or without additional noise reduction, cropping, reconstruction, etc. image processing.

As shown in step S102, a sequence of angiographic images of a preset time segment of the coronary arteries is acquired, during which a periodic movement of the heart is performed. The angiographic image is an image of a vascular structure, and is a contrast image obtained by a Digital Subtraction Angiography (DSA) apparatus or the like. The digital subtraction angiography is to inject a transparent contrast agent containing organic compounds into blood flow rapidly under X-ray irradiation, develop blood vessels under X-ray irradiation, photograph the development process of blood vessel cavities, and see the blood flow sequence containing the contrast agent and the blood vessel filling condition from the development result, thereby knowing the physiological and anatomical changes of the blood vessels.

Specifically, for example, a sequence of angiographic images of a preset time period of the coronary artery is acquired under optimal contrast agent injection. Wherein the optimal contrast agent may refer to an image taken after injecting the contrast agent into a region of interest of the coronary artery, while the blood vessels are in a filled state, capable of continuing to complete at least one cardiac motion cycle. The contrast agent can be added into the blood of the target object in an arterial injection mode, during the shooting process of the image, the characteristic that the contrast agent has strong attenuation on X-ray imaging is utilized, the development of the vascular system of the target object is realized, so that a gray-scale image of blood vessel filling is obtained, and the diagnosis and treatment of vascular diseases can be assisted through the observation of the blood vessels in the image.

Step S101 and step S102 do not limit the sequence, for example, a dynamic fluoroscopic image of the coronary artery may be acquired first, and then a sequence of angiographic images of the coronary artery in a preset time period may be acquired; or acquiring a sequence of angiography images of the coronary artery in a preset time period, and then acquiring a dynamic perspective image of the coronary artery; it is also possible to simultaneously acquire a dynamic fluoroscopic image of the coronary artery and a sequence of angiographic images of the coronary artery for a preset period of time.

In step S103, an angiogram image corresponding to the phase of the fluoroscopic image is matched in the sequence of angiogram images to match the angiogram image that matches best with the current fluoroscopic image, i.e. at least at the corresponding cardiac cycle phase, for generating a dynamic coronary artery road map. Note that the coincident phases include at least the cardiac cycle phase and may further include the phase of the respiratory motion cycle. That is, at least the phases of the cardiac cycle coincide, and both phases of the cardiac cycle and the respiratory motion cycle may coincide. Matching is understood to mean either identical or an acceptable deviation, and in the case of matching, a reasonable deviation is allowed in the phase between the two, which deviation is of course within a reasonable range to ensure the accuracy of matching. The matching method includes, but is not limited to, a feature-based image registration method, for example, extracting image features in two images, and performing registration on the two images based on the image features to obtain a correspondence between image points of the same name in the two images, which is only taken as an example, and other methods capable of achieving registration may also be included. The specific matching method is not particularly limited as long as an angiographic image corresponding to the phase of the fluoroscopic image can be matched in the sequence of angiographic images based on the concept of this embodiment.

In step S104, the fluoroscopic image and the phase-matched angiographic image are fused to generate a dynamic coronary artery road map, so as to obtain a dynamic and real-time coronary artery road map with higher accuracy, thereby providing a precise navigation for a doctor to perform a PCI operation, improving the operation efficiency, and shortening the operation time. Based on the embodiment, more contrast agents are not needed, a more definite and more visual guiding effect is exerted on guide wire operation, and operation risks caused by blind puncture are avoided. For example, when a patient has a heart condition, a physician may inject a contrast agent into the patient in advance, keeping a region of interest of the heart full, before performing a cardiac procedure, and acquire a sequence of angiographic images. During the operation, the doctor can inject a small amount of contrast agent into the patient body even without injecting the contrast agent again, match out the angiographic image corresponding to the phase of the intraoperative real-time fluoroscopic image in the sequence of the angiographic images, and fuse the fluoroscopic image and the angiographic image corresponding to the phase, so as to generate a real-time and accurate coronary artery road map. This embodiment is merely an example, and is not intended as a specific limitation on the coronary artery road map generation method. Furthermore, with this embodiment, no external device intervention is required, and device compatibility is accordingly improved.

In some embodiments, the dynamic fluoroscopic images of coronary arteries include intraoperative real-time fluoroscopic images of coronary arteries to adapt to a beating heart, real-time dynamic PCI surgical environment, meeting the requirement of real-time performance. In the operation process, along with the complex heartbeat process, a real-time dynamic fluoroscopic image of the coronary artery can be obtained, as shown in fig. 2, for the dynamic fluoroscopic image, the accurate registration of the angiography image which is consistent with the phase can be rapidly realized in the operation, and then a real-time dynamic coronary artery path diagram is presented, so that a doctor can directly observe a clear blood vessel path.

Further, the preset time period at least comprises one respiratory motion cycle, or a least common multiple cycle of the respiratory motion cycle and the cardiac cycle. The cardiac cycle can only reflect the heartbeat motion and does not include respiratory motion related states. The invention finds that only the cardiac cycle is considered, the influence of respiratory motion is not considered, the motion cycle of the heart cannot be comprehensively embodied, and accurate registration cannot be realized. This embodiment allows for taking into account the influence of the respiratory motion cycle, so that the heart performs a periodic motion during said preset time period, which is distinguished from the cardiac cycle. The periodic motion performed by the heart in the preset time period may be one or more respiratory motion cycles to represent the periodic motion of the heart, or may also be one or more respiratory motion cycles and a least common multiple cycle of a cardiac cycle to represent the periodic motion of the heart, so that the sequence of the acquired angiographic images can return the heart to an original position (hereinafter, also referred to as a complete cardiac motion cycle) after undergoing several cardiac contraction cycles and several respiratory motion cycles, and further can reflect the joint variation process of the phase of the cardiac cycle and the phase of the respiratory motion cycle in a week-to-week manner. In this way, accurate registration with the real-time fluoroscopic image can be achieved at both phases. The minimum complete cardiac motion cycle is the least common multiple of the respiratory motion cycle and the cardiac cycle.

In some embodiments, the preset time period may be exactly the least common multiple of the respiratory motion cycle and the cardiac cycle, such that it returns the heart to the original position with the shortest duration. Specifically, the start time of the preset time period may be set to the start time of the first cardiac cycle and the start time of the first respiratory motion cycle. Without being limited to this embodiment, the predetermined time period may also be greater than the least common multiple of the respiratory motion cycle and the cardiac cycle to ensure that the heart returns to the original position at least once. For example, the preset time period is a least common multiple cycle and a respiratory motion cycle or a cardiac cycle, or other means. Therefore, key characteristics of heartbeat movement and respiratory movement can be fully considered, the registration precision is improved, a dynamic real-time arterial coronary path diagram is obtained, automatic and real-time image guidance is provided for a PCI operation, continuous and specific position feedback of guide wires and blood vessels is provided for a doctor, and the risk caused by blind puncture is reduced.

In some embodiments, fusing the fluoroscopic image with the phase-coincident angiographic image to generate a dynamic coronary artery path map further comprises rendering segmented vessels of the coronary arteries in the phase-coincident angiographic image over the coronary arteries on the fluoroscopic image. After accurate registration, an angiographic image is obtained that coincides in phase with the fluoroscopy image, which has the same phase as the current fluoroscopy image, e.g. the extracted angiographic image may have the same phase of the cardiac cycle as the current fluoroscopy image, or the same phase of the cardiac cycle together with the phase of the respiratory motion cycle, to ensure accurate registration in both phases. The perspective image and the angiography image which is consistent with the phase are fused, so that blood vessels which cannot be displayed in the perspective image or blood vessels with poor definition are presented clearly and synchronously.

The fusion is to extract respective favorable information to the utmost extent by processing the images to be fused and adopting a computer technology and the like, and finally synthesize the images with high quality to improve the resolution of the original images. In some embodiments, the image fusion can be divided into three levels from low to high, namely data level fusion, feature level fusion and decision level fusion. Data-level fusion, also called pixel-level fusion, refers to a process of directly processing data to obtain a fused image, which is the basis of high-level image fusion. In this embodiment, the specific manner of fusion is not limited.

Furthermore, the fusion of the fluoroscopic image and the phase-matched angiographic image to generate a dynamic coronary artery path map may further include presenting segmented vessels of the coronary artery in the phase-matched angiographic image on the coronary artery on the fluoroscopic image, and changing a visual effect of each segmentation result according to a difference in confidence thereof, so that a user can intuitively know the segmented vessels and a reliability thereof, and a workload is reduced. For example, during surgery, the surgeon is preoccupied with a heavy workload. The system automatically identifies the confidence of the segmented blood vessel and presents different visual effects based on the confidence, specifically, when the confidence of the segmented blood vessel is low, the blood vessel is probably not an actual blood vessel, the system gives the blood vessel light pink, for the blood vessel with high confidence, the system gives the blood vessel with darker red color, and a doctor can quickly distinguish the position and reliability of the blood vessel through the difference of the visual effects of the segmentation results, quickly make an intraoperative reaction, reduce the workload and improve the working efficiency, which is only taken as a specific embodiment and is not a specific limiting mode.

In some embodiments, the confidence level may characterize the uncertainty level. In this way, the part of the certainty level exceeding the threshold can be presented to the doctor, and the part that needs to be presented to the doctor is presented in a visual effect, so as to prompt the doctor to pay attention to which information, and the information that does not need to be presented to the doctor is directly screened out. The doctor can use the segmentation result with high confidence degree to perform detailed analysis, and in contrast, the doctor can only use the remaining power to refer to the segmentation result with low confidence degree, so as to realize the optimal configuration of the workload, and simultaneously, additional information is provided for the doctor to serve as a reference.

For example, when the confidence is high, the system may give a visual effect of cool tone, and may also give a visual effect of warm tone, so as to distinguish the visual effects.

In some embodiments, as shown in fig. 3, matching an angiographic image in the sequence of angiographic images that coincides with the phase of the fluoroscopic image specifically comprises:

step S301 shows a first image segmentation of each angiographic image of the sequence of angiographic images to extract a first marker tracking respiratory motion and a second marker tracking cardiac motion. By tracking the first marker of the respiratory motion and the second marker of the heartbeat motion, an angiography sequence at least containing one complete cardiac motion cycle (namely, a common multiple cycle of the respiratory motion cycle and the cardiac cycle) is obtained, and key features are provided for subsequent registration. The first marker for tracking the respiratory motion is used for representing the periodic variation track of the respiratory motion, the second marker for tracking the heartbeat motion is used for representing the periodic variation track of the heartbeat motion, and image registration is carried out based on the first marker and the second marker, so that the registration accuracy is improved.

Step S302 shows performing second image segmentation on the perspective image to extract a corresponding first marker and a corresponding second marker, where when performing the second image segmentation, the extracted first marker and the extracted second marker are the same as the first marker and the extracted second marker in step S301, and are respectively used for representing a periodic variation trajectory of respiratory motion and a periodic variation trajectory of heartbeat motion.

Step S303 shows registering the fluoroscopic image with the sequence of angiographic images by comparing geometrically relevant features of the first marker and the second marker and/or the relative position between the first marker and the second marker extracted by the first image segmentation and the second image segmentation, respectively, thereby matching an angiographic image in line with the phase of the fluoroscopic image in the sequence of angiographic images. By comprehensively considering the geometrically relevant features of the first and second markers and/or the relative positions between the first and second markers, an efficient and accurate registration can be achieved, registration errors are avoided, and the registration speed is adapted to the intraoperative time-limited dynamic registration requirements.

Specifically, in the actual heart movement process, both the heartbeat movement (i.e., the cardiac cycle reflects the heartbeat movement) and the respiratory movement (i.e., the respiratory movement cycle reflects the respiratory movement) are influenced by the respiratory movement, and when the heart movement process is matched with the respiratory movement process, the whole position may rotate or change in position, and the motion cycle is judged by combining the respiratory movement and the heartbeat movement, so that higher registration accuracy can be realized. In a specific registration process, the registration may be performed in different ways. For example, a first marker extracted from a first image segmentation of an angiographic image sequence is matched with a geometrically related feature of the first marker extracted from a second image segmentation of a real-time fluoroscopic image, and the matching is performed by spatial transformation, structural matching, or pixel superposition, so as to realize registration based on the geometrically related feature of the first marker. Registration based on the geometrically relevant feature of the second marker is then performed in a similar manner. Alternatively, geometric correlation features of a second marker extracted from the angiographic image sequence by the first image segmentation and a second marker extracted from the real-time fluoroscopic image by the second image segmentation may be registered while matching geometric correlation features of the first marker extracted from the angiographic image sequence by the first image segmentation and the first marker extracted from the real-time fluoroscopic image by the second image segmentation.

Alternatively, only the relative position may be registered at the time of matching. For example, the relative positions of the first marker and the second marker extracted from the angiographic image are matched with the relative positions of the first marker and the second marker extracted from the fluoroscopic image, and an angiographic image in phase with the fluoroscopic image is obtained after accurate matching.

Or, when in registration, the geometric relevant features and the relative positions of different key features are comprehensively considered, and based on the two aspects, the registration accuracy is further improved. For example, the first way may be to match the first and second markers extracted from the angiographic image sequence with the geometrically relevant features of the first and second markers extracted from the fluoroscopic image, and then to match the relative position between the first and second markers extracted from the angiographic image sequence with the relative position between the first and second markers extracted from the fluoroscopic image after the matching is completed. Due to the complexity of the heart motion, when the registration is performed based on the geometric correlation feature, the deviation may occur, and by further performing the registration based on the relative position, the situation that the registration is not performed based on the geometric correlation feature is favorably excluded. A second way may be that the matching with respect to the geometrically related features and the matching with respect to the relative position are performed simultaneously, which greatly improves the accuracy of the registration by taking both factors into account.

In some embodiments, the first markers include anatomical landmarks of the diaphragm and/or lungs and the second markers include a catheter and/or silhouette of the heart affixed to the ostium of the coronary artery, wherein the diaphragm is a closely related feature of respiratory motion and can be used to track the trajectory of respiratory motion and the catheter is affixed to the ostium of the coronary artery for tracking the trajectory of heartbeat motion. The anatomical landmarks of the lung include, but are not limited to, trachea, blood vessels. The diaphragm and the catheter have respective geometric related features, and meanwhile, the relative position between the diaphragm and the catheter is changed in the process of breathing movement and heartbeat movement. The diaphragm and the catheter are comprehensively considered for registration, so that not only are necessary image characteristics reserved, but also the range of the image to be registered is reduced, and the image registration efficiency is improved, and meanwhile, the image registration precision is also improved.

In some embodiments, registering the fluoroscopic image with the sequence of angiographic images by comparing the geometrically related features of the first marker and the second marker extracted by the first image segmentation and the second image segmentation, respectively, and/or the relative position between the first marker and the second marker, particularly comprises first registering the fluoroscopic image with the sequence of angiographic images by comparing the geometrically related features of the first marker extracted by the first image segmentation and the second image segmentation, respectively, together with the relative position between the first marker and the second marker, so as to match a first angiographic image in the sequence of angiographic images coinciding with the phase of the fluoroscopic image. Specifically, for example, the first marker is a diaphragm muscle, the second marker is a catheter fixed at the mouth of a coronary artery, and during the matching process, the geometric relevant features of the diaphragm muscle can be compared with the relative position of the diaphragm muscle and the catheter, and the specific manner for comparison can include two. For example, one method is to compare the geometric relevant features of the diaphragm, determine whether the geometric relevant features of the diaphragm are registered, and compare the relative positions of the diaphragm and the catheter after registration to improve the registration accuracy; the other is to compare the relative positions of the diaphragm and the catheter while comparing the geometrically related features of the diaphragm, and to match the geometrically related features and the relative positions. A first angiographic image is obtained which matches the phase of the fluoroscopy images in the sequence of angiographic images.

The registration method further comprises a second registration of the fluoroscopic image with the sequence of angiographic images by comparing geometrically related features of the second marker extracted by the first and second image segmentations, respectively, together with the relative position between the first marker and the second marker, matching a second angiographic image in line with the phase of the fluoroscopic image in the sequence of angiographic images. Specifically, for example, the second marker is a catheter, the first marker is a diaphragm, when the catheter is registered, the geometric relevant features of the catheter are comprehensively registered together with the relative positions of the diaphragm and the catheter, and the specific manner of the registration may also include two. For example, one method is to compare the geometric relevant features of the catheter, determine whether the geometric relevant features of the catheter are registered, and compare the relative positions of the diaphragm and the catheter after registration, so as to improve the registration accuracy; the other is to compare the relative positions of the diaphragm and the catheter while comparing the geometrically related features of the catheter, and to match the geometrically related features and the relative positions. A second angiographic image is obtained by this embodiment which matches out in the sequence of angiographic images a phase which coincides with the phase of the fluoroscopic image.

Because the position changes such as rotation and the like are caused by the influence of respiratory motion, and deviation may occur when registration is performed based on a single key feature and a relative position, therefore, after the first angiographic image and the second angiographic image are acquired, the difference degree between the first angiographic image and the second angiographic image is determined, and when the difference degree is lower than a threshold value, a final matched angiographic image which is consistent with the phase of the perspective image is obtained by using the first angiographic image and/or the second angiographic image, so as to further improve the reliability of the registration result. Specifically, when the difference between the first angiographic image and the second angiographic image is lower than a threshold value, it indicates that the registration accuracy of acquiring the first angiographic image and the second angiographic image is high, and the registration accuracy has high reliability, and a final matching angiographic image that coincides with the phase of the fluoroscopic image may be obtained based on the first angiographic image and/or the second angiographic image. Conversely, if the difference between the first angiographic image and the second angiographic image is higher than a threshold, indicating that the registration accuracy is poor, re-registration is required. The threshold may be set based on actual needs, for example, may be set already at the time of system shipment, or may be a numerical value manually set by a user based on actual conditions.

In another embodiment, the registration method further comprises comparing the geometrically related features of the first marker and the second marker extracted by the first image segmentation and the second image segmentation, respectively, together with the relative position between the first marker and the second marker, registering the fluoroscopic images with the sequence of angiographic images, thereby matching angiographic images in line with the phase of the fluoroscopic images in the sequence of angiographic images, the angiographic images obtained by the embodiment having higher accuracy. For example, if the first marker is the diaphragm and the second marker is the catheter, a mismatch may occur during the registration process if only the catheter or the diaphragm is used. In the heart movement process, the shape of the whole coronary artery can be influenced by the respiratory movement to generate some rotation and movement, if only a catheter is considered, the respiration is not considered, and the deviation on some phase points is larger. Specifically, for example, when the current respiratory cycle moves down at the highest point and up from the lowest point, the heartbeat may have the same cycle, i.e. the cardiac cycle is the same, but the respiratory cycle may be opposite, resulting in a larger deviation of the registration. Thus, whether only the geometric correlation features are considered or only the relative positions are considered, it is possible that the same frame appears at different periods, and is therefore mismatched.

Based on the registration method described in the embodiment, the geometric correlation characteristics of the first marker and the second marker are considered together with the relative positions, and comprehensive comparison registration is performed, so that high-precision registration can be ensured, and the registration error is avoided. For example, registration based on the key features of the diaphragm and the catheter still includes two ways, one is to register the geometrically related features of the diaphragm and the catheter first, and compare the relative positions of the diaphragm and the catheter after registration is correct; the other method is to compare the relative position relationship of the diaphragm and the catheter while registering the geometric related characteristics of the diaphragm and the catheter, so as to avoid the occurrence of mismatching, improve the accuracy of registration and ensure the quality of registration. As shown in fig. 4, the angiographic image obtained after image registration has the capability of segmenting accurate blood vessels, and the location of the blood vessels can be identified based on the angiographic image. By fusing the fluoroscopic image with the angiographic image corresponding to the phase, the dynamic coronary artery road map as shown in fig. 5 is obtained, which is convenient for the doctor to visually distinguish the position of the blood vessel. The relative position between the diaphragm and the catheter may be a distance.

In the above embodiments, the angiographic image corresponding to the phase of the fluoroscopic image is matched in the sequence of angiographic images, so that the calculation load can be greatly reduced. This can be done, for example, in tens to a hundred milliseconds, which is advantageous for obtaining real-time coronary roadmapping, especially for intraoperative real-time applications.

In some embodiments, the geometrically relevant feature of the first marker comprises a fitted curve, curvature, position, shape of the first marker and the geometrically relevant feature of the second marker comprises a corner, position, orientation, curvature, shape, width of the second marker to achieve accurate registration with respect to the geometrically relevant feature.

In some embodiments, in the sequence of angiographic images of the preset time segment of the coronary artery, the vessel segment of interest is kept in a filling state to obtain an accurate sequence of angiographic images.

In some embodiments, the method for generating the dynamic coronary artery roadmap further includes performing the first image segmentation and the second image segmentation using a learning network to obtain a first segmented image and a second segmented image, respectively, for example, the learning method segments features of a catheter, a guide wire, a blood vessel, and a diaphragm, the predicted blood vessel map is used to make a coronary artery roadmap, and the predicted catheter and diaphragm are used for image registration in step S103. For example, the method further includes performing segmentation extraction on the medical image, and before obtaining the blood vessel image, the method further includes: obtaining training data, converting an image into a fixed size (such as 512 x 512), carrying out normalization processing, converting pixels into 0-1 pixels, wherein the training data comprises a medical image of segmentation marking information (blood vessels, guide wires, catheters and diaphragm), carrying out image processing methods such as image horizontal turning, vertical turning, random scaling, random brightness, random contrast, random noise and the like on the training data for data enhancement, and carrying out learning training on a segmentation network model by using the enhanced training data to obtain an image segmentation model.

The segmentation network model can be a ResUnet segmentation network model, a transunet segmentation network model and the like, network blood vessels and other features are extracted, the feature extraction efficiency is greatly improved, and the method is a fundamental guarantee for realizing real-time navigation. The segmentation network model is subjected to learning training by using the training data of the medical images of various segmentation labeling information (blood vessels, guide wires, catheters and diaphragm muscles), so that the image segmentation model can be obtained, and the accuracy and the speed of segmenting the segmentation target by using the obtained image segmentation model are ensured.

Wherein the second segmented image has the same size as the fluoroscopic image and the intensity value of each pixel indicates a probability-related parameter at which the second segmented image belongs to the segmented target, the first segmented image has the same size as the angiographic image and the intensity value of each pixel indicates a probability-related parameter at which the first segmented image belongs to the segmented target, and the segmented target includes at least a blood vessel, the first marker, and the second marker. In the method for generating a deep learning model according to an embodiment, segmenting and extracting a medical image to obtain a segmented image includes: segmenting the medical image by using an image segmentation model, and determining a probability value of each pixel on the medical image representing a segmentation target; and extracting the pixels with the probability values larger than the preset probability to obtain the segmentation target predicted image. Each type of segmented target can obtain a segmented target prediction image, namely the image segmentation model prediction image is a plurality of prediction images containing segmented targets (blood vessels, guide wires, catheters and diaphragm).

In the above embodiment, in the calculation process of segmenting by using the image segmentation model, actually, the image segmentation model predicts each pixel in the image to obtain a probability value that the pixel represents the segmentation target, and when the probability value of the pixel on the segmentation target is predicted to be greater than a preset probability, the pixel is judged to belong to the segmentation target. And predicting one by one, and finally obtaining all pixels representing the segmentation target to finish image segmentation so as to obtain a segmentation result.

Specifically, based on the first segmentation image, extracting a blood vessel, the first marker and the second marker in an angiography image; and extracting the blood vessel, the first marker and the second marker from the perspective image based on the second segmentation image for subsequent registration, thereby greatly improving the registration accuracy.

In some embodiments, based on the first segmentation image, confidence levels are determined for the extracted vessels for display in association with the segments of the segmented vessels. Optionally, in the method for generating a coronary artery map provided in the embodiment, determining the confidence of the blood vessel at each position on the blood vessel image includes: the confidence of the blood vessel at each location on the blood vessel image is determined based on probability values of pixels representing the segmentation target. The probability value of each pixel predicted by the image segmentation model is related to the confidence of the accuracy of the blood vessel segmentation. I.e. the confidence of the vessel everywhere on the vessel image can be determined based on the probability values of the pixels representing the vessel. Based on the confidence of each blood vessel, the pixel color of each blood vessel on the blood vessel image is determined, so that a coronary route map is generated, a doctor can judge the position of the blood vessel visually, and the working efficiency of the doctor is improved.

Fig. 6 illustrates a system for performing a method for generating a coronary roadmap in accordance with embodiments of the invention, in some embodiments the means 600 for generating a coronary roadmap may be a dedicated intelligent device or a general intelligent device. For example, the apparatus 600 for generating a coronary artery road map may be a computer customized for the task of generating the coronary artery road map, or a cloud-end server. For example, the generation apparatus 600 of the coronary artery road map may be integrated into the image processing apparatus.

As an example, in the generation apparatus 600 of the coronary artery road map, at least an interface 601 and a processor 603 are included, and in some embodiments, a memory 602 may also be included.

In some embodiments, the interface 601 is configured to receive dynamic fluoroscopic and/or angiographic images of the coronary arteries acquired by the imaging device, for example, the interface 601 may receive dynamic fluoroscopic and/or angiographic images of the coronary arteries acquired by various imaging devices via a communication cable, a Wireless Local Area Network (WLAN), a Wide Area Network (WAN), a wireless network, such as via radio waves, a cellular or telecommunications network, and/or a local or short range wireless network (e.g., bluetooth (TM)), or other communication methods.

In some embodiments, interface 601 may include an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem to provide a data communication connection. In such implementations, the interface 601 may send and receive electrical, electromagnetic and/or optical signals that carry analog/digital data streams representing various types of information via a direct communication link. In still other embodiments, interface 601 may further include a Local Area Network (LAN) card (e.g., an Ethernet adapter) to provide a data communication connection to a compatible LAN. As an example, the interface 601 may further comprise a network interface 6011, via which the apparatus 600 for generating a coronary artery road map may be connected to a network (not shown), for example including but not limited to a local area network or the internet in a hospital. The network may connect the coronary artery road map generating device 600 with an external device such as an image acquisition device (not shown), an image database 604, an image data storage 605. The image acquisition apparatus may be any apparatus for acquiring an image of an object, such as an MRI imaging device, a CT imaging device, a cardiac nuclide scan, an ultrasound device, or other medical imaging device for obtaining a sequence of dynamic fluoroscopic and/or angiographic images of a coronary artery of a patient.

In some embodiments, the means 600 for generating a coronary artery roadmap may additionally comprise at least one of an input/output 606 and an image display 607.

The processor 603 is configured to perform the method of generating a coronary artery road map according to various embodiments of the present invention, and is a processing device comprising one or more general purpose processing devices, such as a microprocessor, Central Processing Unit (CPU), Graphics Processing Unit (GPU), etc. More specifically, the processor 603 may be a Complex Instruction Set Computing (CISC) microprocessor, Reduced Instruction Set Computing (RISC) microprocessor, Very Long Instruction Word (VLIW) microprocessor, processor executing other instruction sets, or processors executing a combination of instruction sets. The processor 603 may also be one or more special-purpose processing devices such as an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a Digital Signal Processor (DSP), a system on a chip (SoC), or the like. As will be appreciated by those skilled in the art, in some embodiments, the processor 603 may be a special purpose processor rather than a general purpose processor. The processor 603 may include one or more known processing devices, such as the Pentium manufactured by Intel corporation ^TM、Core^TM、Xeon^TMOr Itanium^TMSeries of microprocessors, Turion manufactured by AMD^TM、Athlon ^TM、Sempron ^TM、Opteron ^TMFX ™ or Phenom ™ series microprocessors or any of various processors manufactured by Sun Microsystems. The processor 603 may also comprise a graphics processing unit, such as GeForce, Quadro, Tesla series of GPU from Nvidia, GMA, Iris made by Intel TM^TMGPU series or Raedeon manufactured by AMD^TMA series of GPUs. The processor 603 may also include an accelerated processing unit, such as the desktop A-4 (6, 8) series manufactured by AMD, Xeon Phi manufactured by Intel corporation^TMAnd (4) series. The disclosed embodiments are not limited to any type of processor or processor circuit that is otherwise configured to meet the following computational requirements: a method such as the generation of a coronary roadmap according to embodiments of the invention is performed. In addition, the term "processor" or "imageA processor "may include more than one processor, e.g., a multi-core design or multiple processors, each of which has a multi-core design. The processor 603 may execute sequences of computer program instructions stored in the memory 602 to perform the various operations, processes, methods disclosed herein.

The processor 603 may be communicatively coupled to the memory 602 and configured to execute computer-executable instructions stored therein. The memory 602 may include Read Only Memory (ROM), flash memory, Random Access Memory (RAM), Dynamic Random Access Memory (DRAM) such as synchronous DRAM (sdram) or Rambus DRAM, static memory (e.g., flash memory, static random access memory), etc., on which computer-executable instructions are stored in any format. The computer program instructions may be accessed by the processor 603, read from a ROM or any other suitable storage location, and loaded into RAM for execution by the processor 603. For example, the memory 602 may store one or more software applications. The software applications stored in memory 602 may include, for example, an operating system (not shown) and a soft control device (not shown) for a general purpose computer system. Further, the memory 602 may store the entire software application or only a portion of the software application to be executable by the processor 603. Additionally, memory 602 may store a plurality of software modules for performing the various steps described in connection with the various embodiments of the invention. Further, the memory 602 may store data generated/cached when executing a computer program, such as medical images sent from an image acquisition device, an image database 604, an image data storage 605, and the like.

In some embodiments, a learning network for automatic segmentation may be stored in memory 602. In other embodiments, the learning network for segmentation may be stored in a remote device, a separate database (such as image database 604), a distributed device.

The input/output 606 may be configured to allow the generation apparatus 600 of the coronary roadmap to receive and/or transmit data. Input/output 606 may include one or more digital and/or analog communication devices that allow the coronary roadmap generating apparatus 600 to communicate with a user or other machines and apparatuses. For example, input/output 606 may include a keyboard and mouse that allow a user to provide input.

Network interface 6011 may include a network adapter, a cable connector, a serial connector, a USB connector, a parallel connector, a high-speed data transmission adapter such as fiber optic, USB 3.0, lightning, a wireless network adapter such as a WiFi adapter, a telecommunications (3G, 4G/LTE, etc.) adapter. The generation apparatus 600 of the coronary artery road map may be connected to a network through a network interface 6011. The network may provide the functionality of a Local Area Network (LAN), a wireless network, a cloud computing environment (e.g., as software for a service, as a platform for a service, as infrastructure for a service, etc.), a client server, a Wide Area Network (WAN), etc.

In addition to angiographic images, dynamic coronary roadmapping, the image display 607 may also display other information such as location parameters, thickness of the vessels, etc. For example, the image display 607 may be an LCD, CRT, or LED display.

Various operations or functions are described herein that may be implemented as or defined as software code or instructions. Such content may be source code or differential code ("delta" or "patch" code) ("object" or "executable" form) that may be directly executed. The software code or instructions may be stored in a computer-readable storage medium and, when executed, may cause a machine to perform the functions or operations described, and includes any mechanism for storing information in a form accessible by a machine (e.g., a computing device, an electronic system, etc.), such as recordable or non-recordable media (e.g., Read Only Memory (ROM), Random Access Memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.).

The exemplary methods described herein may be implemented at least in part by a machine or computer. In some embodiments, a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform a method of generating a coronary roadmap according to various embodiments of the present invention. An implementation of such a method may include software code, e.g., microcode, assembly language code, a high-level language code, and the like. Various software programming techniques may be used to create various programs or program modules. For example, the program parts or program modules may be designed in or by Java, Python, C + +, assembly language, or any known programming language. One or more of such software portions or modules may be integrated into a computer system and/or computer-readable medium. Such software code may include computer readable instructions for performing various methods. The software code may form part of a computer program product or a computer program module. Further, in an example, the software code can be tangibly stored on one or more volatile, non-transitory, or non-volatile tangible computer-readable media, e.g., during execution or at other times. Examples of such tangible computer-readable media may include, but are not limited to, hard disks, removable magnetic disks, removable optical disks (e.g., compact disks and digital video disks), magnetic cassettes, memory cards or sticks, Random Access Memories (RAMs), Read Only Memories (ROMs), and the like.

The method and apparatus of the present invention are susceptible to various modifications and alternative forms. Other embodiments may be devised by those skilled in the art in view of the description and practice of the disclosed system and related methods. The individual claims of the invention can be understood as separate embodiments and any combination between them also serves as an embodiment of the invention and these embodiments are considered to be included in the invention.

It is intended that the specification and examples be considered as exemplary only, with a true scope being indicated by the following claims and their equivalents.

Claims

1. A method for generating a coronary artery road map, comprising:

acquiring a dynamic fluoroscopic image of the coronary artery;

acquiring a sequence of angiographic images of a coronary artery over a preset period of time during which the heart performs a periodic movement;

matching an angiographic image corresponding to the phase of the fluoroscopic image in the sequence of angiographic images, specifically comprising:

performing a first image segmentation on each angiographic image of the sequence of angiographic images to extract a first marker tracking respiratory motion and a second marker tracking heartbeat motion;

Performing second image segmentation on the perspective image to extract a corresponding first marker and a corresponding second marker;

matching an angiographic image corresponding to the phase of the fluoroscopic image in the sequence of angiographic images by comparing the first marker and the second marker extracted by the first image segmentation and the second image segmentation, respectively;

the fluoroscopic image is fused with the phase-matched angiographic image to generate a dynamic coronary artery road map during the operation, wherein segmented vessels of the coronary arteries in the phase-matched angiographic image are presented on the coronary arteries on the fluoroscopic image.

2. The generation method according to claim 1,

the dynamic fluoroscopic images of the coronary arteries comprise intraoperative real-time fluoroscopic images of the coronary arteries; and/or

The preset time period at least comprises one respiratory motion cycle or the least common multiple cycle of the respiratory motion cycle and the cardiac cycle.

3. The method for generating as defined in claim 1, wherein fusing the fluoroscopic image with the phase-coincident angiographic image to generate a dynamic coronary roadmap intraoperatively, further comprises:

On the coronary artery on the perspective image, the segmented vessels of the coronary artery in the phase-matched angiography image are presented, and the visual effect of each segmentation result is changed according to the confidence coefficient of each segmentation result.

4. The generation method according to any of claims 1 to 3, characterized in that matching out an angiographic image corresponding to the phase of the fluoroscopic image in the sequence of angiographic images specifically comprises:

registering the fluoroscopic image with the sequence of angiographic images by comparing geometrically relevant features of the first marker and the second marker and/or relative positions between the first marker and the second marker extracted by the first image segmentation and the second image segmentation, respectively, to match an angiographic image in line with the phase of the fluoroscopic image in the sequence of angiographic images.

5. A generating method according to claim 4, characterized in that the first marker comprises an anatomical landmark of the diaphragm and/or lungs and the second marker comprises a catheter and/or a cardiac silhouette fixed to the coronary ostia.

6. The generation method according to claim 4, characterized in that registering the sequence of fluoroscopic images and angiographic images by comparing geometrically relevant features of the first marker and the second marker and/or the relative position between the first marker and the second marker extracted by the first image segmentation and the second image segmentation, respectively, comprises in particular:

matching a first angiographic image corresponding to the phase of the fluoroscopic image in the sequence of angiographic images by first registering the fluoroscopic image with the sequence of angiographic images by comparing geometrically relevant features of the first marker extracted by the first image segmentation and the second image segmentation, respectively, together with the relative position between the first marker and the second marker;

second registering the fluoroscopic image with the sequence of angiographic images by comparing geometrically relevant features of the second marker extracted by the first and second image segmentations respectively together with the relative position between the first and second marker, thereby matching a second angiographic image in line with the phase of the fluoroscopic image in the sequence of angiographic images;

Determining the difference degree of the first angiography image and the second angiography image, and when the difference degree is lower than a threshold value, using the first angiography image and/or the second angiography image to obtain a final matched angiography image which is consistent with the phase of the perspective image;

or alternatively

Geometrically related features of the first and second markers extracted by the first and second image segmentations, respectively, are compared together with the relative positions between the first and second markers, and the fluoroscopic images are registered with the sequence of angiographic images, matching an angiographic image in line with the phase of the fluoroscopic image in the sequence of angiographic images.

7. A generation method according to claim 6, wherein the geometrically related feature of the first marker comprises at least one of a fitted curve, curvature, position and shape of the first marker, and the geometrically related feature of the second marker comprises at least one of a corner, position, orientation, curvature, shape and width of the second marker.

8. A generation method according to any one of claims 1-3, characterized in that in the sequence of angiographic images of a preset time segment of the coronary arteries, the vessel segment of interest remains in a filling state.

9. The generation method according to claim 4, further comprising:

performing the first image segmentation and the second image segmentation respectively by using a learning network to obtain a first segmentation image and a second segmentation image, wherein the second segmentation image has the same size as the perspective image and the intensity value of each pixel indicates a probability-related parameter of the position belonging to a segmentation target, the first segmentation image has the same size as each angiographic image and the intensity value of each pixel indicates a probability-related parameter of the position belonging to a segmentation target, and the segmentation target at least comprises a blood vessel, the first marker and the second marker;

extracting a blood vessel, the first marker and the second marker in an angiography image based on the first segmentation image;

based on the second segmentation image, a blood vessel, the first marker and the second marker in a fluoroscopic image are extracted.

10. The generation method according to claim 9, further comprising:

Based on the first segmentation image, confidences are determined for each of the extracted vessels for display in association with each of the segmented vessels.

11. An apparatus for generating a coronary artery road map, the apparatus comprising a processor configured to perform the method of generating a coronary artery road map according to any one of claims 1-10.

12. A computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform a method of generating a coronary roadmap according to any one of claims 1-10.