JP2021194268A6 - Blood vessel observation system and blood vessel observation method - Google Patents

Abstract

An object of the present invention is to improve convenience for users such as doctors by supporting smooth medical practices such as surgery and easily switching display screens for observing the state of an affected area.
A blood vessel observation system includes an endoscope whose distal end is inserted into a blood vessel of a subject, whose proximal end is fixed, and which images the blood vessel of the subject while being pulled back by a user operation; connected to an angiography device for imaging blood vessels and an endoscope and an endoscope, respectively, and simultaneously acquires and associates an angiography image captured by the angiography device and a blood vessel image captured by the endoscope and a controller for The controller causes a user to operate any one of a first display screen that displays an angiographic image, a second display screen that displays an angiographic image, and a third display screen that displays an angiographic image and an angiographic image. display on the monitor.
[Selection drawing] Fig. 6

Description

The present disclosure relates to a blood vessel observation system and a blood vessel observation method that display an image captured inside a blood vessel.

Patent Document 1 discloses an image analysis method that enables quick and easy determination of the optimal stent, allows easy confirmation of the adequacy of the determined stent before healing, and compares blood vessel images from different diagnosis dates with high accuracy. An apparatus is disclosed. This image analysis apparatus stores a short-axis cross-sectional image of a blood vessel output from an intravascular imaging apparatus, and generates a long-axis cross-sectional image of the blood vessel from a plurality of stored short-axis cross-sectional images. In addition, the image analysis device generates a lumen closed curve along at least the outer circumference of the lumen of the blood vessel from the saved short-axis cross-sectional image, and calculates the diameter of the stent to be inserted into the blood vessel from the generated lumen closed curve. Calculate

JP 2009-240359 A

However, in the configuration of Patent Document 1, the image analysis apparatus only displays either the short-axis cross-sectional image or the long-axis cross-sectional image of the blood vessel. For this reason, in the prior art, for example, during an operation, a display screen showing an intravascular image currently being imaged by an intravascular imaging apparatus, an image showing the current position of the intravascular imaging image, etc. and a display screen for comparing and displaying both of these images. A user such as a doctor often performs an operation on a patient in front of an operating table during an operation, and is often restricted in many movements. Therefore, for example, by enabling simple switching of the display screen by the operation of a doctor's assistant, for example, the doctor can appropriately observe and grasp the inside of the blood vessel simply by changing the direction of the line of sight without moving. It is expected that it will be possible to improve convenience such as

The present disclosure has been devised in view of the conventional situation described above, supports smooth medical practices such as surgery, and can easily switch display screens on which the state of the affected area can be observed, thereby improving the convenience of users such as doctors. An object of the present invention is to provide an improved blood vessel observation system and blood vessel observation method.

The present disclosure images a blood vessel of a subject while a distal end is inserted into a blood vessel of a subject and a proximal end is fixed, and the distal end is pulled back toward the proximal end by user manipulation. Connected to an endoscope, an angiography device for imaging the blood vessel of the subject and the endoscope, and the endoscope, respectively, an angiography image captured by the angiography device and an angiography image captured by the endoscope a controller for simultaneously acquiring and associating the captured blood vessel images, wherein the controller comprises a first display screen displaying the angiographic image, a second display screen displaying the blood vessel image, and the Provided is a blood vessel observation system that switches and displays any one of an angiographic image and a third display screen that displays the blood vessel captured image on a monitor according to a user's operation.

Further, the present disclosure is a blood vessel observation method performed by a blood vessel observation system, wherein a distal end is inserted into a blood vessel of a subject, a proximal end is fixed, and the distal end is moved to the proximal end by a user operation. an angiography apparatus for imaging the blood vessel of the subject and the endoscope, and an angiography apparatus for imaging the blood vessel of the subject and the endoscope while being pulled back toward the blood vessel; An angiographic image captured by an imaging device and a blood vessel captured image captured by the endoscope are simultaneously acquired and associated, and a first display screen for displaying the angiographic image and a second display screen for displaying the blood vessel captured image are provided. 2 display screens and a third display screen displaying the angiographic image and the blood vessel captured image are switched according to a user's operation and displayed on a monitor.

Advantageous Effects of Invention According to the present disclosure, it is possible to support smooth medical practices such as surgery, easily switch display screens on which the state of an affected part can be observed, and improve convenience for users such as doctors.

1 is a diagram showing a configuration example of a blood vessel observation system according to Embodiment 1; FIG. Flowchart showing an example of the operation procedure of the blood vessel observation system according to the first embodiment Flowchart showing an example of an operation procedure for calculating a blood vessel diameter in step S14 of FIG. A diagram showing the arrangement of feature points on a blood vessel image, the positional relationship between the feature points and corresponding points, and the vessel diameter estimated using the three-dimensional points. A diagram showing an example of calculating a three-dimensional position A diagram showing an example of display switching among an angiographic image display screen, an angioscopic image display screen, and a display screen for comparatively displaying the angiographic image and the angioscopic image. A diagram showing an angiographic image and an angioscopic image at a first point designated by user operation. A diagram showing an angiographic image and an angioscopic image at a second point specified by user operation.

Hereinafter, embodiments specifically disclosing a blood vessel observation system and a blood vessel observation method according to the present disclosure will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially the same configurations may be omitted. This is to avoid unnecessary verbosity in the following description and to facilitate understanding by those skilled in the art. It should be noted that the accompanying drawings and the following description are provided for a thorough understanding of the present disclosure by those skilled in the art and are not intended to limit the claimed subject matter.

FIG. 1 is a diagram showing a configuration example of a blood vessel observation system 100 according to Embodiment 1. As shown in FIG. The blood vessel observation system 100 captures an image of blood vessels in a subject such as a human body with an angioscope 10 at the time of surgery or examination (hereinafter referred to as "surgery or the like"). The blood vessel diameter is measured using a plurality of images captured by and the results are displayed. The blood vessel may be, for example, a coronary artery, an upper limb, or a lower limb, and the site may not be particularly limited. A blood vessel observation system 100 includes an angioscope 10 and an image console CSL1. In addition, blood vessel observation system 100 may be configured to further include doctor-facing monitor 80 and angiography device 90 .

1. Configuration of Blood Vessel Observation System The angioscope 10 is an example of an endoscope that constitutes the blood vessel observation system 100, and is inserted along a catheter (not shown) previously inserted into a subject during surgery or the like. It is a medical device that can be pulled back and forth. The angioscope 10 is sometimes referred to as a so-called angioscope catheter. The angioscope 10 has a rigid distal end TP1 mounted with an imaging unit (not shown) that can be inserted into the subject to image the interior of the subject, and a proximal end that is held by a doctor's assistant or the like. BE1. The outer periphery of the angioscope 10 is covered with a flexible outer sheath OS1 from the proximal end BE1 to the rigid portion of the distal end TP1. The rigid portion of the tip portion TP1 is configured as a flange portion made of metal or the like. The outer diameter of the angioscope 10 corresponds to the diameter of the outer sheath OS1, which has a circular outer shape in the direction perpendicular to the optical axis of the imaging unit (not shown) of the angioscope 10. For example, the maximum outer diameter is Although it is 1.8 mmΦ, it is not limited to this size.

The angioscope 10 is operated by a user such as a doctor (hereinafter referred to as "user") along a guide wire GW1 that has been inserted through an observation site (for example, a blood vessel) in the subject prior to surgery or the like. It is inserted through the inside of the blood vessel in the subject so that it can move back and forth. A doctor's assistant (for example, a person who pulls back the angioscope 10) may also be called a user. Here, the direction in which the angioscope 10 is inserted toward the observation site within the subject is defined as the advancing direction, and the direction in which the angioscope 10 is retracted toward the outside of the subject is defined as the withdrawal direction. Define. Therefore, being able to move back and forth means that the angioscope 10 can be inserted into the subject and pulled back. The angioscope 10 can be smoothly inserted to an observation site (for example, an affected area) by being guided by a guide wire GW1 that has been inserted in advance to an observation site (for example, an affected area) for surgery or the like. The angioscope 10 may be one in which an imaging unit (not shown) is replaceably attached to the distal end of a normal catheter (not shown). A catheter is a medical tube used, for example, for the drainage of bodily fluids or the injection of medications. In addition to the angioscope 10, the catheter may be exchangeably attached with a balloon, a stent, or the like.

The angioscope 10 is, for example, a 480,000-pixel high-resolution camera in which an image sensor capable of imaging blood vessels is mounted on the distal end portion TP1. Note that 480,000 pixels is just an example, and the number of pixels need not be limited to 480,000 pixels. When the angioscope 10 is inserted into a blood vessel in the subject, the angioscope 10 can image the inner wall of the blood vessel (blood vessel wall). The angioscope 10 incorporates, as an image sensor, a solid-state imaging device (that is, an image sensor) such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor), and receives optical signals from a subject (for example, a blood vessel wall). An image is formed on an imaging surface, the formed optical image is converted into an electrical signal, and a data signal of the image is output. The angioscope 10 is pulled back in the retraction direction by a user's sensory operation, and outputs a data signal of an image of the vascular wall. Hereinafter, an image of the blood vessel wall captured by the angioscope 10 will be referred to as an "angioscope image" (an example of a blood vessel captured image). The data signal of the angioscope image may be either a still image signal or a moving image signal, and is input to the CCU 30 of the image console CSL1. Further, the angioscope 10 may incorporate an LED (Light Emission Diode) light source or an optical fiber for guiding the irradiation light from the CCU 30 in order to brightly illuminate the affected area.

The image console CSL1 is composed of a CCU 30 (Camera Control Unit), a PC 50 (Personal Computer), and a monitor 70, for example.

The CCU 30 is electrically connected to the angioscope 10 and controls the imaging operation by the angioscope 10 and the data generation of the angioscope image based on the data signal of the image from the angioscope 10 . The CCU 30 adds metadata to data of the angioscope image and sends the data to the PC 50 . The metadata includes data such as the imaging date and time of the angioscope image provided from the angioscope 10 .

The CCU 30 includes at least an image input section (not shown), an image processing section (not shown), and an image output section (not shown). An image input unit (not shown) inputs data of angioscopy images. The image input unit (not shown) uses a dedicated image input interface, as well as HDMI (registered trademark) (High-Definition Multimedia Interface) or USB (Universal Serial Bus) Type-C capable of transferring video data at high speed. It can be an existing interface. An image processing unit (not shown) performs processing such as adding metadata to the input data of an angioscopy image. Further, the image processing unit (not shown) performs predetermined image processing on the data of the angioscopy image sent from the image input unit (not shown) so that the image is visually recognized on the monitor 70 or the doctor's face-to-face monitor 80 . Generates combined data in possible RGB or YUV format. In the following description, angioscopy image data is assumed to be angioscopy image data that has been subjected to predetermined image processing by the CCU 30 . An image output unit (not shown) transmits data of an angioscopy image to which metadata is added to the PC 50 .

The PC 50 is an example of a controller that configures the blood vessel observation system 100, and includes an image capture board 51, an input interface 52, a memory 53, a processor 54, an operation unit 55, a storage 56, and an output interface 57. Configuration. The PC 50 receives data of the angioscope image to which metadata is added by the CCU 30 . The PC 50 records and saves the data of the angioscope image in the storage 56 . In addition, the PC 50 is connected for data communication with an angiography device 90 serving as an angiography device, and receives images captured by the angiography device 90 (hereinafter referred to as “angiographic images”). and save it in the storage 56. The PC 50 acquires the data of the angioscope image from the CCU 30 and the angiographic image from the angiography device 90 at the same time (that is, acquired at the same imaging timing) and associates them. For this association, for example, the PC 50 assigns a common identification number to both the angioscopic image data from the CCU 30 and the angiographic image from the angiography device 90 so that these images can be captured simultaneously. A process is applied to make it possible to identify the fact. It should be noted that the identification number may be information indicating the imaging date and time.

The PC 50 also derives the position information and speed information of the angioscope 10 during pullback based on the data of the angioscope image from the CCU 30 and the angiographic image from the angiography device 90 . For example, a tip TP1 of the angioscope 10 is provided with a marker MK1 (see below), and the angiography device 90 can capture an angiographic image in which the marker MK1 is reflected. Therefore, the PC 50 calculates (derives) the position information and speed information of the angioscope 10 based on the position of the marker MK1 in each angio image captured at a predetermined frame rate (fps: frame per second). can. The position information of the angioscope 10 indicates the position where the angioscope 10 is imaging at that time, in other words, the imaging position. The PC 50 associates the position information and speed information of the angioscope 10 with the data of the angioscopic image (see above), and uses the data of a plurality of associated angioscopic images to calculate the vessel diameter. Perform processing to calculate The PC 50 displays a display screen WD1 (see FIG. 6) showing only an angiographic image, a display screen WD2 (see FIG. 6) showing only an angioscopic image, or an angiographic image and an endoscopic image. Data for the display screen WD3 (see FIG. 6) on which both mirror images are displayed is generated, one of which is selected and switched according to the user's operation using the operation unit 55, and output to the monitor 70 and the doctor's face-to-face monitor 80 ( display), and a process of visualizing the inside of the blood vessel is performed using a display screen showing the observation status of the blood vessel.

In addition, the PC 50 generates a constant-speed moving image composed of a plurality of angioscopic images using the data of the associated angioscopic images. The constant-velocity moving image referred to here is, for example, a plurality of angioscopy images, using the position information and the speed information of the angioscopy 10, and capturing images at different imaging positions at substantially constant time intervals. It is a moving image configured by appropriately selecting a plurality of blood vessel images in time series by the processor 54 . The processor 54 uses this constant-speed moving image to calculate the blood vessel diameter (see FIGS. 3 to 5).

The image capture board 51 receives and temporarily stores the angiographic image data sent from the angiography device 90 and the angiographic image data sent from the CCU 30 , and also receives the data via the input interface 52 . to processor 54.

The input interface 52 may be an interface using HDMI (registered trademark), USB Type-C, or the like capable of high-speed transfer of image data (see above) from the image capture board 51 .

The memory 53 includes a RAM (Random Access Memory) used as a working memory of the processor 54 and a ROM (Read Only Memory) storing programs for various processes executed by the processor 54 .

By executing various processing programs stored in the memory 53, the processor 54 performs, for example, data association processing of angioscopy images, constant-speed moving image generation processing, blood vessel diameter calculation processing, and display screen data processing. Execute each of the generation process and the like. The processor 54 may be, for example, a GPU suitable for image processing, a dedicated electronic circuit designed with an MPU, a CPU, an ASIC (Application Specific Integrated Circuit) or the like, or an electronic circuit designed to be reconfigurable with an FPGA or the like. may be configured. Details of a processing example of the processor 54 will be described later.

The operation unit 55 includes an activation switch for activating the blood vessel observation system 100, and receives operations by the user. The operation unit 55 may include, for example, a mouse, keyboard, touch pad, touch panel, microphone, or other input device in addition to the activation switch described above.

The storage 56 is a large-capacity storage device that accumulates angioscopic images captured by the angioscope 10, data of angiographic images captured by the angiography device 90, and the like. The storage 56 may include, for example, a secondary storage device (eg, HDD (Hard Disk Drive) or SSD (Solid State Drive)) or a tertiary storage device (eg, optical disc, SD card).

The monitor 70 displays data on any one display screen (see FIG. 6) output from the PC 50 . The monitor 70 has a display device such as an LCD (Liquid Crystal Display), an organic EL (Electroluminescence), a CRT (Cathode Ray Tube), or the like. CCU 30, PC 50 and monitor 70 are mounted in a single housing as image console CSL1 and arranged in the operating room.

The doctor-facing monitor 80 is a large display monitor arranged on the opposite side of the operating table on which the subject is lying so as to face the doctor. The doctor-facing monitor 80 displays data on the display screen (see FIG. 6) generated by the PC 50 .

The angiography apparatus 90 is an example of an angiography apparatus that constitutes the blood vessel observation system 100. The angiography apparatus 90 is an angiography apparatus, and is used to determine the shape of blood vessels of a subject (patient) lying on an operating table during surgery or the like, the distribution of abnormalities, etc., and the size of tumors. It is a device that examines and treats the condition of blood vessels or blood flow in the body. Specifically, a contrast medium is injected into blood vessels in the subject through a catheter (not shown), and the angiography apparatus 90 performs examination and treatment of blood flow or tumor distribution, and vascular stenosis or occlusion. The angiography device 90 generates an angiographic image by, for example, imaging the shape of the blood vessel of the subject, and transmits data of the angiographic image to the PC 50 . Although the details will be described later, the angioscope 10 is provided with a marker MK1 at the distal end portion TP1 as a mark at the time of imaging by the angiography device 90 . Therefore, the marker MK1 of the angioscope 10 appears in the angio image so that the user can identify the position of the angioscope 10 .

2. Operation of Blood Vessel Observation System Next, an operation procedure of the blood vessel observation system 100 according to Embodiment 1 will be described with reference to FIG. FIG. 2 is a flow chart showing an example of the operation procedure of the blood vessel observation system 100 according to the first embodiment. FIG. 2 shows the operations of the angioscope 10, the angiography device 90, the image console CSL1, and the doctor's face-to-face monitor 80, which constitute the blood vessel observation system 100, in chronological order. Note that the processor 54 is the subject of processing of the image console CSL1.

For example, the angioscope 10 is inserted into a blood vessel to observe the condition of the blood vessel within the subject, such as the presence of a thrombus in the subject's blood vessel or the formation of plaque on the blood vessel wall. When the user inserts the angioscope 10 into the blood vessel, the guide wire GW1 and the catheter (not shown) are inserted into the blood vessel in this order. When the guide wire GW1 and the catheter (not shown) are delivered to the blood vessel to be observed, the user advances and inserts the angioscope 10 into the blood vessel so as to be guided by the guide wire GW1 and along the inside of the catheter. To go. When the distal end TP1 of the angioscope 10 reaches the site to be observed (for example, a diseased part such as a blood vessel), the user starts to manually pull back the angioscope 10 within the catheter. In order that the angioscope 10 can clearly image the inside of the blood vessel, a transparent liquid such as a contrast medium or physiological saline is injected into the blood vessel from the proximal end BE1 of the angioscope 10 (not shown). injected into

In FIG. 2, when the user presses the activation switch included in the operation unit 55 of the PC 50 housed in the housing of the image console CSL1, the blood vessel observation system 100 is activated (S1). When the blood vessel observation system 100 is activated, the angioscope 10, the angiography device 90, the image console CSL1, and the doctor-facing monitor 80 each start operating.

The user determines the start position and end position of measurement for pulling back the angioscope 10 that has been inserted into the observation site within the subject. At this time, the user actually visually confirms the angioscopy image captured by the angioscopy 10 on the monitor 70 on which the angioscopy image is displayed, and determines the measurement start position. The length from the start position to the end position of the withdrawal of the angioscope 10 determined by the user corresponds to the measured length of the blood vessel desired by the user.

The angioscope 10 images the inside of the blood vessel and generates a data signal of an angioscope image (S3) while being pulled back by the user's operation (S2). Each time the angioscope 10 generates a data signal of a captured angioscope image (frame), the angioscope 10 transmits the frame to the CCU 30 (S4).

The angiography device 90 generates an angiographic image data signal by imaging the shape of the blood vessel of the subject into which the angioscope 10 is inserted (S5), and transmits the angiographic image data signal to the image console CSL1 (for example, PC 50) (S6).

In the image console CSL1, the PC 50 inputs and receives the angiographic image data (see step S6) transmitted from the angiography device 90 (S7), and receives the angiographic data transmitted from the angioscope 10. Image data (see step S4) is input and received (S8). The processes of steps S7 and S8 are executed simultaneously. In other words, the PC 50 simultaneously acquires the data of the angioscope image from the CCU 30 and the angiographic image from the angiography device 90 (that is, acquired at the same imaging timing) and associates them. Based on the position of the marker MK1 (see FIG. 6) of the angioendoscope 10 reflected in the data signal (frame) of each angiographic image acquired in step S7, the PC 50 determines whether the angioendoscope 10 is being pulled back. Position information and speed information of the angioscope 10 corresponding to the data signal of the angioscope image are derived (S9).

The PC 50 uses the position information and the speed information of the angioscope 10, which correspond to the data signal of the angioscope image of the angioscope 10 being pulled back, obtained in step S9, to operate the angioscope. 10 position and velocity information are associated with angioscopic image data (see above). The PC 50 uses data of a plurality of associated angioscopic images to generate a constant-velocity moving image composed of a plurality of angioscopic images, and a plurality of intravascular images forming the constant-velocity moving image. Using the endoscopic image data, the blood vessel diameter of the observation site (for example, blood vessel) is calculated (S10). The details of the process of step S10 will be described later. The PC 50 also displays a display screen WD1 (see FIG. 6) that mainly displays angiographic images, a display screen WD2 (see FIG. 6) that mainly displays angiographic images, angiographic images, and Each of the display screens WD3 (see FIG. 6) on which both of the angioscopy images are mainly displayed is generated and stored in the storage 56 (S10).

The PC 50 also displays a display screen WD1 (see FIG. 6) that mainly displays angiographic images, a display screen WD2 (see FIG. 6) that mainly displays angiographic images, angiographic images, and For example, one of the display screens WD3 (see FIG. 6) on which both angioscopy images are mainly displayed is selected (switched) according to the user's operation using the operation unit 55, read out from the storage 56, and monitored. 70 (S11). Further, the PC 50 displays a display screen WD1 (see FIG. 6) mainly showing an angiographic image and an angioscopic image mainly containing the calculation result of the blood vessel diameter so as to synchronize with the display of the monitor 70 in step S11. Either the display screen WD2 (see FIG. 6) on which an angiographic image or an angioendoscopic image is mainly displayed (see FIG. 6) is displayed, for example, in response to an operation using the operation unit 55 by the user. is selected (switched), read out from the storage 56, and output to the doctor-facing monitor 80 for display (S12). The type of display switching for each of the display screens WD1 to WD3 is not particularly limited. For example, when the display screen WD1 is displayed, the user performs one operation using the operation unit 55 (for example, pressing a switch or button). , the display screen may be switched to WD2, and if the same operation is performed twice, the display screen may be switched to WD3. Similarly, when the display screen WD2 (display screen WD3) is displayed, if the user operates the operation unit 55 once, the screen is switched to the display screen WD3 (display screen WD1). It may be switched to the display screen WD1 (display screen WD2).

Here, an outline of the operation of calculating the blood vessel diameter in step S10 of FIG. 2 will be described with reference to FIGS. 3, 4 and 5. FIG. FIG. 3 is a flow chart showing an example of the operation procedure for calculating the blood vessel diameter in step S10 of FIG. FIG. 4 is a diagram showing the arrangement of feature points on a blood vessel image, the positional relationship between the feature points and corresponding points, and the blood vessel diameter estimated using the three-dimensional points. FIG. 5 is a diagram showing an example of calculation of a three-dimensional position. The process of FIG. 3 is, for example, a process of calculating a blood vessel diameter using data of a plurality of angioscopy images forming the constant-speed moving image created in step S10 of FIG.

The user only needs to input the center and radius of the circle to the PC 50 . The processor 54 receives the center and radius of the circle input by the user via the operation unit 55 . The processor 54 selects a circle based on the center and radius of a circle from among a plurality of (for example, 1156) measurement points arranged on the entire angioscopy image for the angioscopy image data input from the CCU 30. A plurality (for example, 128) of feature points e1 are detected on the circumference (S51). Note that the number of 1156 measurement points and 128 feature points is an example.

In FIG. 4, 128 feature points e1 are superimposed and drawn on the circumference of a circle specified by the user on an angioscope image GZ1 captured by the angioscope 10 . The processor 54 determines whether or not the angioscope image GZ1 in which the feature point e1 is detected is the first frame image (S52). If it is the first frame image (S52, YES), the processor 54 ends the processing shown in FIG. 3 and returns to the original processing without calculating the blood vessel diameter. This is because the processor 54 needs at least two frames of the angioscope image GZ1 to calculate the blood vessel diameter.

On the other hand, if the image is the second frame or later in step S52 (S52, NO), the processor 54 performs first feature point matching (S53). In the first feature point matching, the processor 54 acquires a rectangular area near feature points in the previous frame (in other words, the (n-1)th frame) as a template (n: an integer equal to or greater than 2). As an example, the template size is 16 pixels wide by 16 pixels high.

The processor 54 searches for a rectangular region centered on the feature point position of the current frame (in other words, the n-th frame) as a search range for a region that matches the template. The size of the search range is 128 pixels wide by 128 pixels high. The processor 54 sets the position where the ZNCC (Zero-means Normalized Cross Correction) value is minimum as the corresponding point f2 corresponding to the feature point e1. Note that the n-th frame is an image in which the angioscope 10 is pulled forward with respect to the (n−1)-th frame, so the circle formed by the plurality of corresponding points f2 corresponds to the plurality of It is smaller than the circle formed by the feature point e1.

The processor 54 estimates the vanishing point dp using the feature point e1 included in the (n-1)th frame and the corresponding point f2 included in the nth frame (S54). In estimating the vanishing point dp, the processor 54 finds the flow from the feature point e1 to the corresponding point f2 for all feature points e1 and corresponding points f2. Processor 54 determines the intersection of the two flows. Processor 54 finds vectors from all feature points e1 to the intersection. The processor 54 obtains the degree of similarity (here, angular difference) between this flow and each vector, and determines whether this degree of similarity exceeds a threshold value, for example, whether the angular difference between the flow and the vector is less than 3°. discriminate. The processor 54 determines that the feature point e1 is a valid feature point (hereinafter sometimes referred to as an "inlier") if the angle difference is less than 3°. On the other hand, the processor 54 determines that the feature point e1 is an invalid feature point (hereinafter sometimes referred to as an "outlier") when the angle difference is 3° or more. Processor 54 computes the number of inliers for every flow intersection. The processor 54 sets the intersection point of the flow with the largest number of inliers as the vanishing point dp.

Processor 54 performs a second feature point matching for cross-checking (S55). In the second feature point matching, the processor 54 acquires a rectangular area near the corresponding points in the current frame (that is, the n-th frame) as a template. As an example, the template size is 16 pixels wide by 16 pixels high. The processor 54 searches for a rectangular region centered on the position of the corresponding point in the previous frame (that is, the (n−1)th frame) as a search range for a region that matches the template. The size of the search range is 128 pixels wide by 128 pixels high. The processor 54 sets the position where the ZNCC value is minimum as the feature point (corresponding feature point) corresponding to the corresponding point f2. The processor 54 determines whether or not the feature point e1 and the corresponding feature point in the previous frame (that is, the (n-1)th frame) substantially match. Substantial matching between the feature point e1 and the corresponding feature point can be determined, for example, based on the positional coordinates. The processor 54 determines that the corresponding point f2 is reliable when the feature point e1 substantially matches the corresponding feature point, and determines that the corresponding point f2 is unreliable when the feature point e1 and the corresponding feature point do not substantially match. I judge. The processor 54 adopts the feature points e1 determined to be reliable as inliers, and does not adopt the feature points e1 determined to be unreliable as outliers.

The processor 54 calculates the three-dimensional position of the inlier feature point e1 (S56). Triangulation, for example, is used to calculate the three-dimensional position of the feature point e1. Triangulation is a well-known surveying method using trigonometry and geometry that determines the position of a feature point by measuring the distance between two points and the angle from these two points to the feature point to be measured.

In FIG. 5, let the three-dimensional coordinates E of the feature point e1 be (X, Y, Z). X is the coordinate value of the x-axis representing the diameter (minor axis) direction of the blood vessel. Y is the coordinate value of the y-axis representing the radial direction of the blood vessel perpendicular to the x-axis. Z is the coordinate value of the z-axis representing the longitudinal (major axis) direction of the blood vessel. Assume that the image coordinates p1 of the feature point e1 at the camera position g1 of the (n−1)th frame are (u1, u2). Assume that the image coordinates p2 of the corresponding point f2 at the camera position g2 of the n-th frame are (u2, v2). Let D be the distance between the camera position g1 of the (n-1)th frame and the camera position g2 of the nth frame. Here, when the user pulls back the angioscope 10, the distance D is calculated as the product of the imaging time difference between the camera position of the (n-1)th frame and the camera position of the nth frame and the pullback speed. be.

A matrix K of intrinsic parameters of the camera is represented by Equation (1).

Here, fx is the value obtained by dividing the focal length by the horizontal pixel pitch, fy is the value obtained by dividing the focal length by the vertical pixel pitch, Cx is the x-coordinate of the center of the image, and Cy is the y-coordinate of the center of the image.

The image coordinate p1 and the three-dimensional coordinate E of the feature point e1 are represented by Equation (2).

The image coordinate p2 and the three-dimensional coordinate E of the corresponding point f2 are represented by Equation (3).

Processor 54 uses, for example, a triangulation function to determine the three-dimensional coordinates E of feature point e1, corresponding to image coordinates p1 and p2. The processor 54 can display the three-dimensional coordinates E of the feature point e1 as a three-dimensional stereoscopic image on the monitor 70 or the doctor's face-to-face monitor 80 (3D display).

The processor 54 detects a plane on which ellipse fitting is performed based on the three-dimensional coordinates E of the plurality of feature points e1 (S57). In detecting a plane, the processor 54 selects three points from among a plurality of feature points e1 (referred to as three-dimensional points) and obtains a formula representing a plane containing these three points (referred to as a plane formula). The processor 54 counts the number of three-dimensional points within a predetermined distance from the obtained plane and close to the plane. Processor 54 selects the plane formula that produces the largest number of three-dimensional points counted. Processor 54 extracts 3D points near the plane represented by the selected plane formula. The processor 54 performs principal component analysis (PCA) based on the extracted three-dimensional point group to detect a plane. Principal component analysis is a method of multivariate analysis that synthesizes a small number of uncorrelated variables called principal components that best represent the overall variation from a large number of correlated variables.

The processor 54 adopts feature points e1 that are more than a predetermined distance from the plane as outliers and feature points e1 that are within the predetermined distance as inliers for estimating the vessel diameter.

The processor 54 performs ellipse fitting based on the plurality of feature points e1 to estimate the blood vessel diameter (S58). In ellipse fitting, a plurality of feature points e1 are projected onto the xy plane, and fitting using RANSAC (Random Sample Consensus) is performed on the two-dimensional points projected onto the xy plane. RANSAC is a technique for estimating elliptic parameters so as not to include outliers.

By using RANSAC, the processor 54 can estimate an ellipse by suppressing the influence thereof even if outliers are included in a plurality of two-dimensional points. Therefore, the estimation accuracy of the ellipse is improved. Specifically, the processor 54 inputs a plurality of two-dimensional points and randomly extracts five points from the plurality of input two-dimensional points. Processor 54 determines ellipse parameters using the five extracted points. Ellipse parameters include major and minor axes. The processor 54 calculates the shortest distance from each two-dimensional point to the elliptical arc, counts the number of two-dimensional points (the number of inliers) whose distance is smaller than the threshold, and records it in the memory 53 . The processor 54 extracts another five points from among the plurality of input two-dimensional points, and counts the number of inliers in the same procedure as above. Processor 54 updates the number of inliers recorded in memory 53 when the counted number of inliers exceeds the number of inliers recorded in memory 53 . The processor 54 repeats the same procedure, and when the number of inliers recorded in the memory 53 is not continuously updated a certain number of times, that is, when the maximum number of inliers is obtained, the extraction of two-dimensional points is terminated. . For all two-dimensional points, the processor 54 uses the ellipse parameters that maximize the number of inliers, and determines the two-dimensional points where the shortest distance from each two-dimensional point to the elliptical arc is less than a threshold as inliers. Processor 54 uses all determined inliers to determine ellipse parameters. As a result of finding the ellipse parameters using all the inliers, the processor 54 estimates the major axis, which is one of the ellipse parameters, as the blood vessel diameter φ1. The processor 54 can appropriately determine the size of the stent that can be inserted through the blood vessel by using the diameter of the blood vessel as the major axis.

Here, the processor 54 acquires the corresponding points corresponding to the feature points by template matching. and using a learned model generated as a result of the machine learning, the corresponding points of the n-th frame corresponding to the feature points of the (n−1)-th frame may be obtained.

Also, the processor 54 estimated the major axis of the ellipse parameter as the vessel diameter, but may estimate the minor axis as the vessel diameter. The processor 54 may also estimate the vessel diameter using the major and minor diameters, for example, by adding the major and minor diameters and using half the value.

Also, when the angioscope 10 is pulled back by the user's operation, the PC 50 (for example, the processor 54) controls the position information of the angioscope 10 corresponding to the position information of the angioscope 10 captured by the angioscope 10. The images are acquired sequentially and may be recorded in the storage 56 as blood vessel short-axis cross-sectional images. Here, the short-axis cross-section refers to a cross-section in the diameter direction of the blood vessel (that is, the short-axis direction, which is the diameter direction) when the blood vessel is assumed to have a substantially cylindrical shape. Also, the direction perpendicular to the minor axis direction is defined as the major axis direction. In addition, the PC 50 (for example, the processor 54) specifies the position information of the angioscope 10 based on the angiographic image data signal input from the angiography device 90 in synchronization with the data signal of the angiographic image. Furthermore, the position information of the angioscope 10 may be used to calculate the measured length TAR1 (see FIG. 6) of the blood vessel. The PC 50 generates a three-dimensional image of the blood vessel based on the plurality of short-axis cross-sectional images of the blood vessel recorded in the storage 56, and cuts the three-dimensional image of the blood vessel along the long axis direction of the blood vessel to obtain the length of the blood vessel. An axial cross-sectional image LGT1 (see FIG. 6) is generated. A long-axis cross-section indicates a cross-section in the longitudinal direction (longitudinal direction) of a blood vessel when the blood vessel is similarly assumed to have a substantially cylindrical shape. The PC 50 displays on the monitor 70 display screens WD1 (see FIG. 6), WD2 (see FIG. 6), and WD3 (see FIG. 6) including the calculation result of the blood vessel diameter, the short-axis cross-sectional image of the blood vessel, and the long-axis cross-sectional image of the blood vessel. You can

FIG. 6 is a diagram showing an example of display switching of the display screen WD1 of the angiographic image ANG1, the display screen WD2 of the angioscopic image VED1, and the display screen WD3 for comparatively displaying the angiographic image ANG1 and the angioscopic image VED1. . First, each of the display screens WD1 to WD3 will be explained.

The display screen WD1 has a display area for an angio image ANG1. The angio image ANG1 is generated by the processor 54 based on the angio image data signal from the angiography device 90 input to the PC 50 . The angiographic image ANG1 shows the angioscope 10 whose outer periphery is covered with the outer sheath OS1 and the marker MK1 provided at the distal end TP1. The angio image ANG1 also shows a measurement range VSL1 corresponding to the measurement length TAR1 indicating the range from the start position to the end position of the measurement for pulling back the angioscope 10 determined by a user such as a doctor. . Note that the measurement range VSL1 may be displayed in a color that allows the measurement range VSL1 to be identified on the angio image ANG1 by user operation.

In addition, the display screen WD1 displays a display area of a blood vessel long-axis cross-sectional image LGT1 having a length equal to or longer than the measurement length TAR1 corresponding to the measurement range VSL1, a display area of a button area BT1 for accepting user operations, and the measurement result. and a display area of the measurement result display area MRT1. It shows that the blood vessel diameter was calculated to be 5.3 mm at the measurement start position of the measurement length TAR1, and that the blood vessel diameter was calculated to be 6.1 mm at the measurement end position of the measurement length TAR1. there is A designation bar BAR1 indicating the position of the angioscope 10 indicated by the angiographic image ANG1 is superimposed and displayed on the blood vessel long-axis cross-sectional image LGT1 by user operation. By appropriately sliding the designated bar BAR1 in the horizontal direction within the measurement length TAR1 by the user's operation, the processor 54 reads the angio image ANG1 in which the marker MK1 appears at the position indicated by the designated bar BAR1 from the storage 56, and displays the display screen WD1. Generate. That is, the user viewing the display screen WD1 can see the inside of the blood vessel in the angio image ANG1 and the position of the angioscope 10 (more specifically, depending on the position of the designation bar BAR1 (see FIGS. 7 and 8). is the position of the marker MK1).

Next, the display screen WD2 has a display area for the angioscope image VED1. The angioscope image VED1 is generated by the processor 54 based on the data signal of the angioscope image from the angioscope 10 input to the PC 50 .

Further, the display screen WD2 displays a display area of a blood vessel long-axis cross-sectional image LGT1 having a length equal to or longer than the measurement length TAR1 corresponding to the measurement range VSL1, a display area of a button area BT2 for receiving user operations, and the measurement results. and a display area of the measurement result display area MRT2. It shows that the blood vessel diameter was calculated to be 5.3 mm at the measurement start position of the measurement length TAR1, and that the blood vessel diameter was calculated to be 6.1 mm at the measurement end position of the measurement length TAR1. there is A designation bar BAR1 indicating the position of the angioscope 10 is superimposed and displayed on the blood vessel long-axis cross-sectional image LGT1 by user operation. By appropriately sliding the designation bar BAR1 in the horizontal direction within the measurement length TAR1 by the user's operation, the processor 54 reads out the angioscopy image VED1 corresponding to the position indicated by the designation bar BAR1 from the storage 56 and displays the display screen WD1. to generate In other words, the user viewing the display screen WD2 can visually and intuitively understand that the state of the inside of the blood vessel in the angioscope image VED1 is different due to the difference in the position of the designation bar BAR1 (see FIGS. 7 and 8). comprehensible.

Next, the display screen WD3 has respective display areas for the angiographic image ANG1 and the angioscopic image VED1. Since the display areas of the angiographic image ANG1 and the angioscopic image VED1 overlap with the above description, the description thereof is omitted.

Further, the display screen WD3 displays a display area of a blood vessel long-axis cross-sectional image LGT1 having a length equal to or longer than the measurement length TAR1 corresponding to the measurement range VSL1, a display area of a button area BT3 for receiving user operations, and the measurement results. and a display area of the measurement result display area MRT3. It shows that the blood vessel diameter was calculated to be 5.3 mm at the measurement start position of the measurement length TAR1, and that the blood vessel diameter was calculated to be 6.1 mm at the measurement end position of the measurement length TAR1. there is A designation bar BAR1 indicating the position of the angioscope 10 is superimposed and displayed on the blood vessel long-axis cross-sectional image LGT1 by user operation. By appropriately sliding the designated bar BAR1 in the horizontal direction within the measurement length TAR1 by the user's operation, the processor 54 responds to the angio image ANG1 in which the marker MK1 appears at the position indicated by the designated bar BAR1 and the position indicated by the designated bar BAR1. Each of the blood vessel endoscope images VED1 is read out from the storage 56 to generate the display screen WD1. That is, the user viewing the display screen WD3 can see the state of the inside of the blood vessel in the angio image ANG1 and the position of the angioscope 10 (more specifically, depending on the position of the designation bar BAR1 (see FIGS. 7 and 8). is the position of the marker MK1) and that the intravascular state in the angioscope image VED1 is different can be visually and intuitively grasped.

FIG. 7 is a diagram showing an angiographic image ANG1 and an angioscopic image VED1 at a first point designated by user operation. FIG. 8 is a diagram showing an angiographic image ANG2 and an angioscopic image VED2 at a second point designated by user operation. 7 and 8, the display area of the button area BT3 for accepting user operations and the display area of the measurement result display area MRT3 for displaying the measurement results are omitted from the display screen WD3 shown in FIG. ing. 7 and 8, elements that are the same as those in FIG. 6 are denoted by the same reference numerals, descriptions thereof are simplified or omitted, and different contents are described.

The first point is a position where the designation bar BAR1 is designated on the proximal side of the measurement length TAR1 (scale: near 40 mm) by user operation, and the marker MK1 in the angio image ANG1 is positioned on the proximal side of the measurement range VSL. are doing. The angioscope image VED1 shows the state inside the blood vessel when the angioscope 10 captures the image at the first point.

On the other hand, the second point is a position where the designated bar BAR1 is designated by the user's operation on the tip side (scale: around 20 mm) of the measurement length TAR1, and the marker MK1 in the angio image ANG2 is positioned on the tip side of the measurement range VSL. are doing. The angioscope image VED2 shows the state inside the blood vessel when the angioscope 10 captures the image at the second point.

In this way, the user can comparatively observe the angiographic image and the angioscope image at the position indicated by the designated bar BAR1 simply by simply sliding the designated bar BAR1 to designate the position to be observed. Convenience can be improved.

As described above, in the blood vessel observation system 100 according to the first embodiment, the distal end TP1 is inserted into the blood vessel of the subject, the proximal end BE1 is fixed, and the distal end TP1 is moved toward the proximal end BE1 by the user's operation. It has an angioscope 10 that images a blood vessel of a subject while being pulled back. The blood vessel observation system 100 is connected to an angiography apparatus (for example, angiography apparatus 90) for imaging the subject's blood vessel and the angiography apparatus 10, and an angiography apparatus 90, respectively, and an angiography imaged by the angiography apparatus. It has a controller (eg, PC 50) that simultaneously acquires an image (eg, angiographic image) and a blood vessel captured image (eg, angioscopic image) captured by the angioscope 10 and associates them. The controller displays a first display screen (for example, display screen WD1) that displays an angiographic image, a second display screen (for example, display screen WD2) that displays an angiographic image, and a second display screen that displays an angiographic image and an angiographic image. Any one of the three display images (for example, the display screen WD3) is switched according to the user's operation and displayed on the monitor 70 or the doctor-facing monitor 80. FIG.

As a result, the blood vessel observation system 100 allows a user such as a doctor to manually pull back the angioscope 10 so as not to damage the blood vessel wall or the like in the subject during surgery or the like. Any one of a display screen WD1 for an angiographic image, a display screen WD2 for an angioscopic image captured by the angioscope 10, and a display screen WD3 for displaying both an angiographic image and an angioscopic image can be easily displayed. It can be displayed by switching. Therefore, the blood vessel observation system 100 can safely support the smooth execution of medical procedures such as surgery, and can easily switch the display screen on which the state of the affected part can be observed. can be appropriately improved, and the user can be assisted in selecting a stent with an appropriate diameter to be inserted into the subject. In particular, the PC 50 associates the simultaneously input angiographic image and the angioscope as images taken at the same time. Velocity can be easily determined, and using the data of position information and velocity information acquired at the timing synchronized (coincided) with data acquisition of angioscopy images, constant-velocity moving images of blood vessel images are generated in real time, The diameter of the blood vessel, which is the observation site of the specimen, can be calculated with high accuracy.

In addition, the controller (eg, PC 50) generates a blood vessel long-axis cross-sectional image (eg, blood vessel long-axis cross-sectional image LGT1) using a plurality of blood vessel captured images input from the angioscope 10, and displays the first display screen, Both the second display screen and the third display screen are displayed including the longitudinal tomographic image. This allows the user to view the long-axis section of the blood vessel regardless of which of the angiographic image on the display screen WD1, the angioscopic image on the display screen WD2, and the angioscopic image and angioscopic image on the display screen WD3 are displayed. Since the image LGT1 is also displayed, the inside of the blood vessel can be observed while comparing the state of the blood vessel long-axis cross-sectional image with the angiographic image or the angioscopic image.

Also, a controller (e.g., PC 50) derives position and speed information of the endoscope during the pullback based on the position of the endoscope in the angiographic image associated with the angiographic image; A measured length of the vessel is derived based on the position and velocity information of the endoscope. As a result, the image console CSL1 can appropriately derive the position information and speed information of the angioscope 10 during pullback from the position of the angioscope 10 in the angiographic image, indicating the length of the blood vessel to be observed. The blood vessel length can be derived with high accuracy.

In addition, the controller uses the position information and speed information of the angioscope 10 and a plurality of corresponding blood vessel captured images of the angioscope 10 to calculate the diameter of the blood vessel. As a result, the blood vessel observation system 100 can accurately generate a constant-speed moving image necessary for calculating the blood vessel diameter, and can derive the blood vessel diameter with high accuracy.

Also, the angiographic image (for example, angiographic image) has a marker MK1 indicating the distal end of the angioscope 10 . The controller derives the position information and speed information of the angioscope 10 during pullback based on the change in the position of the marker MK1. As a result, the blood vessel observation system 100 can appropriately determine the position of the angioendoscope 10 from the angiographic image using the marker MK1, and can appropriately derive the position information and speed information of the angioendoscope 10 during pullback.

Further, when the third display screen (for example, the display screen WD3) is displayed, the controller displays an angiographic image ( For example, an angio image) and an angiography image (for example, angiography image) are displayed in contrast. As a result, the PC 50 can observe the angiographic image and the angioscopic image of the position that the user wants to observe while comparing them at a glance by the user's designation operation using the simple designation bar BAR1.

The angioscope 10 also has an image sensor capable of imaging the blood vessel of the subject. An image sensor is mounted on the distal end of the angioscope. As a result, the blood vessel observation system 100 can generate a display screen on which the state inside the blood vessel can be observed using the high-resolution blood vessel image captured by the angioscope 10, thereby improving user convenience. can.

Various embodiments have been described above with reference to the drawings, but it goes without saying that the present disclosure is not limited to such examples. It is obvious that a person skilled in the art can conceive of various modifications, modifications, substitutions, additions, deletions, and equivalents within the scope of the claims. Naturally, it is understood that it belongs to the technical scope of the present disclosure. In addition, the constituent elements of the various embodiments described above may be combined arbitrarily without departing from the gist of the invention.

INDUSTRIAL APPLICABILITY The present disclosure is useful as a blood vessel observation system and a blood vessel observation method that support smooth medical practices such as surgery, easily switch display screens for observing the state of an affected area, and improve convenience for users such as doctors. be.

10 Angioscope 30 CCU
50 PCs
51 image capture board 52 input interface 53 memory 54 processor 55 operation unit 56 storage 57 output interface 70 monitor 80 doctor-facing monitor 90 angiography device 100 blood vessel observation system BE1 proximal end CSL1 image console GW1 guide wire OS1 outer sheath TP1 distal end

Claims (8)

an endoscope whose distal end is inserted into a blood vessel of a subject, whose proximal end is fixed, and which images the blood vessel of the subject while the distal end is pulled back toward the proximal end by a user's operation; ,
An angiographic image captured by the angiographic device and a blood vessel captured image captured by the endoscope are respectively connected to an angiographic device for imaging the blood vessel of the subject and the endoscope, and the endoscope. and a controller that simultaneously obtains and associates with
The controller is
A user selects any one of a first display screen that displays the angiographic image, a second display screen that displays the captured blood vessel image, and a third display screen that displays the angiographic image and the captured blood vessel image. Switching according to the operation and displaying on the monitor,
Blood vessel observation system. The controller is
A long-axis cross-sectional image of the blood vessel is generated using a plurality of the imaged blood vessel images input from the endoscope, and displayed on any one of the first display screen, the second display screen, and the third display screen Displaying including the long-axis cross-sectional image,
The blood vessel observation system according to claim 1. The controller is
deriving position and velocity information of the endoscope during the pullback based on the position of the endoscope in the angiographic image associated with the angiographic image; deriving a measured length of the vessel based on the velocity information;
The blood vessel observation system according to claim 1. The controller is
calculating the diameter of the blood vessel using a plurality of pieces of the position information and speed information of the endoscope and the corresponding blood vessel captured images of the endoscope;
The blood vessel observation system according to claim 3. The angiographic image has a marker indicating the distal end of the endoscope,
The controller is
deriving position and velocity information of the endoscope during the pullback based on variations in the position of the marker;
The blood vessel observation system according to claim 3. The controller is
When the third display screen is displayed, the angiographic image and the captured blood vessel image corresponding to the specified position are displayed in comparison in response to the position specified by the user operation on the long-axis cross-sectional image;
The blood vessel observation system according to claim 2. The endoscope has an image sensor capable of imaging blood vessels of the subject,
The image sensor is mounted on the distal end of the endoscope,
The blood vessel observation system according to claim 1. A blood vessel observation method executed by a blood vessel observation system, comprising:
The distal end is inserted into the blood vessel of the subject, the proximal end is fixed, and the endoscope images the blood vessel of the subject while the distal end is pulled back toward the proximal end by a user operation. ,
connecting to an angiography device for imaging the blood vessel of the subject and the endoscope and the endoscope, respectively;
Simultaneously acquiring and associating an angiographic image captured by the angiography device and a blood vessel captured image captured by the endoscope,
A user selects any one of a first display screen that displays the angiographic image, a second display screen that displays the captured blood vessel image, and a third display screen that displays the angiographic image and the captured blood vessel image. Switching according to the operation and displaying on the monitor,
Blood vessel observation method.

Images