WO2021085017A1

WO2021085017A1 - Vascular endoscopic system and blood vessel diameter measurement method

Info

Publication number: WO2021085017A1
Application number: PCT/JP2020/037202
Authority: WO
Inventors: 貴光荒井; 章二田口; 伸一塚原; 芳明矢萩; 博文榎本
Original assignee: パナソニックｉ－ＰＲＯセンシングソリューションズ株式会社
Priority date: 2019-10-30
Filing date: 2020-09-30
Publication date: 2021-05-06
Also published as: JP2021069919A; JP2022183211A; JP7157098B2

Abstract

This vascular endoscopic system comprises: an endoscope which has a distal end side inserted into the blood vessel of a subject and a proximal end side pulled by a driving apparatus and which captures an image of the blood vessel of the subject; a repeater which receives and combines a blood vessel image captured by the endoscope and the position information and velocity information on the endoscope sent from the driving apparatus; and a calculation device which stores, in a memory, the blood vessel image and the position information and velocity information sent from the repeater in association with each other, generates a constant-velocity moving image of the blood vessel image by using the blood vessel image and the position information and velocity information on the endoscope stored in the memory, and calculates the blood vessel diameter of the blood vessel on the basis of a plurality of blood vessel images constituting the constant-velocity moving image.

Description

Vascular endoscopy system and method of measuring blood vessel diameter

The present disclosure relates to a blood vessel endoscopy system and a method for measuring blood vessel diameter.

In Patent Document 1, it is possible to quickly and easily determine the optimum stent, and the validity of the determined stent can be easily confirmed before healing, while image analysis for comparing blood vessel images with different diagnosis dates and times with high accuracy. The device is disclosed. This image analysis device stores a short-axis cross-sectional image of a blood vessel output from an intravascular image imaging device, and generates a long-axis cross-sectional image of a blood vessel from a plurality of saved short-axis cross-sectional images. In addition, the image analyzer generates a lumen closure curve at least along the outer circumference of the blood vessel lumen from the stored short-axis cross-sectional image, and the diameter of the stent to be inserted into the blood vessel from this generated lumen closure curve. Is calculated.

Japanese Patent Application Laid-Open No. 2009-240359

However, in the configuration of Patent Document 1, in determining the stent diameter, a large number of each short-axis cross-sectional image displayed on the display unit along the outer circumference of the lumen reflected in the short-axis cross-sectional image by a user such as a doctor. It is premised that the designated point of is specified. For this reason, a user such as a doctor is required to perform a plurality of designated operations during medical treatment such as surgery, which may make it difficult for the medical treatment to proceed smoothly, which is improved in terms of improving usability. It can be said that there was room for.

This disclosure was devised in view of the above-mentioned conventional situation, and observation of a subject such as a patient using an image taken by an angioscope without interrupting the smooth progress of medical practice by a user such as a doctor. It is an object of the present invention to provide a blood vessel endoscopy system and a blood vessel diameter measuring method for measuring a blood vessel diameter at a site with high accuracy.

The present disclosure describes a blood vessel of a subject whose distal end is inserted into the blood vessel of the subject and whose proximal side is pulled by a drive device at a constant speed in an automatic mode or manually in a manual mode at a variable speed. From the endoscope that can be imaged, a repeater that inputs and combines the blood vessel image captured by the endoscope and the position information and speed information of the endoscope sent from the drive device, and the repeater. The transmitted blood vessel image is associated with the position information and speed information of the endoscope and stored in a memory, and the blood vessel image stored in the memory and the position information and speed information of the endoscope are used. Provided is a blood vessel endoscopy system including a calculation device that generates a constant velocity moving image of the blood vessel image and calculates the blood vessel diameter of the blood vessel based on a plurality of blood vessel images constituting the constant velocity moving image.

Further, in the present disclosure, the distal end side is inserted into the blood vessel of the subject and the proximal end side is pulled at a constant speed by an automatic mode by a driving device, or manually pulled at a variable speed by a manual mode. An endoscope capable of capturing a blood vessel, a blood vessel image captured by the endoscope, and position information and speed information of the endoscope sent from the driving device are input, associated with each other, and stored in a memory. A constant velocity moving image of the blood vessel image is generated based on the difference between the input timing of the blood vessel image stored in the memory and the input timing of the position information and the velocity information of the endoscope, and the constant velocity moving image is configured. Provided is a blood vessel endoscopy system including an arithmetic device for measuring the blood vessel diameter of the blood vessel based on a plurality of blood vessel images.

Further, the present disclosure is a blood vessel diameter measuring method performed by a blood vessel endoscopy system, in which the distal end side is inserted into the blood vessel of a subject and the proximal end side is pulled by a driving device at a constant speed in an automatic mode. The blood vessel is imaged by an endoscope capable of imaging the blood vessel of the subject, which is manually pulled at a variable speed in the manual mode, and the blood vessel image captured by the endoscope and the blood vessel image sent from the driving device are sent. The position information and speed information of the endoscope are input by the repeater and combined, and the blood vessel image sent from the repeater is associated with the position information and speed information of the endoscope and stored in the memory. A constant velocity moving image of the blood vessel image is generated using the blood vessel image stored in the memory and the position information and velocity information of the endoscope, and based on a plurality of blood vessel images constituting the constant velocity moving image. Provided is a blood vessel diameter measuring method for calculating a blood vessel diameter of the blood vessel.

Further, the present disclosure is a blood vessel diameter measuring method performed by a blood vessel endoscopy system, in which the distal end side is inserted into the blood vessel of a subject and the proximal end side is driven by a driving device at a constant speed in an automatic mode. The blood vessel is imaged by an endoscope capable of imaging the blood vessel of the subject, which is pulled or manually pulled at a variable speed in a manual mode, and the blood vessel image captured by the endoscope and the driving device are used. The position information and speed information of the endoscope sent from the endoscope are input and associated with each other and stored in the memory, and the input timing of the blood vessel image saved in the memory and the position information and speed information of the endoscope are recorded. Provided is a blood vessel diameter measuring method for generating a constant velocity moving image of the blood vessel image based on a difference from an input timing and measuring the blood vessel diameter of the blood vessel based on a plurality of blood vessel images constituting the constant velocity moving image. ..

Further, in the present disclosure, an endoscope capable of imaging the blood vessel of the subject while the guide wire is inserted into the blood vessel of the subject inserted in advance and pulled through a driving device, and the endoscope are used for imaging. A repeater that inputs and combines the blood vessel image and the position information and speed information of the endoscope sent from the drive device, the blood vessel image sent from the repeater, the position information of the endoscope, and the position information of the endoscope. The speed information is associated with the memory and stored in the memory, and based on the size of the guide wire in the width direction stored in the memory and a single blood vessel image corresponding to the position information and the speed information of the endoscope. Provided is a blood vessel endoscopy system including an arithmetic device for calculating the blood vessel diameter of the blood vessel.

Further, the present disclosure is a blood vessel diameter measuring method performed by a blood vessel endoscopy system, in which a guide wire is inserted into a blood vessel of a subject inserted in advance, and the subject is pulled through a driving device. The blood vessel is imaged by an endoscope capable of imaging the blood vessel of the blood vessel, and the blood vessel image captured by the endoscope and the position information and speed information of the endoscope sent from the driving device are input by a repeater. The blood vessel image sent from the repeater is associated with the position information and the speed information of the endoscope and stored in a memory, and the size of the guide wire stored in the memory in the width direction and the above. Provided is a blood vessel diameter measuring method for calculating the blood vessel diameter of the blood vessel based on one blood vessel image corresponding to the position information and the speed information of the endoscope.

According to the present disclosure, it is possible to measure the blood vessel diameter of an observation site of a subject such as a patient with high accuracy by using an image taken by an angioscope without interrupting the smooth progress of medical practice by a user such as a doctor. ..

The figure which shows the structural example of the vascular endoscopy system which concerns on Embodiment 1. Flow chart showing an example of the operation procedure of the vascular endoscopy system according to the first embodiment A flowchart showing an example of the operation procedure of the blood vessel diameter calculation process in step S43 of FIG. A diagram showing the arrangement of feature points on a blood vessel image, the positional relationship between feature points and corresponding points, and the blood vessel diameter estimated using three-dimensional points. The figure explaining the calculation example of the 3D position Diagram showing an example of a screen displayed on a monitor The figure which shows the structural example of the vascular endoscopy system which concerns on Embodiment 2. Flow chart showing an example of the operation procedure of the vascular endoscopy system according to the second embodiment Flow chart showing an example of the operation procedure of the vascular endoscopy system according to the third embodiment The figure which shows the outline example of the 1st calculation process of the blood vessel diameter which concerns on Embodiment 3. A flowchart showing an example of an operation procedure of the first calculation process of the blood vessel diameter in step S43B of FIG. The figure which shows the outline example of the 2nd calculation process of the blood vessel diameter which concerns on Embodiment 3. A flowchart showing an example of an operation procedure of the second calculation process of the blood vessel diameter in step S43B of FIG. The figure which shows the outline example of the 3rd calculation process of the blood vessel diameter which concerns on Embodiment 3. A flowchart showing an example of an operation procedure of the third calculation process of the blood vessel diameter in step S43B of FIG. The figure which shows the outline example of the 4th calculation process of the blood vessel diameter which concerns on Embodiment 3. A flowchart showing an example of an operation procedure of the fourth calculation process of the blood vessel diameter in step S43B of FIG.

Hereinafter, embodiments in which the blood vessel endoscopy system and the blood vessel diameter measurement method according to the present disclosure are specifically disclosed will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of already well-known matters and duplicate explanations for substantially the same configuration may be omitted. This is to avoid unnecessary redundancy of the following description and to facilitate the understanding of those skilled in the art. It should be noted that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

(Embodiment 1)
FIG. 1 is a diagram showing a configuration example of the blood vessel endoscopy system 5 according to the first embodiment. The vascular endoscopy system 5 targets a subject such as a human body and measures the diameter of blood vessels in the subject. The vascular endoscope system 5 includes a vascular endoscope 100, an auto pullback device 80, a repeater 20, a camera control unit 30, a PC 50 (Personal Computer), and a monitor 70.

(Structure of angioscopy system)
The vascular endoscope 100 as an example of the endoscope is a dedicated medical device used at the time of surgery or examination, in which the vascular endoscope camera 10 is attached to the tip of the catheter 200. The vascular endoscope 100 is sometimes referred to as a so-called vascular endoscopic catheter. The outer diameter of the vascular endoscope 100 is, for example, 1.8 mmΦ as the maximum outer diameter, but the outer diameter is not limited to this size. The angioscope 100 is inserted in the blood vessel in the subject freely along a guide wire 150 inserted in advance in the observation site (for example, a blood vessel) in the subject. Here, the direction in which the vascular endoscopic camera 10 is inserted toward the observation site in the subject is defined as the traveling direction, and conversely, the direction in which the vascular endoscopic camera 10 is pulled out toward the outside of the subject is retracted. Defined as direction. Therefore, advancing and retreating means that the vascular endoscopic camera 10 can be inserted and removed toward the inside of the subject. The vascular endoscopic camera 10 can be smoothly inserted to the site to be observed by being guided by a guide wire 150 inserted in advance to the site to be operated or examined (for example, the affected area). The vascular endoscope 100 may be one in which the vascular endoscope camera 10 is interchangeably attached to the tip of a normal catheter. The catheter 200 is, for example, a medical tube used for discharging a body fluid or injecting a drug solution. In addition to the vascular endoscopic camera 10, a balloon, a stent, or the like may be interchangeably attached to the catheter 200.

The blood vessel endoscopy camera 10 is, for example, a 480,000-pixel high-resolution camera equipped with an image sensor (see below) capable of imaging blood vessels on the tip side. Note that 480,000 pixels are just an example, and the number of pixels does not have to be limited to 480,000 pixels. When the blood vessel endoscope 100 is inserted into a blood vessel in a subject, the blood vessel endoscope camera 10 can image the inner wall of the blood vessel (hereinafter, referred to as “blood vessel wall”). The angioscope camera 10 incorporates a solid-state image sensor (that is, an image sensor) such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Signal) as an image sensor, and incorporates a solid-state image sensor (that is, an image sensor) such as a subject (for example, the inside of a blood vessel which is an affected area). The light from the blood vessel wall) is imaged on the image pickup surface, the formed optical image is converted into an electric signal, and the data signal of the image pickup image is output. When the blood vessel endoscopic camera 10 is inserted into the blood vessel, it is pulled back at a constant speed by driving the auto pullback device 80 based on the user's operation, so that the blood vessel wall is imaged at equal intervals, and the data of the captured image of the blood vessel wall is taken. Output a signal. Hereinafter, the captured image of the blood vessel wall or the like will be referred to as a “blood vessel image”. The data signal of the blood vessel image may be either a still image or a moving image. Further, the blood vessel endoscope 100 may include a fiber that guides the irradiation light from the LED (Light Mission Diagram) light source or the camera control unit 30 in the subsequent stage in order to illuminate the subject.

The auto pullback device 80 as an example of the drive device pulls back the vascular endoscope 100, which is guided by the guide wire 150 and inserted into the observation target site (for example, a blood vessel), to the proximal end side at a pullback speed (for example, a constant speed). Performs an action (in other words, towing). At this time, the length at which the auto-pullback device 80 pulls back the catheter 200 (in other words, the vascular endoscope 100) is the length at which the vascular endoscope camera 10 inserted at a position near the affected part is pulled back toward the proximal end side. It can be considered that it is almost the same as. The vascular endoscope camera 10 images the blood vessel wall at equal intervals when the vascular endoscope 100 is pulled back by the auto pullback device 80. The auto pullback device 80 transmits data of position information of the blood vessel endoscope 100 (more specifically, the blood vessel endoscope camera 10; the same applies hereinafter) and speed information of the imaging position to the repeater 20.

Here, the speed information of the imaging position of the angioscope 100 is the imaging position of the angioscope camera 10 (that is, when the angioscope camera 10 of the angioscope 100 is pulled back to the proximal end side). This is the speed at which the position of the angioscope camera 10) moves to the proximal end side at a constant speed, which is substantially the same as the pullback speed (see above). The speed information of the imaging position of the blood vessel endoscope 100 is set by the auto pullback device 80 by the operation of a user such as a doctor, and as an example, it is 0.5 mm / sec, 1.0 mm / sec, or 2.0 mm / sec. is there. Further, the start position of imaging is determined after, for example, a user such as a doctor confirms the captured image of the observation site (for example, blood vessel) of the subject imaged by the blood vessel endoscope 100 on the monitor 70. The pullback of the vascular endoscope 100 is started by, for example, pressing the button bn2 of the auto pullback device 80 by a user such as a doctor.

On the front surface of the housing 80z of the auto pullback device 80, there is a button bn1 that can select the speed information of the imaging position, a button bn2 that switches between start and stop by pressing the button, and a catheter 200 (in other words, a blood vessel). Buttons bn3 that automatically return the endoscope 100) to the initial position on the proximal end side are arranged respectively. The velocity information of the imaging position of the angioscope 100 (sometimes simply referred to as “velocity information”) is a constant velocity selected by pressing the button bn1, for example, 0.5 mm / sec, 1.0 mm / sec, or It is 2.0 mm / sec. Further, on the front surface of the housing 80z, a measurement length representing the amount of movement from the start position of imaging by the blood vessel endoscopy camera 10 (in other words, the start position of the observation site) to the proximal end side of the blood vessel endoscopy camera 10. , Display d1, d2, d3 that display the speed and acceleration of movement of the catheter 200 (in other words, the blood vessel endoscope 100) are arranged. Further, on the front surface of the housing 80z, a display d4 that schematically displays the position of the blood vessel endoscope 100 is arranged.

The repeater 20 has a configuration including an FPGA (Field Programmable Gate Array) 21, an AFE (Analog Front End) 22, input interfaces 23 and 24, and an output interface 25. In each of FIGS. 1 and 7, the interface is abbreviated as "I / F".

The input interface 23 is connected to the vascular endoscope 100 so as to be able to input a data signal, and inputs a data signal (for example, a moving image or a still image) of a vascular image captured by the vascular endoscope camera 10. Is output to AFE22.

The input interface 24 is connected to the auto pullback device 80 so that data can be input, and inputs the position information and speed information data of the vascular endoscope 100 output from the auto pullback device 80 and outputs the data to the FPGA 21. To do.

AFE22 is composed of an integrated circuit including at least an amplifier, an AD (Analog Digital) converter and a filter. The AFE 22 performs amplification processing, analog-digital conversion processing, filtering processing, and the like on the data signal of the blood vessel image input via the input interface 23, and outputs the data signal to the FPGA 21.

The FPGA 21 combines the data of the blood vessel image after being processed by the AFE 22 with the data of the position information and the velocity information of the blood vessel endoscope 100. In the following description, the combined data of the blood vessel image data and the data of the position information and the velocity information may be simply referred to as "combined data".

The output interface 25 outputs the combined data generated by the FPGA 21 (that is, the combined data of the blood vessel image data and the position information and speed information data of the blood vessel endoscope 100) to the image console 90. Here, the FPGA 21 is used as an example of the processor used in the repeater 20, but in addition to the FPGA 21, a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), an MPU (Micro Processing Unit), and the like are used. You may.

The repeater 20 relays various signals performed between the angioscope camera 10 and the camera control unit 30. The various signals include, for example, various control signals for the camera control unit 30 to control the vascular endoscopic camera 10 in addition to the data signal of the vascular image captured by the vascular endoscopic camera 10.

As described above, the repeater 20 combines the data of the blood vessel image captured by the blood vessel endoscope camera 10 with the data of the position information and the speed information of the blood vessel endoscope 100 pulled back by the auto pullback device 80. .. Here, the processing of combining the blood vessel image data with the position information and velocity information data is performed by concatenating these data, for example, by adding these data into one data. In this way, by coupling with the repeater 20, the imaging position in the blood vessel provided by the auto pullback device 80 and the acquisition timing of the blood vessel image provided by the blood vessel endoscope 100 are matched (synchronized). Attached.

Note that the above-mentioned data combination may be performed, for example, by drawing position information and velocity information on the blood vessel image, that is, by superimposing them. Further, the blood vessel image data and the position information and velocity information data of the blood vessel endoscope 100 may be associated with each other as separate files. This association is performed, for example, by storing the identification information of the other data file in the metadata storage area of one data file. The combined blood vessel image and the position information and the velocity information may be displayed on separate screens, or may be displayed on the same screen by the superimpose method. When the auto pullback device 80 pulls back the catheter 200 (that is, the blood vessel endoscope 100) at a constant speed in the automatic mode, the catheter 200 (that is, the blood vessel endoscope 100) is caught in the middle of the blood vessel which is the observation site, so that the speed is set to a preset constant speed. May not be obtained. In such a case, the blood vessel image provided by the blood vessel endoscope 100 when a constant speed cannot be obtained is omitted (that is, not used) at high speed when creating a constant velocity moving image described later, and is low speed. Sometimes the same image is added as many times as necessary.

The camera control unit 30 (CCU: Camera Control Unit) is electrically connected to the vascular endoscopy camera 10 via a repeater 20, and the imaging operation by the vascular endoscopy camera 10 and the imaging operation from the vascular endoscopy camera 10 Controls the generation of vascular image data based on the vascular image data signal. The camera control unit 30 adds metadata to the data in which the blood vessel image data and the position information and velocity information data of the angioscope are combined (that is, the above-mentioned combined data). The metadata includes data such as the imaging date and time of the blood vessel image provided by the blood vessel endoscopic camera 10.

The camera control unit 30 includes at least an image input unit (not shown), an image processing unit (not shown), and an image output unit (not shown). The image input unit (not shown) inputs the combined data in which the blood vessel image data and the position information and velocity information data of the blood vessel endoscope 100 are combined. The image input unit (not shown) uses HDMI (registered trademark) (High-Definition Multimedia Interface) or USB (Universal Serial Bus) Type-C, which can transfer video data at high speed, in addition to a dedicated image input interface. It may be the interface that was used. The image processing unit (not shown) performs processing such as adding metadata to the input combined data. Further, the image processing unit (not shown) generates the combined data in RGB format or YUV format that can be visually recognized on the monitor 70 by performing predetermined image processing on the combined data sent from the repeater 20. You may. The image output unit (not shown) transmits the combined data to which the metadata is added to the PC 50.

The PC 50 as an example of the arithmetic unit has a configuration including a processor 51, a memory 52, an input / output interface 53, an operation unit 54, and a storage 55. The PC 50 receives the combined data generated by the repeater 20 via the camera control unit 30. The PC 50 records and stores the blood vessel image data included in the combined data, the blood vessel image data after performing predetermined image processing on the blood vessel image data, and the like in the storage 55. The PC 50 performs a process of calculating the blood vessel diameter of the observation site (for example, the blood vessel of the affected portion of the patient as the subject) based on the blood vessel image data. The PC 50 outputs the measurement result of the blood vessel diameter or the data of the blood vessel image to the monitor 70, and performs a process of visualizing the calculation result of the blood vessel diameter.

Further, the PC 50 creates a constant velocity moving image composed of a plurality of blood vessel images by using the combined data input from the repeater 20 at any time. The constant-velocity moving image referred to here is, for example, an image taken from a plurality of blood vessel images at the same constant speed using the imaging position of the blood vessel endoscope camera 10 and the speed information of the blood vessel endoscope 100. This is a moving image composed by sequentially selecting a plurality of blood vessel images having different positions in chronological order.

The processor 51 executes, for example, the above-mentioned blood vessel diameter measurement process and visualization process by executing various processing programs stored in the memory 52. The processor 51 may be, for example, a GPU suitable for image processing, a dedicated electronic circuit designed by an MPU, a CPU, an ASIC (Application Specific Integrated Circuit), or the like, or an electronic circuit designed to be reconfigurable by an FPGA or the like. It may be configured.

The memory 52 is a RAM (Random Access Memory) used as a working memory of the processor 51. It includes a ROM (Read Only Memory) that stores programs for various processes executed by the processor 51.

The input / output interface 53 may be an interface using HDMI (registered trademark) or USB Type-C, which can transfer video data at high speed, in addition to the dedicated image input interface.

The operation unit 54 includes an activation switch for activating the vascular endoscopy system 5, and accepts operations by a user such as a doctor. In addition to the activation switch described above, the operation unit 54 may include, for example, a mouse, keyboard, touch pad, touch panel, microphone or other input device.

The storage 55 as an example of the memory is a large-capacity storage device, and stores blood vessel image data and the like captured by the blood vessel endoscopy camera 10. The storage 55 may include, for example, a secondary storage device (for example, HDD (Hard Disk Drive) or SSD (Solid State Drive)), or a tertiary storage device (for example, an optical disk or SD card).

The monitor 70 displays the blood vessel diameter measurement result or blood vessel image data output from the PC 50. The monitor 70 has a display device such as an LCD (Liquid Crystal Display), an organic EL (Electroluminescence), or a CRT (Cathode Ray Tube). The camera control unit 30, the PC 50, and the monitor 70 are mounted in a single housing as an image console 90.

(Operation of angioscopy system)
Next, the operation of the vascular endoscopy system 5 according to the first embodiment will be described.

For example, the blood vessel endoscope 100 is inserted into a blood vessel in order to observe the state of the blood vessel in the subject, such as a thrombus in the blood vessel in the subject or a plaque formed on the blood vessel wall. To. When a user such as a doctor inserts the blood vessel endoscope 100 into the blood vessel, the guide wire 150 is inserted into the blood vessel in advance. When the guide wire 150 reaches the inside of the blood vessel to be observed, the user can use the blood vessel endoscope 100 (in other words, the blood vessel at the tip) in which the blood vessel endoscope camera 10 is mounted on the tip side so as to be guided by the guide wire 150. A catheter 200) to which the endoscope 100 is attached is advanced and inserted into the blood vessel. When the angioscope camera 10 reaches the inside of the blood vessel to be observed, the user activates the auto pullback device 80 and pulls back the catheter 200 (in other words, the angioscope 100) at a pullback speed (that is, a constant speed). Start the operation. Further, when the inside of the blood vessel is imaged by the blood vessel endoscope camera 10, the user can use a low molecular weight dextran or a low molecular weight dextran from the proximal end side of the catheter 200 to which the blood vessel endoscope 100 is attached so that the inside of the blood vessel can be clearly imaged. A contrast medium or a clear solution such as physiological saline is injected into the blood vessel.

FIG. 2 is a flowchart showing an example of the operation procedure of the blood vessel endoscopy system 5 according to the first embodiment. In FIG. 2, in addition to the blood vessel endoscope 100, the repeater 20, and the image console 90 constituting the blood vessel endoscopy system 5, the operations of the auto pullback device 80 are shown in chronological order.

In FIG. 2, when a user such as a doctor presses the activation switch included in the operation unit 54 of the PC 50 housed in the housing of the image console 90, the blood vessel endoscopy system 5 is activated (S1). When the vascular endoscope system 5 is activated, the vascular endoscope 100, the auto pullback device 80, the repeater 20, and the image console 90 each start operating.

The user sets the start position and the end position of the measurement for pulling back the blood vessel endoscope 100 inserted into the blood vessel, which is the observation site in the subject, with respect to the auto pullback device 80. At this time, the user actually visually confirms the blood vessel image captured by the blood vessel endoscopy camera 10 on the monitor 70 on which the blood vessel image is displayed, and determines the start position and the end position of the measurement. The length from the start position to the end position of the pullback (measurement) of the blood vessel endoscope 100 by the auto pullback device 80 corresponds to the measurement length of the blood vessel.

The auto pullback device 80 acquires the position information and speed information of the vascular endoscope 100 measured at regular intervals when the catheter 200 (that is, the vascular endoscope 100) is pulled back at a constant speed within the measurement length. (S11). The auto pullback device 80 outputs data of the position information of the blood vessel endoscope 100 and the speed information of the imaging position to the repeater 20 (S12).

On the other hand, the blood vessel endoscope 100 acquires a data signal of a blood vessel image captured by the blood vessel endoscope camera 10 (S21). The blood vessel endoscope 100 outputs a data signal of a blood vessel image to the repeater 20 (S22).

On the other hand, the repeater 20 inputs the data signal of the blood vessel image from the blood vessel endoscope 100 (S31). The repeater 20 inputs the position information and the speed information of the vascular endoscope 100 from the auto pullback device 80 (S32). The repeater 20 inputs the position information and speed information data of the vascular endoscope 100 input from the auto pullback device 80 at the input timing (that is, the same time as the input) of the data signal of the blood vessel image to the above-mentioned input timing. It is combined with the data of the blood vessel image input to (S33). That is, the data of the blood vessel image from the blood vessel endoscope 100 input to the repeater 20 and the position information and the speed information of the blood vessel endoscope 100 from the auto pullback device 80 are time-matched (synchronized). Data is combined by the repeater 20.

The repeater 20 transmits the combined data generated in step S33 (that is, the combined data of the blood vessel image and the position information and speed information of the blood vessel endoscope 100) to the image console 90 (S34). The transmission of the combined data is performed, for example, at a frame rate at which a user such as a doctor can view the blood vessel image captured by the blood vessel endoscope 100 in real time and display it on the monitor 70.

On the other hand, in the image console 90, the camera control unit 30 receives the combined data generated by the repeater 20 (that is, the combined data of the blood vessel image and the position information and the speed information of the blood vessel endoscope 100). (S41). The PC 50 stores this combined data in the storage 55. The PC 50 creates a constant velocity moving image (see above) based on the combined data (S42).

The PC50 calculates the blood vessel diameter in the blood vessel, which is the observation site of the subject, using the data of at least two blood vessel images constituting the constant velocity moving image (S43). The details of the process in step S43 will be described later. Further, the PC 50 separates the combined blood vessel image and the position information and speed information data of the blood vessel endoscope 100 in order to separately display the blood vessel image and the position information and speed information of the blood vessel endoscope 100. (S44). The PC 50 displays the blood vessel image, the position information and velocity information of the blood vessel endoscope 100, and the blood vessel diameter on the monitor 70 (S45). The position information and the speed information may be displayed at the same time or may be displayed individually. After this, the vascular endoscopy system 5 ends the operation shown in FIG.

FIG. 3 is a flowchart showing an example of an operation procedure of the blood vessel diameter calculation process in step S43 of FIG. This process is, for example, a process of calculating the blood vessel diameter using two blood vessel images constituting the constant velocity moving image created in step S42 of FIG. The user only has to enter the center and radius of the circle for the PC50. The processor 51 receives the center and radius of the circle input by the user via the operation unit 54. The processor 51 has a plurality of (for example, 1156) measurement points arranged on the entire image on the circumference based on the center and radius of the circle with respect to the data of the blood vessel image input from the camera control unit 30. For example, 128) feature points e1 are detected (S51). The number of 1156 measurement points and 128 feature points is an example.

FIG. 4 is a diagram showing the arrangement of feature points on a blood vessel image, the positional relationship between feature points and corresponding points, and the blood vessel diameter estimated using three-dimensional points. 128 feature points e1 are superimposed and drawn on the circumference of a circle designated by the user on the blood vessel image GZ1 captured by the blood vessel endoscopy camera 10.

The processor 51 determines whether or not the blood vessel image GZ1 that has detected the feature point e1 is the image of the first frame (S52). In the case of the image of the first frame (S52, YES), the processor 51 ends the process shown in FIG. 3 and returns to the original process without calculating the blood vessel diameter.

On the other hand, when the image is from the second frame onward in step S52 (S52, NO), the processor 51 performs the first feature point matching (S53). In the first feature point matching, the processor 51 acquires a rectangular area near the feature point of the previous frame (in other words, the (n-1) th frame) as a template (n: an integer of 2 or more). As an example, the template size is a size of 16 pixels in width × 16 pixels in height.

The processor 51 searches for a region that matches the template with a rectangular region centered on the feature point position of the current frame (in other words, the nth frame) as a search range. The size of the search range is 128 pixels wide x 128 pixels high. The processor 51 has a corresponding point f2 corresponding to the feature point e1 at a position where the ZNCC (Zero-means Normalized Cross Direction) value is minimized. Since the nth frame is an image in which the blood vessel endoscope 100 is pulled toward the front with respect to the (n-1) th frame, a plurality of circles formed by the plurality of corresponding points f2 are formed. It is smaller than the circle formed by the feature point e1.

The processor 51 estimates the vanishing point dp using the feature point e1 included in the (n-1) th frame and the corresponding point f2 included in the nth frame (S54). In the estimation of the vanishing point dp, the processor 51 obtains the flow from the feature point e1 to the corresponding point f2 for all the feature points e1 and the corresponding point f2. The processor 51 finds the intersection of the two flows. The processor 51 obtains a vector from all the feature points e1 to the intersection. The processor 51 obtains the similarity between the flow and each vector (here, the angle difference), and determines whether or not the similarity exceeds the threshold value, for example, whether or not the angle difference between the flow and the vector is less than 3 °. Determine. When the angle difference is less than 3 °, the processor 51 determines that the feature point e1 is an effective feature point (hereinafter, may be referred to as an "inlier"). On the other hand, when the angle difference is 3 ° or more, the processor 51 determines that the feature point e1 is an invalid feature point (hereinafter, may be referred to as an “outlier”). The processor 51 calculates the number of inliers at the intersections of all the flows. The processor 51 uses the intersection of the flows having the largest number of inliers as the vanishing point dp.

The processor 51 performs a second feature point matching for cross-checking (S55). In the second feature point matching, the processor 51 acquires a rectangular area near the corresponding point of the current frame (that is, the nth frame) as a template. As an example, the template size is a size of 16 pixels in width × 16 pixels in height. The processor 51 searches for a region that matches the template with a rectangular region centered on the corresponding point position of the previous frame (that is, the (n-1) th frame) as a search range. The size of the search range is 128 pixels wide x 128 pixels high. The processor 51 sets the position where the ZNCC value becomes the minimum as a feature point (corresponding feature point) corresponding to the corresponding point f2. The processor 51 determines whether or not the feature point e1 and the corresponding feature point in the previous frame (that is, the (n-1) th frame) substantially match. The substantially match between the feature point e1 and the corresponding feature point can be determined based on, for example, the position coordinates. The processor 51 determines that the corresponding point f2 is reliable when the feature point e1 and the corresponding feature point substantially match, and when the feature point e1 and the corresponding feature point do not substantially match, the corresponding point f2 is unreliable. Judge. The processor 51 adopts the feature point e1 judged to be reliable as an inlier, and does not adopt the feature point e1 judged to be unreliable as an outlier.

The processor 51 calculates the three-dimensional position of the feature point e1 which is an in-liner (S56). For example, triangulation is used to calculate the three-dimensional position of the feature point e1. Triangulation is a well-known surveying method using trigonometry and geometry that finds the position of a feature point by measuring the distance between the two points and the angle from each of these two points to the feature point to be measured.

FIG. 5 is a diagram illustrating an example of calculating a three-dimensional position. Here, let the three-dimensional coordinates E of the feature point e1 be (X, Y, Z). X is a coordinate value on the x-axis representing the diameter (minor axis) direction of the blood vessel. Y is a coordinate value on the y-axis representing the radial direction of the blood vessel perpendicular to the x-axis. Z is a coordinate value of the z-axis representing the longitudinal (major axis) direction of the blood vessel. Let (u1, u2) be the image coordinates p1 of the feature point e1 at the camera position g1 of the (n-1) th frame. Let (u2, v2) be the image coordinate p2 of the corresponding point f2 at the camera position g2 of the nth frame. Let D be the distance between the camera position g1 of the (n-1) th frame and the camera position g2 of the nth frame. Here, the distance D is the imaging time difference and pullback speed between the camera position of the (n-1) th frame and the camera position of the nth frame when the auto pullback device 80 pulls the guide wire 150 at a constant pullback speed. It is calculated by the product of and.

The matrix K of the internal parameters of the camera is expressed by the mathematical formula (1).

Here, fx: the value obtained by dividing the focal length by the horizontal pixel pitch, fy: the value obtained by dividing the focal length by the vertical pixel pitch, Cx: the x-coordinate of the image center, and Cy: the y-coordinate of the image center.

The image coordinates p1 and the three-dimensional coordinates E of the feature point e1 are expressed by the mathematical formula (2).

The image coordinates p2 and the three-dimensional coordinates E of the corresponding point f2 are expressed by the mathematical formula (3).

The processor 51 uses, for example, a triangulation function to obtain the three-dimensional coordinates E of the feature point e1 corresponding to the image coordinates p1 and the image coordinates p2.

The processor 51 can display (3D display) the three-dimensional coordinates E of the feature point e1 on the monitor 70 as a three-dimensional stereoscopic image.

The processor 51 detects a plane on which ellipse fitting is performed based on the three-dimensional coordinates E of the plurality of feature points e1 (S57). In the plane detection, the processor 51 selects three points from a plurality of feature points e1 (referred to as three-dimensional points), and obtains an equation (referred to as a planar equation) representing a plane including these three points. The processor 51 counts the number of three-dimensional points that are close to a plane within a predetermined distance from the obtained plane. The processor 51 selects the planar expression having the largest number of counted three-dimensional points. The processor 51 extracts a three-dimensional point close to the plane represented by the selected plane formula. The processor 51 performs principal component analysis (PCA: Principal Component Analysis) based on the extracted three-dimensional point cloud, and detects a plane. Principal component analysis is a method of multivariate analysis that synthesizes a variable called the principal component that best represents the overall variability with a small number of uncorrelated variables from a large number of correlated variables.

The processor 51 adopts the feature point e1 whose distance from the plane is more than a predetermined distance as an outliner and the feature point e1 which is within a predetermined distance as an inlier for estimating the blood vessel diameter.

The processor 51 estimates the blood vessel diameter by performing elliptical fitting based on the plurality of feature points e1 (S58). In the elliptical fitting, a plurality of feature points e1 are projected onto the xy plane, and the two-dimensional points projected on the xy plane are fitted using RANSAC (Random Sample Consensus). RANSAC is a method of estimating elliptical parameters so as not to include outliers.

By using RANSAC, the processor 51 can estimate an ellipse by suppressing the influence of outliers even if a plurality of two-dimensional points include outliers. Therefore, the estimation accuracy of the ellipse is improved. Specifically, the processor 51 inputs a plurality of two-dimensional points and randomly extracts five points from the input two-dimensional points. The processor 51 obtains an ellipse parameter using the extracted five points. Elliptical parameters include major and minor diameters. The processor 51 calculates the shortest distance from each two-dimensional point to the elliptical arc, counts the number of two-dimensional points (inlier number) whose distance is smaller than the threshold value, and records it in the memory 52. The processor 51 extracts another five points from the plurality of input two-dimensional points, and counts the number of inliers in the same procedure as described above. When the counted number of inliers exceeds the number of inliers recorded in the memory 52, the processor 51 updates the number of inliers recorded in the memory 52. The processor 51 repeats the same procedure, and ends the extraction of the two-dimensional points when the number of inliers recorded in the memory 52 is not continuously updated a certain number of times, that is, when the maximum number of inliars is obtained. .. The processor 51 uses an elliptical parameter that maximizes the number of inliers for all two-dimensional points, and determines as an inlyre a two-dimensional point in which the shortest distance from each two-dimensional point to the elliptical arc is smaller than the threshold value. Processor 51 uses all the determined aligners to determine the elliptical parameters. As a result of obtaining the ellipse parameter using all the aligners, the processor 51 estimates that the major axis, which is one of the ellipse parameters, is the blood vessel diameter φ1. The processor 51 can appropriately determine the size of the stent that can be inserted into the blood vessel by setting the diameter of the blood vessel to be the major diameter.

Here, the processor 51 acquires the corresponding points corresponding to the feature points by template matching. For example, the processor 51 uses the feature points of the (n-1) th frame and the corresponding points of the nth frame for deep learning. The machine learning by the above is performed, and the trained model generated as a result of the machine learning may be used to acquire the corresponding points of the nth frame corresponding to the feature points of the (n-1) th frame.

Further, although the processor 51 estimates the major axis of the elliptical parameter as the blood vessel diameter, the minor axis may be estimated as the blood vessel diameter. Further, the processor 51 may use the major axis and the minor axis, for example, add the major axis and the minor axis and use half the value to estimate the blood vessel diameter.

When the blood vessel endoscope 100 is pulled back at a constant speed by the auto pullback device 80, the PC 50 sequentially acquires blood vessel images corresponding to the position information of the blood vessel endoscope 100 captured by the blood vessel endoscope camera 10. Record as a blood vessel image of a short axis cross section in the storage 55. Here, the minor axis cross section indicates a cross section in the direction of the blood vessel diameter (that is, the minor axis direction which is the diameter direction) when the blood vessel is regarded as having a substantially cylindrical shape. Further, the direction perpendicular to the minor axis direction is defined as the major axis direction. Further, the PC 50 calculates the measurement length of the blood vessel based on the position information of the blood vessel endoscope 100 sequentially input from the auto pullback device 80. The PC 50 generates a three-dimensional image of a blood vessel based on a plurality of short-axis cross-sectional blood vessel images recorded in the storage 55, and cuts the three-dimensional image of the blood vessel along the long axis direction of the blood vessel to lengthen the blood vessel. Generates a blood vessel image of the axial cross section. Similarly, the major axis cross section indicates a cross section in the longitudinal direction (longitudinal direction) of a blood vessel when the blood vessel is regarded as having a substantially cylindrical shape. The PC 50 displays the blood vessel image of the long axis cross section on the monitor 70 together with the measurement information of the blood vessel diameter and the blood vessel image of the short axis cross section.

FIG. 6 is a diagram showing an example of a screen displayed on the monitor 70. On the screen GM displayed on the monitor 70, for example, the measurement result of the blood vessel diameter, the blood vessel image GZ5 having a short axis cross section, and the blood vessel image GZ6 having a long axis cross section are displayed at the same time. Information on the measurement result of the blood vessel diameter includes, for example, imaging date: October 1, 2019, patient ID: ABC100, measured value, blood vessel inner diameter: 2.5 mm, blood vessel length: 30 mm, stenosis inner diameter: 2.0 mm. An indicator i1 representing a blood vessel diameter (blood vessel inner diameter): 2.5 mm is superimposed on the blood vessel image GZ5 having a short axis cross section together with a numerical value. An indicator i2 indicating the measurement length is superimposed on the blood vessel image GZ6 having a long axis cross section together with a numerical value having a length of 13 mm.

In the blood vessel endoscopy system 5 according to the first embodiment, the repeater 20 uses the blood vessel image data signal provided (transmitted) from the blood vessel endoscope 100 and the blood vessel provided (transmitted) from the auto pullback device 80. The data of the position information and the speed information of the endoscope 100 are combined. The repeater 20 transmits the combined data to the PC 50 of the image console 90. The PC 50 is imaged at the same constant speed using the position information of the blood vessel endoscope 100 and the speed information of the imaging position of the blood vessel endoscope 100 from a plurality of blood vessel images, and the imaging positions are different. Create a constant velocity moving image composed by sequentially selecting multiple blood vessel images in chronological order. The PC 50 calculates the blood vessel diameter using at least two blood vessel images constituting the constant velocity moving image (see FIG. 3). In the calculation of the blood vessel diameter, the feature points arranged in a circle are detected from the measurement points reflected in the blood vessel image, and the displacement amount of the feature points between the two blood vessel images is used. By using two (constant velocity) blood vessel images taken at the same constant speed, the variation in the displacement amount of the feature points is suppressed, and the calculation accuracy of the blood vessel diameter is improved. Therefore, a user such as a doctor can appropriately select the size of the stent when inserting the stent into the blood vessel portion where the blood vessel is narrowed. Further, the repeater 20 uses the data signal of the blood vessel image captured by the blood vessel endoscope camera 10 and the blood vessel endoscope 100 when the auto pullback device 80 pulls back the catheter 200 (in other words, the blood vessel endoscope 100). Since the position information and the speed information data of the above are acquired at the synchronized timing and these data are combined, it is easy to associate the blood vessel image with the position information and the speed information of the blood vessel endoscope 100.

As described above, in the blood vessel endoscopy system 5, the blood vessel endoscope 100 is inserted into the blood vessel of the subject at the distal end side and pulled at a constant speed by the auto pullback device 80 at the proximal end side to image the blood vessel of the subject. A possible vascular endoscopic camera 10 is mounted on the tip side. The repeater 20 inputs and combines the data of the blood vessel image captured by the blood vessel endoscope 100 and the data of the position information and the speed information of the blood vessel endoscope 100 sent from the auto pullback device 80. The PC 50 stores in the storage 55 data in which the blood vessel image sent from the repeater 20 and the position information and the velocity information of the blood vessel endoscope 100 are combined. Further, the PC 50 generates a constant velocity moving image of the blood vessel image using the data obtained by combining the blood vessel image stored in the storage 55 with the position information and the velocity information of the blood vessel endoscope 100, and constitutes the constant velocity moving image. The blood vessel diameter of a blood vessel is calculated based on a plurality of blood vessel images.

As a result, the vascular endoscopy system 5 uses the image captured by the vascular endoscope without interrupting the smooth progress of medical practice by a user such as a doctor, and the blood vessel, which is an observation site of a subject such as a patient, is used. The blood vessel diameter can be measured with high accuracy. Therefore, the vascular endoscopy system 5 can assist a user such as a doctor in selecting a stent having an appropriate diameter to be inserted into a subject. In particular, since the repeater 20 combines the blood vessel image data with the position information and velocity information data of the blood vessel endoscope 100, the PC 50 acquires the position information at the timing synchronized (matched) with the blood vessel image data acquisition. And, using the data of the speed information, a constant velocity moving image of the blood vessel image can be generated in real time, and the blood vessel diameter of the blood vessel which is the observation site of the subject can be calculated with high accuracy.

Further, the PC 50 superimposes the measurement result of the blood vessel diameter of the blood vessel, the indicator i1 indicating the blood vessel diameter, and the numerical value (that is, the numerical value of the blood vessel diameter) on the blood vessel image GZ1 and displays it on the monitor 70. As a result, a user such as a doctor can visually grasp the blood vessel diameter in an easy-to-understand manner.

Further, the PC 50 uses the input vascular images GZ5 of a plurality of short-axis cross sections to generate a vascular image GZ6 (an example of a long-axis cross-section image) of the long-axis cross section of the blood vessel, and together with the vascular image GZ5 of the short-axis cross section It is displayed on the monitor 70 in association with the blood vessel image GZ6 having a long-axis cross section. This allows the user to grasp the blood vessel diameter along the long axis direction of the blood vessel. Therefore, the user can grasp the diameter of the blood vessel even when the blood vessel is distorted in the long axis direction, and can select an appropriate stent in consideration of the distortion in the long axis direction of the blood vessel.

Further, the PC 50 calculates the measurement length of the blood vessel based on the data of the position information and the speed information of the blood vessel endoscope 100 sequentially input from the auto pullback device 80, and sets the indicator i2 indicating the calculation result of the measurement length as the numerical value. It is also superimposed and displayed on the blood vessel image GZ6 of the long axis cross section. This allows the user to visually and accurately grasp the length of the observation site, for example, the stenosis of the blood vessel.

Further, the PC 50 selects two blood vessel images corresponding to the position information of the vascular endoscope 100 from a plurality of blood vessel images constituting the constant velocity moving image, and for each of these two blood vessel images. The disappearance points are estimated from the movement directions of the plurality of feature points arranged in a circular shape, and the disappearance points based on the movement of the angioscope 100 by the auto pullback device 80 are selected from the plurality of feature points. The effective feature points having a vector in the direction are extracted, the three-dimensional coordinates of the effective feature points are calculated using the movement change amount of the effective feature points corresponding to the movement distance of the

angioscope

100, and 3 of the effective feature points are calculated. The diameter of the blood vessel is calculated based on the three-dimensional coordinates. Thereby, the blood vessel endoscopy system 5 can accurately measure the blood vessel diameter by using the feature points arranged on the blood vessel image and effective for measuring the blood vessel diameter.

Further, the PC 50 selects a plurality of blood vessel images having different position information of the blood vessel endoscope 100 and the same velocity information from a plurality of blood vessel images stored in the storage 55, and a plurality of selected blood vessel images. To generate a constant velocity moving image using. Thereby, the blood vessel endoscopy system 5 can calculate the blood vessel diameter with high accuracy by using at least two blood vessel images having the same velocity information, which constitute a constant velocity moving image. In addition, since the blood vessel diameter is calculated along the long axis direction of the blood vessel in which the constant velocity moving image is created, the user can accurately grasp the blood vessel diameter along the long axis direction of the blood vessel, and the stent to be inserted into the blood vessel. The diameter of can be selected appropriately.

The auto pullback device 80 is operated in a manual mode at the discretion of a user such as a doctor, in addition to the automatic mode in which the catheter 200 (that is, the vascular endoscope 100) is pulled at a constant speed described above (in other words, the catheter 200 (in other words, the catheter 200 (in other words)). That is, the angioscope 100) can be pulled). In the manual mode, the auto-pullback device 80 measures at regular intervals when the catheter 200 (that is, the angioscope 100) is pulled back at a variable speed within the measurement length range (because it is manual, it does not reach a constant speed). The position information and acceleration information of the angioscope 100 (that is, the acceleration information obtained by the time change of the variable speed at regular time intervals) are acquired. The auto pullback device 80 outputs the data of the position information and the distance information of the blood vessel endoscope 100 to the repeater 20. The PC 50 may display the distance information (for example, the measurement length of the blood vessel) on the monitor 70 in addition to the position information of the blood vessel endoscope 100.

Further, in the manual mode, the auto pullback device 80 pulls back the catheter 200 (that is, the vascular endoscope 100) at a variable speed within the measurement length range (because it is manual, it does not reach a constant speed) at regular intervals. The position information, acceleration information, and velocity information of the vascular endoscope 100 measured in the above are acquired, and the data of the imaging position information in which the position information of the vascular endoscope 100 and the frame position are combined is output to the repeater 20. It may be displayed on the monitor 70.

(Embodiment 2)
In the first embodiment, the repeater 20 is inside the blood vessel when the data of the blood vessel image captured by the blood vessel endoscope camera 10 and the auto pullback device 80 pull back the catheter 200 (in other words, the blood vessel endoscope 100). Data of position information and speed information of the endoscope 100 were acquired, and these synchronized data (that is, data having the same acquisition timing) were combined. In the second embodiment, the PC 50A uses the blood vessel image data captured by the blood vessel endoscope camera 10, and the auto pullback device 80 pulls back the catheter 200 (in other words, the blood vessel endoscope 100). An example of inputting 100 position information and speed information data separately will be described. In this case, the blood vessel image data and the position information and speed information data of the blood vessel endoscope 100 are not always synchronized data (that is, data whose acquisition timings match) depending on the input timing to the PC 50. .. Therefore, in the second embodiment, the PC50A adds a predetermined delay time to the data acquisition time of the position information and the speed information of the angioscope 100, for example, when it takes time to acquire the data of the blood vessel image. This makes it possible to associate these data (that is, to generate the synchronized data described above). Here, the PC50A assigns a time stamp indicating the time input to the PC50A to each of the input blood vessel image data and the position information and speed information data of the blood vessel endoscope 100, and these data are attached. Correspond.

FIG. 7 is a diagram showing a configuration example of the blood vessel endoscopy system 5A according to the second embodiment. The configuration of the vascular endoscopy system 5A according to the second embodiment has substantially the same configuration as the vascular endoscopy system 5 according to the first embodiment. In the description of the configuration of the vascular endoscopy system 5A according to the second embodiment, the same reference numerals are given to the same configurations as the configurations of the vascular endoscopy system 5 according to the first embodiment to simplify the description. It will be omitted and different contents will be explained.

In the vascular endoscope system 5A, the auto pullback device 80 transfers the position information and speed information data of the vascular endoscope 100 when pulling back the catheter 200 (in other words, the vascular endoscope 100) to the PC 50A of the image console 90. Send. The data of the blood vessel image captured by the blood vessel endoscopy camera 10 is sent to the repeater 20A as in the first embodiment. The repeater 20A transmits the blood vessel image data to the camera control unit 30 of the image console 90.

The processor 51 of the PC 50A adds a time stamp to the data of the position information and the speed information of the vascular endoscope 100 received from the auto pullback device 80. This time stamp indicates, for example, the time when the processor 51 stores the data received from the auto pullback device 80 in the storage 55. Further, the processor 51 of the PC 50A adds a time stamp to the blood vessel image data input from the blood vessel endoscope 100 via the repeater 20A and the camera control unit 30. This time stamp indicates, for example, the time when the processor 51 stores the data received from the camera control unit 30 in the storage 55.

When the time difference between the acquisition of the blood vessel image data and the acquisition of the position information and the speed information data of the blood vessel endoscope 100 is known in advance, the processor 51 of the PC50A adds a delay time to the acquisition time of one of the data or adds a delay time. By subtracting, these data are associated. Further, when there is no particular time difference between the acquisition of the blood vessel image data and the acquisition of the position information and speed information data of the angioscope 100, and these data are simply input to the PC 50A at different timings, the processor of the PC 50A 51 associates these data having substantially the same time stamps. The processor 51 of the PC 50A sequentially records the data of the associated blood vessel image and the data of the position information and the speed information of the blood vessel endoscope in the storage 55 for each position information of the different blood vessel endoscope 100.

FIG. 8 is a flowchart showing an example of the operation procedure of the blood vessel endoscopy system 5A according to the second embodiment. In the description of FIG. 8, the same process as that shown in FIG. 2 is given the same step number to simplify or omit the description, and different contents will be described.

In FIG. 8, when the user presses the start switch of the operation unit 54 of the image console 90, the blood vessel endoscopy system 5A is started (S1). When the vascular endoscope system 5A is activated, the auto pullback device 80, the repeater 20A, the vascular endoscope 100, and the image console 90 each start operating.

The user sets the start position and the end position of the measurement for pulling back the vascular endoscope 100 inserted into the blood vessel for the auto pullback device 80.

The auto pullback device 80 measures the position information and the velocity information measured when the inside of the blood vessel is imaged by the blood vessel endoscope 100 while pulling back the blood vessel endoscope 100 within the range (measurement length) of the start position and the end position of the measurement. (S11). The auto pullback device 80 outputs position information and speed information data to the image console 90 (S12A). The processor 51 of the PC 50A adds a time stamp to the data of the position information and the speed information of the blood vessel endoscope 100. This time stamp indicates, for example, the time when the processor 51 stores the data received from the auto pullback device 80 in the storage 55.

On the other hand, the blood vessel endoscope 100 acquires the data of the blood vessel image captured by the blood vessel endoscope camera 10 (S21). The blood vessel endoscope 100 outputs the blood vessel image data to the repeater 20 (S22).

On the other hand, the repeater 20 inputs blood vessel image data from the blood vessel endoscope 100 (S31). The repeater 20 transmits the blood vessel image data to the image console 90 (S34A). In the image console 90, the camera control unit 30 receives the blood vessel image data (S41A). The processor 51 of the PC 50A adds a time stamp to the blood vessel image data received by the camera control unit 30. As described above, this time stamp indicates, for example, the time when the processor 51 stores the data received from the camera control unit 30 in the storage 55.

On the other hand, the PC50A associates the blood vessel image data with the position information and velocity information data of the blood vessel endoscope 100 based on the time stamps of both. This association is performed, for example, by including the identification information of the other data file in the metadata of one data file. The PC 50A stores the associated blood vessel image data and the position information and velocity information data of the blood vessel endoscope 100 in the storage 55. The PC50A creates a constant velocity moving image based on the associated blood vessel image data and the position information and velocity information data of the blood vessel endoscope 100 (S42A). The creation of the constant velocity moving image is the same as that of the first embodiment.

The PC50A calculates the blood vessel diameter using two blood vessel images constituting the constant velocity moving image (S43). Further, the PC 50A separates the associated blood vessel image data and the position information and speed information data of the blood vessel endoscope 100 in order to separately display the blood vessel image and the position information and speed information of the blood vessel endoscope 100. (S44). The calculation of the blood vessel diameter is the same as in the first embodiment. After that, the PC 50A displays the blood vessel image, the position information and the velocity information, and the blood vessel diameter on the monitor 70 (S45). This display mode is the same as that of the first embodiment.

In the blood vessel endoscopy system 5A according to the second embodiment, the PC 50A associates the blood vessel image data with the position information and speed information data of the blood vessel endoscope 100 by software processing. Therefore, the repeater 20A only needs to transfer the blood vessel image data acquired from the blood vessel endoscope 100 to the image console 90, the configuration of the repeater 20A can be simplified, and the hardware such as electronic components can be reduced.

As described above, in the blood vessel endoscopy system 5A, the vascular endoscope 100 can image the blood vessel of the subject by inserting the tip side into the blood vessel of the subject and pulling the proximal end side by the auto pullback device 80 at a constant speed. An vascular endoscopic camera 10 (an example of an image sensor) is mounted on the tip side. The PC 50A inputs and associates the data of the blood vessel image captured by the blood vessel endoscope 100 with the data of the position information and the speed information of the blood vessel endoscope 100 sent from the auto pullback device 80 and stores them in the storage 55. .. The PC50A generates a constant velocity moving image of the blood vessel image based on the difference between the input timing of the blood vessel image data stored in the storage 55 and the input timing of the position information and the speed information data of the blood vessel endoscope 100, and the like. The blood vessel diameter is measured based on a plurality of blood vessel images constituting a rapid moving image.

As a result, the blood vessel endoscopy system 5A uses the image captured by the blood vessel endoscope without interrupting the smooth progress of medical practice by a user such as a doctor, and has a blood vessel diameter which is an observation site of a subject such as a patient. Can be measured with high accuracy. Therefore, the vascular endoscopy system 5A can assist in the selection of a stent of appropriate diameter to be inserted into the subject. In particular, since the PC50A associates the blood vessel image data with the endoscopic position information and velocity information data, the repeater can be easily configured.

Further, the PC50A superimposes the measurement result of the blood vessel diameter of the blood vessel, the indicator i1 indicating the blood vessel diameter, and the numerical value (that is, the numerical value of the blood vessel diameter) on the blood vessel image GZ1 and displays it on the monitor 70. As a result, a user such as a doctor can visually grasp the blood vessel diameter in an easy-to-understand manner.

Further, the PC50A generates a blood vessel image GZ6 (an example of a long axis cross section image) of a long axis cross section of a blood vessel by using a plurality of input blood vessel images GZ5 of a short axis cross section, and together with a blood vessel image GZ5 of the short axis cross section. It is displayed on the monitor 70 in association with the blood vessel image GZ6 having a long-axis cross section. This allows the user to grasp the blood vessel diameter along the long axis direction of the blood vessel. Therefore, the user can grasp the diameter of the blood vessel even when the blood vessel is distorted in the long axis direction, and can select an appropriate stent in consideration of the distortion in the long axis direction of the blood vessel.

Further, the PC 50A calculates the measurement length of the blood vessel based on the data of the position information and the speed information of the blood vessel endoscope 100 sequentially input from the auto pullback device 80, and sets an indicator i2 indicating the calculation result of the measurement length as the numerical value. It is also superimposed and displayed on the blood vessel image GZ6 of the long axis cross section. This allows the user to visually and accurately grasp the length of the observation site, for example, the stenosis of the blood vessel.

Further, the PC50A selects two blood vessel images corresponding to the position information of the angioscope 100 from a plurality of blood vessel images constituting the constant velocity moving image, and for each of these two blood vessel images. The disappearance points are estimated from the movement directions of the plurality of feature points arranged in a circular shape, and the disappearance points based on the movement of the angioscope 100 by the auto pullback device 80 are selected from the plurality of feature points. The effective feature points having a vector in the direction are extracted, the three-dimensional coordinates of the effective feature points are calculated using the movement change amount of the effective feature points corresponding to the movement distance of the

angioscope

Further, the PC 50A selects a plurality of blood vessel images having different position information of the blood vessel endoscope 100 and the same velocity information from a plurality of blood vessel images stored in the storage 55, and a plurality of selected blood vessel images. To generate a constant velocity moving image using. Thereby, the blood vessel endoscopy system 5 can calculate the blood vessel diameter with high accuracy by using at least two blood vessel images having the same velocity information, which constitute a constant velocity moving image. In addition, since the blood vessel diameter is calculated along the long axis direction of the blood vessel in which the constant velocity moving image is created, the user can accurately grasp the blood vessel diameter along the long axis direction of the blood vessel, and the stent to be inserted into the blood vessel. The diameter of can be selected appropriately.

(Embodiment 3)
In the first and second embodiments, the

PCs

50 and 50A create a constant velocity moving image based on the combined data (that is, the data in which the blood vessel image and the position information and the speed information of the blood vessel endoscope 100 are combined), and the like. The blood vessel diameter in the blood vessel, which is the observation site of the subject, was calculated using the data of at least two blood vessel images constituting the rapid moving image. In other words, the

PCs

50 and 50A require data of at least two blood vessel images constituting a constant velocity moving image in order to calculate the blood vessel diameter in the blood vessel. Therefore, in the third embodiment, the PC (for example, PC50) is inserted into the subject with the combined data (that is, the data in which the blood vessel image and the position information and the velocity information of the blood vessel endoscope 100 are combined). Using the size of the guide wire in the width direction and the learning model of artificial intelligence (AI) formed (constructed) by the learning process in advance, the blood vessel diameter in the blood vessel, which is the observation site of the subject, is obtained from one blood vessel image. An example of calculating is described.

The configuration of the vascular endoscopy system according to the third embodiment is the same as that of the vascular endoscopy system 5 according to the first embodiment. Therefore, in the description of the configuration of the vascular endoscopy system 5 according to the third embodiment, the same reference numerals are given to the same configurations as the configuration of the vascular endoscopy system 5 according to the first embodiment. It will be simplified or omitted, and different contents will be explained. The configuration of the vascular endoscopy system 5 according to the third embodiment may be the same as that of the vascular endoscopy system 5A according to the second embodiment.

In the vascular endoscopy system 5 according to the third embodiment, information on the width direction size (in other words, thickness) of the guide wire 150 inserted into the subject is previously stored in the memory 52 or the storage 55 of the PC 50. It has been saved (so-called preset). The size of the guide wire 150 is, for example, 0.36 mm (0.14 inch). Since the size of the guide wire 150 differs depending on the type of guide wire, it does not have to be a fixed value. For example, the size of the guide wire 150 can be arbitrarily set by user operation according to the type of guide wire used. The same applies to the first and second embodiments described above. As a result, the PC 50 can flexibly adjust the size of the guide wire according to the type of guide wire used in surgery, examination, or the like, and can support the calculation of an appropriate blood vessel diameter.

The PC 50 is formed (constructed) through a predetermined learning process before calculating the blood vessel diameter in an actual (that is, raw) blood vessel image imaged by the blood vessel endoscope 100 using the blood vessel endoscopy system 5. The trained learning model may be stored in the memory 52 or the storage 55. The predetermined learning process may be executed by the PC 50 or by an external device (not shown). When the learning process is executed by the external device, the learning model created as a result of the learning process from the external device is input to the PC 50 and stored in the memory 52 or the storage 55.

The PC 50 forms an AI (for example, a neural network) based on a learning model stored in the memory 52 or the storage 55, and the AI intersects the inner wall of the blood vessel which is shown to be in focus in the blood vessel image or the pseudo blood vessel image. A reference point (see below) indicating a position in the vicinity of the (overlapping) guide wire 150 and a plurality of detection points (see below) capable of forming the same shape as the shape of the blood vessel (for example, a circular shape) including the reference point. Identify. The pseudo blood vessel image is not an actual blood vessel image, but is an image in which at least the same subject (that is, blood vessel) as the blood vessel image is arranged and can be regarded as an image substantially equivalent to the actual blood vessel image. Therefore, the learning process shows a learning process for being able to estimate (detect) each of the reference point and the detection point in each blood vessel image or pseudo blood vessel image by using a plurality of blood vessel images or pseudo blood vessel images. .. The PC 50 does not necessarily have to use the AI described above when specifying the reference point and the detection point described above. For example, the PC 50 performs predetermined image processing (for example, the same as the shape of a blood vessel) on one blood vessel image whose blood vessel diameter is a target in the blood vessel image input from the repeater 20 and stored in the memory 52 or the storage 55. A reference point and a detection point may be specified by performing a search for a plurality of feature points constituting the shape (see FIG. 11).

The PC 50 is a blood vessel of a subject based on the size of the guide wire 150 in the width direction and one blood vessel image corresponding to the position information and the velocity information of the blood vessel endoscope 100 stored in the memory 52 or the storage 55. Calculate the blood vessel diameter of. The detailed operation procedure for calculating the blood vessel diameter will be described later.

FIG. 9 is a flowchart showing an example of the operation procedure of the vascular endoscopy system 5 according to the third embodiment. In the description of FIG. 9, for the same process as the process shown in FIG. 2, the same step number is assigned to simplify or omit the description, and different contents will be described.

In FIG. 9, in the image console 90, the camera control unit 30 receives the combined data generated by the repeater 20 (that is, the combined data of the blood vessel image and the position information and the speed information of the blood vessel endoscope 100). (S41). The PC 50 stores this combined data in the storage 55. The PC 50 reads out information on the size of the guide wire 150 in the width direction stored in the memory 52 or the storage 55 (S42B). The PC 50 has information on the size of the guide wire 150 read in step S42B in the width direction and one blood vessel image (that is, position information of the blood vessel endoscope 100) included in the combined data received in step S41. The blood vessel diameter of the blood vessel of the subject is calculated based on (one blood vessel image corresponding to the velocity information) (S43B). Details of step S43B will be described in detail with reference to FIGS. 10 to 17.

The PC 50 separates the combined blood vessel image and the position information and speed information data of the blood vessel endoscope 100 in order to display the blood vessel image and the position information and speed information of the blood vessel endoscope 100 separately (S44). ). The PC 50 displays the blood vessel image, the position information and velocity information of the blood vessel endoscope 100, and the blood vessel diameter on the monitor 70 (S45). The position information and the speed information may be displayed at the same time or may be displayed individually. After this, the vascular endoscopy system 5 ends the operation shown in FIG.

Next, the operation procedure of the blood vessel diameter calculation process according to the third embodiment will be described with reference to FIGS. 10 to 17. This blood vessel diameter calculation process is mainly executed by the processor 51 of the PC 50. In the third embodiment, four ways of calculating the blood vessel diameter will be described. The PC 50 may calculate the blood vessel diameter by using any of the four calculation processes. In the description of FIGS. 12 to 17, the same elements as those of FIGS. 10 or 11 are given the same reference numerals to simplify or omit the description, and different contents will be described.

[First calculation process]
FIG. 10 is a diagram showing a schematic example of the first calculation process of the blood vessel diameter according to the third embodiment. FIG. 11 is a flowchart showing an example of an operation procedure of the first calculation process of the blood vessel diameter in step S43B of FIG. The process of FIG. 11 is executed each time a single blood vessel image (so-called frame) constituting the combined data (see above) stored in the memory 52 or the storage 55 of the PC 50 is acquired by the processor 51.

FIG. 10 shows the blood vessel image IMG1 of the actual (that is, raw) blood vessel VSL1 in the subject imaged by the blood vessel endoscope 100. Since the guide wire 150 is inserted into the subject in the a direction, the guide wire 150 is also reflected in the blood vessel image IMG1. The size G1 (in other words, the thickness) of the guide wire 150 in the width direction is, for example, 0.36 mm (0.14 inch). That is, since the blood vessel image IMG1 is imaged so as to focus on the inner wall of the blood vessel VSL1 in the substantially central portion of the entire blood vessel image IMG1 which is the subject, the inner wall of the blood vessel VSL1 in focus (simply "inner wall of blood vessel"". The length of the line segment WD1 straddling the guide wire 150 on the blood vessel image IMG1 from the position where the guide wire 150 intersects (overlaps) (reference point W1) is 0.36 mm (0.14 inch).

In FIG. 11, when the processor 51 of the PC 50 acquires the actual (that is, raw) blood vessel image IMG1 imaged by the blood vessel endoscope 100, it is in the vicinity of the guide wire 150 in the blood vessel image IMG1 and in the blood vessel image IMG1. The reference point W1 corresponding to the inner wall of the blood vessel that is in focus is determined and positioned (S61). The processor 51 of the PC50 has a plurality of detection points W2, W3, W4, W5, which are estimated to form the same shape (that is, a circular shape) as the shape of the blood vessel VSL1 including the reference point W1 by AI based on the above-mentioned learning model. W6, W7, and W8 are specified and positioned (S62). For example, each of the reference point W1 and the plurality of detection points W2 to W8 constitutes an in-focus blood vessel inner wall in the blood vessel image IMG1 and is arranged at equal intervals.

The processor 51 of the PC 50 fits the shape FT1 (for example, a shape imitating the shape of the inner wall of the blood vessel in focus in the blood vessel image IMG1) including all of the reference point W1 and the plurality of detection points W2 to W8 (S63). For this shape fitting, a so-called known fitting process such as an elliptical fitting or a plaque type fitting may be executed.

The processor 51 of the PC 50 has the same size G1 of the guide wire 150 in focus in the blood vessel image IMG 1 and the distance from the blood vessel endoscopy camera 10 (in other words, the shape FT1 fitted in step S63 (in other words, the shape FT1 fitted in step S63). The blood vessel diameter Dm is calculated based on the length from the reference point W1 (existing on the circular shape) to the adjacent detection point (for example, the detection point W2) (S64).

Specifically, the processor 51 of the PC 50 utilizes the fact that the length of the line segment WD1 from the reference point W1 in the blood vessel image IMG1 is the size G1 (in other words, the thickness) in the width direction of the guide wire 150. , The length of the arc from the reference point W1 to the detection point W2 in the blood vessel image IMG1 is calculated. Further, the processor 51 of the PC 50 calculates the circumference length of the fitting shape (that is, the shape FT1 indicating a circle) in step S63 by using the calculation result of the arc length from the reference point W1 to the detection point W2. The blood vessel diameter Dm (that is, the length from the detection point W2 to the detection point W6) is calculated by dividing the calculated circumference length by the circumference ratio π.

[Second calculation process]
FIG. 12 is a diagram showing a schematic example of the second calculation process of the blood vessel diameter according to the third embodiment. FIG. 13 is a flowchart showing an example of an operation procedure of the second calculation process of the blood vessel diameter in step S43B of FIG. The process of FIG. 13 is executed every time one blood vessel image (so-called frame) constituting the combined data (see above) stored in the memory 52 or the storage 55 of the PC 50 is acquired by the processor 51.

The blood vessel image IMG2 of FIG. 12 may be a blood vessel image of an actual (that is, raw) blood vessel VSL1 in the subject imaged by the blood vessel endoscope 100, or a blood vessel image prepared in advance for the above-mentioned learning process. It may be a pseudo blood vessel image. Further, when the blood vessel image IMG2 is a pseudo blood vessel image, the guide wire 150 is arranged so as to be reflected in the blood vessel image IMG2 on the assumption that the guide wire 150 is inserted in the subject in the a direction. .. The size G1 (in other words, the thickness) of the guide wire 150 in the width direction is, for example, 0.36 mm (0.14 inch). That is, since the blood vessel image IMG2 is imaged so as to focus on the blood vessel VSL1 in the substantially central portion of the entire blood vessel image IMG2, the in-focus blood vessel inner wall and the guide wire 150 intersect (overlap). ) The length of the line segment WD1 on the blood vessel image IMG2 is 0.36 mm (0.14 inch). The reference point W9, like the reference point W1 in FIG. 11, is a portion of the inner wall of the blood vessel that is in focus in the blood vessel image IMG2, and indicates an arbitrary position of the guide wire 150 or its vicinity.

In FIG. 13, the processor 51 of the PC 50 uses a plurality of blood vessel images IMG2 (which may be a pseudo blood vessel image) for AI (for example, monocular estimation by known supervised learning or automatic learning monocular estimation), and the reference point W9. A learning model for positioning is formed by a learning process (S60). The process of step S60 may be executed by an external device different from the PC 50, and the learning model obtained by the learning process may be input to the PC 50 and stored in the storage 55.

Then, the processor 51 of the PC 50 is the blood vessel of the actual (that is, raw) blood vessel VSL1 in the subject imaged by the blood vessel endoscope 100 by the AI based on the learning model formed through the learning process of step S60A. A reference point W9 corresponding to the inner wall of the blood vessel that is in the vicinity of the guide wire 150 in the image and is in focus in the blood vessel image is determined and positioned (S61A). Since the processing after step S61A is the same as that in FIG. 12, the description thereof will be omitted.

[Third calculation process]
FIG. 14 is a diagram showing a schematic example of a third calculation process of the blood vessel diameter according to the third embodiment. FIG. 15 is a flowchart showing an example of an operation procedure of the third calculation process of the blood vessel diameter in step S43B of FIG. The process of FIG. 15 is executed each time a single blood vessel image (so-called frame) constituting the combined data (see above) stored in the memory 52 or the storage 55 of the PC 50 is acquired by the processor 51.

The blood vessel image IMG3 of FIG. 14 may be a blood vessel image or a pseudo blood vessel image prepared in advance for the above-mentioned learning process. Further, when the blood vessel image IMG3 is a pseudo blood vessel image, the guide wire 150 is arranged so as to be reflected in the blood vessel image IMG2 on the assumption that the guide wire 150 is inserted in the subject in the a direction. .. Further, in the blood vessel image IMG3, scales SCL1 (that is, a ruler) having scales at equal intervals are arranged symmetrically with the guide wire 150. The size G1 (in other words, the thickness) of the guide wire 150 in the width direction is, for example, 0.36 mm (0.14 inch). That is, since the blood vessel image IMG3 is imaged so as to focus on the blood vessel VSL1 in the substantially central portion of the entire blood vessel image IMG3, which is the subject, the line segment WD1 in which the inner wall of the blood vessel in focus and the guide wire 150 overlap. The length on the blood vessel image IMG3 is 0.36 mm (0.14 inch). The reference point W10 is a portion of the blood vessel inner wall that is in focus in the blood vessel image IMG3, and indicates a position where the blood vessel inner wall and the scale SCL1 intersect (overlap).

In the learning process, the position of the inner wall of the blood vessel in focus (for example, the position of the reference point W10) and the value of the scale from the starting point B1 of the scale SCL1 are stored in association with each other for each blood vessel image IMG3. Further, similarly to the reference point W1 in FIG. 11, an arbitrary position on the inner wall of the blood vessel in focus in the blood vessel image IMG3 and at or near the guide wire 150 may be used as the reference point W11.

In FIG. 15, the processor 51 of the PC 50 determines the position of the inner wall of the blood vessel in focus in the blood vessel image IMG3 (that is, in focus in the blood vessel image IMG3) for each blood vessel image IMG3 described with reference to FIG. The reference is performed by performing learning processing using the association between the part of the inner wall of the blood vessel and the position where the inner wall of the blood vessel and the scale SCL1 intersect (overlap) and the value of the scale from the starting point B1 of the scale SCL1. A learning model for positioning the point W10 is formed (S60B). The process of step S60B may be executed by an external device different from the PC 50, and the learning model obtained by the learning process may be input to the PC 50 and stored in the storage 55.

Then, the processor 51 of the PC 50 is a blood vessel of the actual (that is, raw) blood vessel VSL1 in the subject imaged by the blood vessel endoscope 100 by the AI based on the learning model formed through the learning process of step S60B. A reference point W10 corresponding to the in-focus blood vessel inner wall in the image, or a reference point W11 corresponding to the blood vessel inner wall in the vicinity of the guide wire 150 in the blood vessel image and in focus in the blood vessel image is determined. Positioning (S61B). Since the processing after step S61B is the same as that in FIG. 12, the description thereof will be omitted.

[Fourth calculation process]
FIG. 16 is a diagram showing a schematic example of the fourth calculation process of the blood vessel diameter according to the third embodiment. FIG. 17 is a flowchart showing an example of an operation procedure of the fourth calculation process of the blood vessel diameter in step S43B of FIG. The process of FIG. 17 is executed each time a single blood vessel image (so-called frame) constituting the combined data (see above) stored in the memory 52 or the storage 55 of the PC 50 is acquired by the processor 51.

The blood vessel image IMG4 of FIG. 16 may be a blood vessel image of an actual (that is, raw) blood vessel VSL1 in the subject taken by the blood vessel endoscope 100, or a blood vessel image prepared in advance for the above-mentioned learning process. Alternatively, it may be a pseudo blood vessel image. Further, when the blood vessel image IMG4 is a pseudo blood vessel image, the guide wire 150A is arranged so as to be reflected in the blood vessel image IMG4 on the assumption that the guide wire 150A is inserted in the subject in the a direction. .. The guide wire 150A is graduated at equal intervals, for example, at arbitrary values of 0.01 mm to 1 mm. The size G1 (in other words, the thickness) of the guide wire 150A in the width direction is, for example, 0.36 mm (0.14 inch). That is, since the blood vessel image IMG4 is imaged so as to focus on the blood vessel VSL1 in the substantially central portion of the entire blood vessel image IMG4, which is the subject, the line segment WD1 in which the inner wall of the blood vessel in focus and the guide wire 150 overlap. The length on the blood vessel image IMG4 is 0.36 mm (0.14 inch). The reference point W12 is a portion of the inner wall of the blood vessel that is in focus in the blood vessel image IMG4, and indicates a position in the vicinity of the guide wire 150A.

In the learning process, for each blood vessel image IMG4, the coordinates on the blood vessel image IMG4 at the position where the position of the inner wall of the blood vessel in focus (for example, the position of the reference point W12) and the guide wire 150A intersect (overlap) and the guide wire. The value of the scale from the starting point of 150A (see FIG. 15) is stored in association with the value of the scale.

In FIG. 17, the processor 51 of the PC 50 determines the position of the inner wall of the blood vessel in focus in the blood vessel image IMG 4 (that is, in focus in the blood vessel image IMG 4) for each blood vessel image IMG 4 described with reference to FIG. Learning process using the association between the part of the inner wall of the blood vessel and the position where the inner wall of the blood vessel and the guide wire 150A intersect (overlap) and the value of the scale from the starting point of the guide wire 150A (see FIG. 15). To form a learning model for positioning the reference point W12 (S60C). The process of step S60C may be executed by an external device different from the PC 50, and the learning model obtained by the learning process may be input to the PC 50 and stored in the storage 55.

Then, the processor 51 of the PC 50 is a blood vessel of the actual (that is, raw) blood vessel VSL1 in the subject imaged by the blood vessel endoscope 100 by the AI based on the learning model formed through the learning process of step S60C. A reference point W12 corresponding to the in-focus blood vessel inner wall in the image is determined and positioned (S61C). After step S61C, the processes of steps S62 and S63 are executed. After step S63, the processor 51 of the PC 50 sets the scale from the size G1 of the in-focus guide wire 150 or the starting point (see FIG. 15) of the guide wire 150A in the blood vessel image captured by the blood vessel endoscope 100. The value (scale interval) and the distance from the blood vessel endoscopy camera 10 are the same (in other words, they exist on the shape FT1 (for example, a circular shape) fitted in step S63). The blood vessel diameter Dm is calculated based on the length up to (for example, detection point W2) (S64C).

As described above, in the blood vessel endoscopy system 5 according to the third embodiment, the blood vessel endoscope 100 is inserted into the blood vessel of the subject into which the guide wire 150 is inserted in advance, and the driving device (for example, the auto pullback device 80) is inserted. It is possible to image the blood vessel of the subject while being pulled through. The repeater 20 inputs the blood vessel image captured by the blood vessel endoscope 100 and the position information and speed information of the blood vessel endoscope 100 (specifically, the blood vessel endoscope camera 10) sent from the driving device. And combine. The PC 50 associates the blood vessel image sent from the repeater 20 with the position information and speed information of the blood vessel endoscope 100 and stores them in the memory 52 or the storage 55, and the guide wire 150 stored in the memory 52 or the storage 55. The blood vessel diameter of the blood vessel is calculated based on the size G1 in the width direction and one blood vessel image (for example, blood vessel image IMG1) corresponding to the position information and the speed information of the blood vessel endoscope 100.

As a result, the blood vessel endoscopy system 5 uses the image captured by the blood vessel endoscope without interrupting the smooth progress of medical practice by a user such as a doctor, and the blood vessel, which is an observation site of a subject such as a patient, is used. The blood vessel diameter can be measured simply and with high accuracy using a single blood vessel image. Therefore, the vascular endoscopy system 5 can assist a user such as a doctor in selecting a stent having an appropriate diameter to be inserted into a subject. In particular, since the repeater 20 combines the blood vessel image data with the position information and speed information data of the blood vessel endoscope 100, the PC 50 acquires the position information at the timing synchronized (matched) with the blood vessel image data acquisition. And, using the data of the speed information, the blood vessel diameter of the blood vessel which is the observation site of the subject can be calculated simply and with high accuracy by using one blood vessel image.

Further, the size G1 in the width direction of the guide wire 150 is a value that can be changed according to the user operation. This makes it possible to flexibly calculate the blood vessel diameter according to the type of guide wire used during surgery or examination.

Further, the PC 50 has a reference point W1 indicating the position of the inner wall of the blood vessel in the vicinity of the guide wire 150 in one blood vessel image, and a plurality of detection points indicating a position capable of forming the same shape as the shape of the blood vessel including the reference point W1. W2 to W8 are specified, and the shape of the blood vessel including the reference point W1 and the plurality of detection points W2 to W8 is fitted. The PC 50 calculates the blood vessel diameter of the blood vessel based on the fitting result of the shape of the blood vessel, the length from the reference point W1 to any detection point, and the size G1 in the width direction of the guide wire 150. As a result, the blood vessel endoscopy system 5 utilizes the size G1 (in other words, the thickness) in the width direction of the guide wire 150 inserted into the subject, and the blood vessel diameter of the blood vessel which is the observation site of the subject. Can be calculated simply and with high accuracy.

Further, the PC 50 stores a learning model capable of detecting a reference point W9 indicating the position of the inner wall of the blood vessel in the vicinity of the guide wire 150 in the blood vessel image in which the guide wire 150 inserted into the subject is reflected through the learning process. Save in storage 55. The PC50 has a reference point W9 in one blood vessel image and a plurality of detection points W2 to indicate positions capable of forming the same shape as the shape of the blood vessel including the reference point W9 by AI (artificial intelligence) based on the learning model. W8 is identified and the shape of the blood vessel including the reference point W9 and the plurality of detection points W2 to W8 is fitted. The PC 50 calculates the blood vessel diameter of the blood vessel based on the fitting result of the shape of the blood vessel, the length from the reference point W9 to any detection point, and the size G1 in the width direction of the guide wire 150. As a result, the blood vessel endoscopy system 5 can easily and accurately position the reference point W9 in the blood vessel image by using the AI for positioning the reference point already formed by the learning process, and the inside of the subject can be positioned. By using the size G1 (in other words, the thickness) in the width direction of the guide wire 150 inserted into the guide wire 150, the blood vessel diameter of the blood vessel which is the observation site of the subject can be calculated simply and with high accuracy.

Further, the PC 50 is a reference point indicating the position of the inner wall of the blood vessel in the vicinity of the scale SCL1 in the blood vessel image in which the guide wire 150 inserted into the subject and the scale SCL1 having graduations at equal intervals are shown after the learning process. A learning model capable of detecting W10 is stored in the memory 52 or the storage 55. The PC50 is a position capable of forming the same shape as the reference point W10 or the reference point W11 in one blood vessel image and the shape of the blood vessel including the reference point W10 or the reference point W11 by AI (artificial intelligence) based on the learning model. A plurality of detection points W2 to W8 indicating the above are specified, and the shape of the blood vessel including the reference point W10 or the reference point W11 and the plurality of detection points W2 to W8 is fitted. The PC 50 calculates the blood vessel diameter of the blood vessel based on the fitting result of the shape of the blood vessel, the length from the reference point W10 or the reference point W11 to any detection point, and the size G1 in the width direction of the guide wire 150. As a result, the blood vessel endoscopy system 5 can easily and accurately position the reference point W10 or the reference point W11 in the blood vessel image by using the AI for positioning the reference point already formed by the learning process. By using the size G1 (in other words, the thickness) of the guide wire 150 inserted into the subject in the width direction, the blood vessel diameter of the blood vessel which is the observation site of the subject can be calculated simply and with high accuracy.

Further, the PC 50 is a reference point indicating the position of the inner wall of the blood vessel in the vicinity of the guide wire 150A in the blood vessel image in which the guide wire 150A is inserted into the subject and has scales at equal intervals in the width direction after the learning process. A learning model capable of detecting W12 is stored in the memory 52 or the storage 55. The PC50 has a plurality of detection points W2 to indicate a reference point W12 in one blood vessel image and a position capable of forming the same shape as the shape of the blood vessel including the reference point W12 by AI (artificial intelligence) based on a learning model. W8 is identified and the shape of the blood vessel including the reference point W12 and the plurality of detection points W2 to W8 is fitted. The PC50 calculates the blood vessel diameter of the blood vessel based on the fitting result of the shape of the blood vessel, the length from the reference point W12 to any detection point, and the size G1 in the width direction of the guide wire 150A. As a result, the blood vessel endoscopy system 5 can easily and accurately position the reference point W12 in the blood vessel image by using the AI for positioning the reference point already formed by the learning process, and the inside of the subject can be positioned. The observation site of the subject using the size G1 (in other words, the thickness) of the guide wire 150A inserted into the guide wire 150A or the scale value (scale interval) from the starting point of the guide wire 150A (see FIG. 15). The blood vessel diameter of the blood vessel can be calculated simply and with high accuracy.

Further, the PC 50 superimposes the measurement result of the blood vessel diameter of the blood vessel on the blood vessel image GZ5 and displays it on the monitor 70 (see FIG. 6). As a result, a user such as a doctor can visually grasp the blood vessel diameter in an easy-to-understand manner.

Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can come up with various modifications, modifications, substitutions, additions, deletions, and equality within the scope of the claims. It is understood that it naturally belongs to the technical scope of the present disclosure. Further, each component in the various embodiments described above may be arbitrarily combined as long as the gist of the invention is not deviated.

For example, in the first embodiment described above, the constant velocity moving image is a moving image formed by sequentially selecting a plurality of blood vessel images having different imaging positions in chronological order, but the plurality of blood vessel images have a time axis. It may be selected at equal intervals above or at unequal intervals.

This application is based on a Japanese patent application filed on October 30, 2019 (Japanese Patent Application No. 2019-197660) and a Japanese patent application filed on March 24, 2020 (Japanese Patent Application No. 2020-052918). The contents of are incorporated herein by reference.

In the present disclosure, the blood vessel diameter of the observation site of a subject such as a patient is measured with high accuracy by using an image taken by an angioscope without interrupting the smooth progress of medical practice by a user such as a doctor. It is useful as an endoscopic system and a method for measuring blood vessel diameter.

5,5A Vascular Endoscopy System 10

Vascular Endoscopy Camera

20, 20A Repeater 21 FPGA
22 AFE
23, 24 Input interface 25 Output interface 30

Camera control unit

50, 50A PC
51 Processor 52 Memory 53 Input / output interface 54 Operation unit 55 Storage 70 Monitor 80 Auto pullback device 100 Vascular endoscope

Claims

An endoscope in which the distal end side is inserted into the blood vessel of the subject and the proximal end side is pulled by a driving device to image the blood vessel of the subject.
A repeater that inputs and combines the blood vessel image captured by the endoscope and the position information and speed information of the endoscope sent from the driving device.
The blood vessel image sent from the repeater is associated with the position information and speed information of the endoscope and stored in a memory, and the blood vessel image and the position information and speed information of the endoscope stored in the memory are stored. A calculation device for generating a constant-velocity moving image of the blood vessel image using the above and calculating the blood vessel diameter of the blood vessel based on a plurality of blood vessel images constituting the constant-velocity moving image.
Vascular endoscopy system.
An endoscope in which the distal end side is inserted into the blood vessel of the subject and the proximal end side is pulled by a driving device to image the blood vessel of the subject.
The blood vessel image captured by the endoscope and the position information and speed information of the endoscope sent from the driving device are input and stored in a memory in association with each other, and the blood vessel image saved in the memory. A constant velocity moving image of the blood vessel image is generated based on the difference between the input timing and the input timing of the position information and the speed information of the endoscope, and the blood vessel is based on a plurality of blood vessel images constituting the constant velocity moving image. It is equipped with an arithmetic device for measuring the blood vessel diameter of the
Vascular endoscopy system.
The arithmetic unit
The measurement result of the blood vessel diameter of the blood vessel is superimposed on the blood vessel image and displayed on the monitor.
The vascular endoscopy system according to claim 1 or 2.
The arithmetic unit
A long-axis cross-sectional image of the blood vessel is generated using the plurality of input blood vessel images, and the blood vessel image and the long-axis cross-sectional image of the blood vessel are associated and displayed on the monitor.
The vascular endoscopy system according to claim 1 or 2.
The arithmetic unit
The measurement length of the blood vessel is calculated based on the position information and speed information of the endoscope sequentially input from the drive device, and an indicator showing the calculation result of the measurement length is superimposed and displayed on the long-axis cross-sectional image. To do,
The vascular endoscopy system according to claim 4.
The arithmetic unit
Two blood vessel images corresponding to the position information of the endoscope are selected from the plurality of blood vessel images constituting the constant velocity moving image, and arranged in a circle with respect to each of the two blood vessel images. The vanishing point is estimated from the movement direction of each of the plurality of feature points to be created.
From the plurality of feature points, an effective feature point having a vector in the direction of the vanishing point based on the movement of the endoscope by the driving device is extracted, and the movement distance of the endoscope corresponds to the movement distance of the endoscope. The three-dimensional coordinates of the effective feature point are calculated using the amount of change in movement of the effective feature point.
The blood vessel diameter of the blood vessel is calculated based on the three-dimensional coordinates of the effective feature point.
The vascular endoscopy system according to claim 1 or 2.
The arithmetic unit
From the plurality of blood vessel images stored in the memory, a plurality of blood vessel images having different position information of the endoscope and the same velocity information are selected, and the selected blood vessel images are used. Generate the constant velocity video,
The vascular endoscopy system according to claim 1 or 2.
The endoscope has an image sensor capable of capturing a blood vessel of the subject.
The image sensor is mounted on the tip side of the endoscope.
The vascular endoscopy system according to claim 1 or 2.
A method of measuring blood vessel diameter performed by an angioscopy system,
The distal end side is inserted into the blood vessel of the subject and the proximal end side is pulled by the driving device, and the blood vessel is imaged by an endoscope capable of imaging the blood vessel of the subject.
The blood vessel image captured by the endoscope and the position information and speed information of the endoscope sent from the driving device are input by a repeater and combined.
The blood vessel image sent from the repeater is associated with the position information and speed information of the endoscope and stored in the memory.
A constant velocity moving image of the blood vessel image is generated using the blood vessel image stored in the memory and the position information and velocity information of the endoscope, and based on a plurality of blood vessel images constituting the constant velocity moving image. Calculate the blood vessel diameter of the blood vessel,
Blood vessel diameter measurement method.
A method of measuring blood vessel diameter performed by an angioscopy system,
The distal end side is inserted into the blood vessel of the subject and the proximal end side is pulled by the driving device, and the blood vessel is imaged by an endoscope capable of imaging the blood vessel of the subject.
The blood vessel image captured by the endoscope and the position information and speed information of the endoscope sent from the driving device are input, associated with each other, and stored in the memory.
A constant velocity moving image of the blood vessel image is generated based on the difference between the input timing of the blood vessel image stored in the memory and the input timing of the position information and the speed information of the endoscope, and the constant velocity moving image is configured. The blood vessel diameter of the blood vessel is measured based on a plurality of blood vessel images.
Blood vessel diameter measurement method.
An endoscope capable of imaging the blood vessel of the subject while the guide wire is inserted into the blood vessel of the subject inserted in advance and pulled through a driving device.
A repeater that inputs and combines the blood vessel image captured by the endoscope and the position information and speed information of the endoscope sent from the driving device.
The blood vessel image sent from the repeater is stored in a memory in association with the position information and the speed information of the endoscope, and the size of the guide wire stored in the memory in the width direction and the endoscope. A calculation device for calculating the blood vessel diameter of the blood vessel based on one blood vessel image corresponding to the position information and the velocity information.
Vascular endoscopy system.
The size of the guide wire in the width direction is a value that can be changed according to the user operation.
The vascular endoscopy system according to claim 11.
The arithmetic unit
A reference point indicating the position of the inner wall of the blood vessel in the vicinity of the guide wire in the one blood vessel image and a plurality of detection points indicating a position capable of forming the same shape as the shape of the blood vessel including the reference point are specified. ,
Fitting the shape of the blood vessel containing the reference point and the plurality of detection points
The blood vessel diameter of the blood vessel is calculated based on the fitting result of the shape of the blood vessel, the length from the reference point to any of the detection points, and the size in the width direction of the guide wire.
The vascular endoscopy system according to claim 11.
The arithmetic unit
After the learning process, a learning model capable of detecting a reference point indicating the position of the inner wall of the blood vessel in the vicinity of the guide wire in the blood vessel image in which the guide wire inserted into the subject is shown is stored in the memory.
By artificial intelligence based on the learning model, the reference point in the one blood vessel image and a plurality of detection points indicating positions capable of forming the same shape as the shape of the blood vessel including the reference point are identified.
Fitting the shape of the blood vessel containing the reference point and the plurality of detection points
The blood vessel diameter of the blood vessel is calculated based on the fitting result of the shape of the blood vessel, the length from the reference point to any of the detection points, and the size in the width direction of the guide wire.
The vascular endoscopy system according to claim 11.
The arithmetic unit
A learning model that can detect a reference point indicating the position of the inner wall of the blood vessel near the scale in the blood vessel image in which the guide wire inserted into the subject and the scale with equidistant scales are shown through the learning process. In the memory
By artificial intelligence based on the learning model, the reference point in the one blood vessel image and a plurality of detection points indicating positions capable of forming the same shape as the shape of the blood vessel including the reference point are identified.
Fitting the shape of the blood vessel containing the reference point and the plurality of detection points
The blood vessel diameter of the blood vessel is calculated based on the fitting result of the shape of the blood vessel, the length from the reference point to any of the detection points, and the size in the width direction of the guide wire.
The vascular endoscopy system according to claim 11.
The arithmetic unit
Learning that can detect a reference point indicating the position of the inner wall of the blood vessel in the vicinity of the guide wire in the blood vessel image in which the guide wire inserted into the subject and having the scales at equal intervals in the width direction is reflected through the learning process. Save the model in the memory
By artificial intelligence based on the learning model, the reference point in the one blood vessel image and a plurality of detection points indicating positions capable of forming the same shape as the shape of the blood vessel including the reference point are identified.
Fitting the shape of the blood vessel containing the reference point and the plurality of detection points
The blood vessel diameter of the blood vessel is calculated based on the fitting result of the shape of the blood vessel, the length from the reference point to any of the detection points, and the size in the width direction of the guide wire.
The vascular endoscopy system according to claim 11.
The arithmetic unit
The measurement result of the blood vessel diameter of the blood vessel is superimposed on the blood vessel image and displayed on the monitor.
The vascular endoscopy system according to claim 11.
A method of measuring blood vessel diameter performed by an angioscopy system,
The guide wire is inserted into the blood vessel of the subject inserted in advance, and the blood vessel is imaged by an endoscope capable of imaging the blood vessel of the subject while being pulled through a driving device.
The blood vessel image captured by the endoscope and the position information and speed information of the endoscope sent from the driving device are input by a repeater and combined.
The blood vessel image sent from the repeater is associated with the position information and speed information of the endoscope and stored in the memory.
The blood vessel diameter of the blood vessel is calculated based on the size of the guide wire in the width direction stored in the memory and one blood vessel image corresponding to the position information and the velocity information of the endoscope.
Blood vessel diameter measurement method.