JP2024051351A - IMAGE PROCESSING APPARATUS, IMAGE DISPLAY SYSTEM, IMAGE DISPLAY METHOD, AND IMAGE PROCESSING PROGRAM - Google Patents

Abstract

[Problem] To reduce the effect of pulsation when generating a three-dimensional image representing biological tissue. [Solution] An image processing device includes a control unit that acquires a plurality of cross-sectional images obtained at different times using a sensor moving through the lumen of the biological tissue for each position in the movement direction of the sensor, analyzes the acquired plurality of cross-sectional images, calculates a cross-sectional area of the lumen in the plurality of cross-sectional images, selects one cross-sectional image from the plurality of cross-sectional images according to the calculated cross-sectional area, and generates a three-dimensional image representing the biological tissue from a group of selected images acquired for a plurality of positions in the movement direction of the sensor. [Selected Figure] Figure 3

Description

The present disclosure relates to an image processing device, an image display system, an image display method, and an image processing program.

Patent Documents 1 to 3 describe techniques for generating three-dimensional images of cardiac chambers or blood vessels using an US imaging system. "US" is an abbreviation for ultrasound.

US Patent Application Publication No. 2010/0215238 U.S. Pat. No. 6,385,332 U.S. Pat. No. 6,251,072

Imaging diagnostics using IVUS is widely used for the cardiac cavity, cardiovascular system, and lower limb arterial region. "IVUS" is an abbreviation for intravascular ultrasound. IVUS is a device or method that provides two-dimensional images in a plane perpendicular to the catheter long axis.

Currently, surgeons must perform surgery by imagining the three-dimensional structure in their mind while sequentially observing two-dimensional IVUS images, which poses particular obstacles for younger or inexperienced physicians. In order to remove such obstacles, it is conceivable to automatically generate three-dimensional images that represent the structure of biological tissues such as cardiac chambers or blood vessels from two-dimensional IVUS images, and display the generated three-dimensional images to the surgeon.

The size and shape of biological tissues such as cardiac chambers and blood vessels change over time due to the influence of cardiac pulsation. The two-dimensional IVUS images used to generate three-dimensional images are not all acquired at the same timing in the pulsation cycle, so the resulting three-dimensional images have irregularities. It is difficult for surgeons to accurately grasp the structure of biological tissues from such three-dimensional images.

The objective of this disclosure is to reduce the effects of pulsation when generating three-dimensional images representing biological tissue.

Some aspects of this disclosure are described below.

[1]
An image processing device comprising: a control unit that acquires multiple cross-sectional images obtained at different times using a sensor for each position in a movement direction of the sensor moving through a cavity of a biological tissue, analyzes the acquired multiple cross-sectional images to calculate a cross-sectional area of the cavity in the multiple cross-sectional images, selects one cross-sectional image from the multiple cross-sectional images according to the calculated cross-sectional area, and generates a three-dimensional image representing the biological tissue from a group of selected images obtained for multiple positions in the movement direction of the sensor.

[2]
The image processing device according to claim 1, wherein the control unit selects, for each position in the movement direction of the sensor, from among the multiple acquired cross-sectional images, a cross-sectional image having a maximum cross-sectional area of the lumen.

[3]
The image processing device according to claim 1, wherein the control unit selects, for each position in the movement direction of the sensor, from among the multiple acquired cross-sectional images, a cross-sectional image having a smallest cross-sectional area of the lumen.

[4]
The image processing device according to claim 1, wherein the control unit selects, for each position in the movement direction of the sensor, from among the multiple acquired cross-sectional images, a cross-sectional image having a median cross-sectional area of the lumen.

[5]
The image processing device according to claim 1, wherein the control unit, when each position in the movement direction of the sensor is defined as a first position and one or more other positions are defined as a second position, compares a cross-sectional area of the inner cavity in a plurality of cross-sectional images acquired for the first position with a cross-sectional area of the inner cavity in a selected image acquired for the second position, and selects one cross-sectional image from the plurality of cross-sectional images depending on the obtained comparison result.

[6]
The control unit selects, from among the multiple cross-sectional images acquired for the first position, a cross-sectional image in which a difference between a cross-sectional area of the inner cavity and a cross-sectional area of the inner cavity in a selected image acquired for the second position is the smallest.

[7]
The image processing device according to any one of claims 5 to 6, wherein the second position includes at least one position before and after the first position in a movement direction of the sensor.

[8]
[1] to [7], and an image processing device according to any one of the above.
and a display for displaying the three-dimensional image.

[9]
acquiring a plurality of cross-sectional images obtained at different times using a sensor for each position in a moving direction of the sensor moving through a lumen of the biological tissue;
Analyzing the acquired cross-sectional images to calculate a cross-sectional area of the lumen in the cross-sectional images;
selecting one cross-sectional image from the plurality of cross-sectional images according to the calculated cross-sectional area;
generating a three-dimensional image representative of the biological tissue from a set of selected images obtained for a plurality of positions in a direction of movement of the sensor;
An image display method for displaying the generated three-dimensional image on a display.

[10]
A process of acquiring a plurality of cross-sectional images obtained at different times using a sensor moving through a lumen of a biological tissue for each position in a moving direction of the sensor;
A process of analyzing the acquired multiple cross-sectional images and calculating a cross-sectional area of the lumen in the multiple cross-sectional images;
selecting one cross-sectional image from the plurality of cross-sectional images according to the calculated cross-sectional area;
and generating a three-dimensional image representing the biological tissue from a group of selected images obtained for a plurality of positions in the movement direction of the sensor.

According to the present disclosure, it is possible to reduce the effect of pulsation when generating a three-dimensional image representing biological tissue.

FIG. 1 is a perspective view of an image display system according to an embodiment of the present disclosure. FIG. 2 is a perspective view of a probe and a drive unit according to an embodiment of the present disclosure. 1A and 1B are diagrams illustrating an example of a plurality of cross-sectional images and a selection image according to an embodiment of the present disclosure. A figure showing an example of a cross-sectional image obtained using a sensor according to an embodiment of the present disclosure and an example of a cross-sectional image after classification by a trained model. 1 is a block diagram showing a configuration of an image processing device according to an embodiment of the present disclosure. 4 is a flowchart showing an operation of the image display system according to the embodiment of the present disclosure. 10 is a flowchart showing a specific example of the operation of the image display system according to an embodiment of the present disclosure. 10 is a flowchart showing a modified example of the operation of the image display system according to the embodiment of the present disclosure.

An embodiment of the present disclosure will be described below with reference to the drawings.

In each figure, the same or corresponding parts are given the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

The configuration of the image display system 10 according to this embodiment will be described with reference to FIG.

The image display system 10 includes an image processing device 11, a cable 12, a drive unit 13, a keyboard 14, a mouse 15, and a display 16.

In this embodiment, the image processing device 11 is a dedicated computer specialized for image diagnosis, but it may also be a general-purpose computer such as a PC. "PC" is an abbreviation for personal computer.

The cable 12 is used to connect the image processing device 11 and the drive unit 13.

The drive unit 13 is connected to the probe 20 shown in FIG. 2 and drives the probe 20. The drive unit 13 is also called an MDU. "MDU" is an abbreviation for motor drive unit. The probe 20 is applied to IVUS. The probe 20 is also called an IVUS catheter or an imaging diagnostic catheter.

The keyboard 14, mouse 15, and display 16 are connected to the image processing device 11 via any cable or wirelessly. The display 16 is, for example, an LCD, an organic EL display, or an HMD. "LCD" is an abbreviation for liquid crystal display. "EL" is an abbreviation for electro luminescent. "HMD" is an abbreviation for head-mounted display.

The image display system 10 further includes a connection terminal 17 and a cart unit 18 as options.

The connection terminal 17 is used to connect the image processing device 11 to an external device. The connection terminal 17 is, for example, a USB terminal. "USB" is an abbreviation for Universal Serial Bus. The external device is, for example, a recording medium such as a magnetic disk drive, a magneto-optical disk drive, or an optical disk drive.

The cart unit 18 is a cart with casters for mobility. The image processing device 11, cable 12, and drive unit 13 are installed in the cart body of the cart unit 18. The keyboard 14, mouse 15, and display 16 are installed on the table at the top of the cart unit 18.

The configuration of the probe 20 and drive unit 13 according to this embodiment will be described with reference to FIG. 2.

The probe 20 comprises a drive shaft 21, a hub 22, a sheath 23, an outer tube 24, an ultrasonic transducer 25, and a relay connector 26.

The drive shaft 21 passes through a sheath 23 inserted into a body cavity of a living body and an outer tube 24 connected to the base end of the sheath 23, and extends to the inside of a hub 22 provided at the base end of the probe 20. The drive shaft 21 has an ultrasonic transducer 25 at its tip that transmits and receives signals, and is rotatably provided within the sheath 23 and outer tube 24. The relay connector 26 connects the sheath 23 and outer tube 24.

The hub 22, drive shaft 21, and ultrasonic transducer 25 are connected to each other so that they can move forward and backward in the axial direction as a unit. Therefore, for example, when the hub 22 is pushed toward the tip side, the drive shaft 21 and ultrasonic transducer 25 move toward the tip side inside the sheath 23 in the opposite direction to the direction indicated by the arrow. For example, when the hub 22 is pulled toward the base end, the drive shaft 21 and ultrasonic transducer 25 move toward the base end inside the sheath 23 in the direction indicated by the arrow.

The drive unit 13 includes a scanner unit 31, a slide unit 32, and a bottom cover 33.

The scanner unit 31 is also called a pullback unit. The scanner unit 31 is connected to the image processing device 11 via the cable 12. The scanner unit 31 includes a probe connection section 34 that connects to the probe 20, and a scanner motor 35 that is a drive source that rotates the drive shaft 21.

The probe connection section 34 is detachably connected to the probe 20 via a socket 36 of the hub 22 provided at the base end of the probe 20. Inside the hub 22, the base end of the drive shaft 21 is rotatably supported, and the rotational force of the scanner motor 35 is transmitted to the drive shaft 21. In addition, signals are transmitted between the drive shaft 21 and the image processing device 11 via the cable 12. In the image processing device 11, a tomographic image of the biological lumen is generated and the image is processed based on the signal transmitted from the drive shaft 21.

The slide unit 32 carries the scanner unit 31 so that it can move back and forth, and is mechanically and electrically connected to the scanner unit 31. The slide unit 32 includes a probe clamp section 37, a slide motor 38, and a group of switches 39.

The probe clamp section 37 is disposed coaxially with the probe connection section 34 and closer to the tip, and supports the probe 20 that is connected to the probe connection section 34.

The slide motor 38 is a drive source that generates a driving force in the axial direction. The scanner unit 31 moves forward and backward when driven by the slide motor 38, and the drive shaft 21 moves forward and backward in the axial direction accordingly. The slide motor 38 is, for example, a servo motor.

The switch group 39 includes, for example, a forward switch and a pullback switch that are pressed when moving the scanner unit 31 forward and backward, and a scan switch that is pressed when starting and ending image drawing. The switches are not limited to the examples shown here, and various other switches may be included in the switch group 39 as necessary.

When the forward switch is pressed, the slide motor 38 rotates forward, and the scanner unit 31 moves forward. On the other hand, when the pullback switch is pressed, the slide motor 38 rotates backward, and the scanner unit 31 moves backward.

When the scan switch is pressed, image depiction begins, and the scanner motor 35 and slide motor 38 are driven to repeatedly move the scanner unit 31 forward and backward. A user such as an operator connects the probe 20 to the scanner unit 31 in advance, and when image depiction begins, the drive shaft 21 rotates while repeatedly moving axially toward the tip end and toward the base end. When the scan switch is pressed again, the scanner motor 35 and slide motor 38 stop, and image depiction ends.

The bottom cover 33 covers the bottom surface of the slide unit 32 and the entire side surface on the bottom side, and can be moved toward and away from the bottom surface of the slide unit 32.

An overview of this embodiment will be explained with reference to Figures 3 to 5.

The image processing device 11 acquires a plurality of cross-sectional images 54 obtained at different times using the sensor 71 for each position in the movement direction of the sensor 71 moving through the lumen 61 of the biological tissue 60. The image processing device 11 analyzes the acquired plurality of cross-sectional images 54 for each position in the movement direction of the sensor 71 and calculates the cross-sectional area of the lumen 61 in the plurality of cross-sectional images 54. The image processing device 11 selects one cross-sectional image 55 from the plurality of cross-sectional images 54 according to the calculated cross-sectional area for each position in the movement direction of the sensor 71. In this embodiment, as a method for selecting the cross-sectional image 55, a method for selecting the cross-sectional image 54 having the largest cross-sectional area of the lumen 61 among the plurality of cross-sectional images 54 is used, but a method for selecting the cross-sectional image 54 having the smallest or median cross-sectional area of the lumen 61 among the plurality of cross-sectional images 54 may also be used. The image processing device 11 generates a three-dimensional image 53 representing the biological tissue 60 from a group of selected images obtained for a plurality of positions in the movement direction of the sensor 71. The display 16 displays the three-dimensional image 53 generated by the image processing device 11.

According to this embodiment, by selecting the two-dimensional image group used to generate the three-dimensional image 53 according to the cross-sectional area of the lumen 61 in each two-dimensional image, it becomes easier to select the two-dimensional image group acquired at the same timing in the pulsation cycle. As a result, it is possible to reduce the influence of pulsation when generating the three-dimensional image 53. In other words, it becomes possible to generate a three-dimensional image 53 that is almost free of unevenness caused by the influence of pulsation. Therefore, it becomes easier for the surgeon to accurately grasp the structure of the biological tissue 60 from the three-dimensional image 53.

The biological tissue 60 includes, for example, blood vessels or organs such as the heart. The biological tissue 60 is not limited to a single anatomical organ or a part thereof, but also includes tissue having a lumen across multiple organs. A specific example of such tissue is a part of the vascular tissue that extends from the upper part of the inferior vena cava through the right atrium to the lower part of the superior vena cava.

In this embodiment, the image processing device 11 generates and updates three-dimensional data 52 representing the biological tissue 60 by referring to the tomographic data 51, which is a data set of the cross-sectional image 54 obtained using the sensor 71. The image processing device 11 displays the three-dimensional data 52 as a three-dimensional image 53 on the display 16. That is, the image processing device 11 displays the three-dimensional image 53 on the display 16 by referring to the tomographic data 51.

The image processing device 11 may form an opening in the three-dimensional data 52 that exposes the inner cavity 61 of the biological tissue 60 in the three-dimensional image 53. The image processing device 11 may adjust the viewpoint when displaying the three-dimensional image 53 on the screen according to the position of the opening. The viewpoint refers to the position of a virtual camera placed in three-dimensional space.

The sensor 71 is provided in the catheter 70. In this embodiment, the catheter 70 is the probe 20, i.e., an IVUS catheter, but may be an OFDI catheter or an OCT catheter. "OFDI" is an abbreviation for optical frequency domain imaging. "OCT" is an abbreviation for optical coherence tomography. That is, in this embodiment, the sensor 71 is an ultrasonic transducer 25, which transmits ultrasonic waves in the lumen 61 of the biological tissue 60 to obtain the tomographic data 51, but may also emit light in the lumen 61 of the biological tissue 60 to obtain the tomographic data 51.

3, the Z direction corresponds to the movement direction of the sensor 71, but for convenience, the Z direction may be considered to correspond to the longitudinal direction of the inner cavity 61 of the biological tissue 60. The X direction perpendicular to the Z direction, and the Y direction perpendicular to the Z direction and the X direction may each be considered to correspond to the transverse direction of the inner cavity 61 of the biological tissue 60.

In this embodiment, each time an IVUS two-dimensional image is obtained as the cross-sectional image 54, the image processing device 11 stores the obtained two-dimensional image in a bucket B[i] corresponding to the position in the moving direction of the sensor 71. The position in the moving direction of the sensor 71 is identified, for example, by the position of the linear scaler of the pullback unit, and is assigned to each image as position information. The position information may be partitioned to include a range of a predetermined width. The capacity of each bucket may be set arbitrarily, for example, for four images. If there is no free space in the capacity of the bucket B[i] when an image is added to the bucket B[i], the oldest image may be deleted. The image processing device 11 calculates the number of pixels representing the lumen 61 contained in the image stored in the bucket B[i] as the cross-sectional area of the lumen 61. The pixels representing the lumen 61 are identified using the trained model 56, for example, as shown in FIG. 4. The image processing device 11 selects an image with the largest cross-sectional area among the images stored in each bucket. Alternatively, an image with the smallest or median cross-sectional area may be selected among the images stored in each bucket. The image processing device 11 uses a group of images selected for multiple buckets to draw a three-dimensional image 53.

In this embodiment, the image processing device 11 classifies a plurality of pixels included in each cross-sectional image 54 into two or more classes using the trained model 56. These two or more classes include a biological tissue class 80 corresponding to pixels representing biological tissue 60, a blood cell class 81 corresponding to pixels representing blood cells contained in blood flowing through the lumen 61, and a medical device class 82 corresponding to pixels representing medical devices 62 inserted into the lumen 61, such as catheters other than the catheter 70 or guidewires. An indwelling object class corresponding to pixels representing indwelling objects such as stents may also be included. A lesion class corresponding to pixels representing lesions such as calcification or plaque may also be included. Each class may be further subdivided. For example, the medical device class 82 may be divided into a catheter class, a guidewire class, and other medical device classes.

The learned model 56 is trained to detect regions corresponding to each class from a sample two-dimensional image by performing machine learning in advance. When a cross-sectional image 54 is input, the learned model 56 outputs a cross-sectional image 57 as a classification result in which each pixel of the input cross-sectional image 54 is assigned a classification of one of a biological tissue class 80, a blood cell class 81, and a medical instrument class 82. In this embodiment, pixels classified into the blood cell class 81 are treated as pixels representing the lumen 61, but pixels classified into other classes other than the biological tissue class 80, such as pixels classified into the medical instrument class 82, may also be treated as pixels representing the lumen 61. In each bucket, the cross-sectional image 54 before classification or the cross-sectional image 57 after classification may be accumulated.

In this embodiment, the image processing device 11 generates a three-dimensional image 53 by stacking and three-dimensionalizing a pixel group classified into the biological tissue class 80 included in the image group selected for the multiple buckets. The image processing device 11 may further generate a three-dimensional image of the medical device 62 by stacking and three-dimensionalizing a pixel group classified into the medical device class 82 included in the image group selected for the multiple buckets. Alternatively, the image processing device 11 may further generate a linear three-dimensional model connecting the coordinates of the pixel group classified into the medical device class 82 included in the image group selected for the multiple buckets as a three-dimensional image of the medical device 62.

The configuration of the image processing device 11 will be described with reference to Figure 5.

The image processing device 11 includes a control unit 41, a memory unit 42, a communication unit 43, an input unit 44, and an output unit 45.

The control unit 41 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination of these. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for a specific process. "CPU" is an abbreviation for central processing unit. "GPU" is an abbreviation for graphics processing unit. The programmable circuit is, for example, an FPGA. "FPGA" is an abbreviation for field-programmable gate array. The dedicated circuit is, for example, an ASIC. "ASIC" is an abbreviation for application specific integrated circuit. The control unit 41 executes processes related to the operation of the image processing device 11 while controlling each part of the image display system 10 including the image processing device 11.

The storage unit 42 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, a RAM or a ROM. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. The RAM is, for example, an SRAM or a DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. The ROM is, for example, an EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. The storage unit 42 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 42 stores data used in the operation of the image processing device 11, such as the tomographic data 51, and data obtained by the operation of the image processing device 11, such as the three-dimensional data 52 and the three-dimensional image 53. The storage unit 42 has a storage area corresponding to each bucket shown in FIG. 3 set therein.

The communication unit 43 includes at least one communication interface. The communication interface is, for example, a wired LAN interface compatible with a communication standard such as Ethernet (registered trademark), a wireless LAN interface compatible with a communication standard such as IEEE802.11, or an image diagnosis interface that receives and A/D converts IVUS signals. "LAN" is an abbreviation for local area network. "IEEE" is an abbreviation for Institute of Electrical and Electronics Engineers. "A/D" is an abbreviation for analog to digital. The communication unit 43 receives data used in the operation of the image processing device 11 and transmits data obtained by the operation of the image processing device 11. In this embodiment, the drive unit 13 is connected to the image diagnosis interface included in the communication unit 43.

The input unit 44 includes at least one input interface. The input interface is, for example, a USB interface, an HDMI (registered trademark) interface, or an interface compatible with a short-range wireless communication standard such as Bluetooth (registered trademark). "HDMI (registered trademark)" is an abbreviation for High-Definition Multimedia Interface. The input unit 44 accepts user operations such as an operation to input data used in the operation of the image processing device 11. In this embodiment, the keyboard 14 and the mouse 15 are connected to a USB interface included in the input unit 44 or an interface compatible with short-range wireless communication. When a touch screen is provided integrally with the display 16, the display 16 may be connected to a USB interface or an HDMI (registered trademark) interface included in the input unit 44.

The output unit 45 includes at least one output interface. The output interface is, for example, a USB interface, an HDMI (registered trademark) interface, or an interface compatible with a short-range wireless communication standard such as Bluetooth (registered trademark). The output unit 45 outputs data obtained by the operation of the image processing device 11. In this embodiment, the display 16 is connected to the USB interface or the HDMI (registered trademark) interface included in the output unit 45.

The functions of the image processing device 11 are realized by executing the image processing program according to this embodiment in a processor serving as the control unit 41. That is, the functions of the image processing device 11 are realized by software. The image processing program causes a computer to execute the operations of the image processing device 11, thereby causing the computer to function as the image processing device 11. That is, the computer functions as the image processing device 11 by executing the operations of the image processing device 11 in accordance with the image processing program.

The program may be stored in a non-transitory computer-readable medium. Examples of the non-transitory computer-readable medium include a flash memory, a magnetic recording device, an optical disk, a magneto-optical recording medium, or a ROM. The program may be distributed, for example, by selling, transferring, or lending a portable medium such as an SD card, DVD, or CD-ROM on which the program is stored. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program may be distributed by storing the program in the storage of a server and transferring the program from the server to another computer. The program may be provided as a program product.

For example, a computer temporarily stores a program stored in a portable medium or a program transferred from a server in a main storage device. The computer then reads the program stored in the main storage device with a processor and executes processing according to the read program with the processor. The computer may read the program directly from the portable medium and execute processing according to the program. The computer may execute processing according to the received program each time a program is transferred from the server to the computer. Processing may be executed by a so-called ASP-type service that does not transfer a program from the server to the computer and achieves functions only by issuing execution instructions and obtaining results. "ASP" is an abbreviation for application service provider. Programs include information used for processing by a computer and equivalent to a program. For example, data that is not a direct command to a computer but has properties that define computer processing falls under " equivalent to a program".

Some or all of the functions of the image processing device 11 may be realized by a programmable circuit or a dedicated circuit as the control unit 41. In other words, some or all of the functions of the image processing device 11 may be realized by hardware.

The operation of the image display system 10 according to this embodiment will be described with reference to FIG. 6. The operation of the image display system 10 corresponds to the image display method according to this embodiment.

Before starting the flow in FIG. 6, the user primes the probe 20. After that, the probe 20 is fitted into the probe connection portion 34 and the probe clamp portion 37 of the drive unit 13, and is connected and fixed to the drive unit 13. The probe 20 is then inserted to a target site in the biological tissue 60, such as a blood vessel or the heart.

In step S101, the scan switch included in the switch group 39 is pressed, and then the pullback switch included in the switch group 39 is pressed, thereby performing a so-called pullback operation. The probe 20 transmits ultrasonic waves from the ultrasonic transducer 25, which is moved back in the axial direction by the pullback operation, inside the biological tissue 60. The ultrasonic transducer 25 transmits ultrasonic waves radially while moving inside the biological tissue 60. The ultrasonic transducer 25 receives reflected waves of the transmitted ultrasonic waves. The probe 20 inputs the signal of the reflected waves received by the ultrasonic transducer 25 to the image processing device 11. The control unit 41 of the image processing device 11 processes the input signal to sequentially generate cross-sectional images 54 of the biological tissue 60, thereby acquiring tomographic data 51 including multiple cross-sectional images 54.

Specifically, the probe 20 transmits ultrasonic waves in multiple directions from the center of rotation to the outside, while rotating the ultrasonic transducer 25 in the circumferential direction and moving it in the axial direction inside the biological tissue 60. The probe 20 receives reflected waves from reflectors present in each of the multiple directions inside the biological tissue 60 using the ultrasonic transducer 25. The probe 20 transmits the received reflected wave signals to the image processing device 11 via the drive unit 13 and the cable 12. The communication unit 43 of the image processing device 11 receives the signal transmitted from the probe 20. The communication unit 43 A/D converts the received signal. The communication unit 43 inputs the A/D converted signal to the control unit 41. The control unit 41 processes the input signal to calculate the intensity value distribution of the reflected wave from the reflector present in the transmission direction of the ultrasonic wave from the ultrasonic transducer 25. The control unit 41 sequentially generates two-dimensional images having a luminance value distribution corresponding to the calculated intensity value distribution as cross-sectional images 54 of the biological tissue 60, thereby acquiring tomographic data 51, which is a data set of the cross-sectional images 54. The control unit 41 stores the acquired tomographic data 51 in the memory unit 42.

In this embodiment, the reflected wave signal received by the ultrasonic transducer 25 corresponds to the raw data of the tomographic data 51, and the cross-sectional image 54 generated by the image processing device 11 by processing the reflected wave signal corresponds to the processed data of the tomographic data 51.

As a modified example of this embodiment, the control unit 41 of the image processing device 11 may store the signal input from the probe 20 as it is in the storage unit 42 as the tomographic data 51. Alternatively, the control unit 41 may store data indicating the intensity value distribution of the reflected wave calculated by processing the signal input from the probe 20 in the storage unit 42 as the tomographic data 51. In other words, the tomographic data 51 is not limited to a data set of the cross-sectional image 54 of the biological tissue 60, but may be data that represents in some form the cross section of the biological tissue 60 at each moving position of the ultrasonic transducer 25.

As a variation of this embodiment, instead of the ultrasonic transducer 25 that rotates in a circumferential direction and transmits ultrasonic waves in multiple directions, a so-called array type in which multiple ultrasonic transducers are arranged in a circumferential direction may be used to transmit ultrasonic waves in multiple directions without rotating.

As a modified example of this embodiment, instead of the image processing device 11 generating a data set of the cross-sectional image 54 of the biological tissue 60, another device may generate a similar data set, and the image processing device 11 may acquire the data set from the other device. That is, instead of the control unit 41 of the image processing device 11 processing an IVUS signal to generate the cross-sectional image 54 of the biological tissue 60, the other device may process an IVUS signal to generate the cross-sectional image 54 of the biological tissue 60, and the generated cross-sectional image 54 may be input to the image processing device 11.

In step S102, the control unit 41 of the image processing device 11 generates three-dimensional data 52 of the biological tissue 60 based on the tomographic data 51 acquired in step S101. That is, the control unit 41 generates the three-dimensional data 52 based on the tomographic data 51 acquired by the sensor 71. Here, if three-dimensional data 52 that has already been generated exists, it is preferable to update only the data corresponding to the updated tomographic data 51, rather than regenerating all of the three-dimensional data 52 from scratch. In that case, the amount of data processing when generating the three-dimensional data 52 can be reduced, and the real-time nature of the three-dimensional image 53 in the subsequent step S103 can be improved.

Specifically, the control unit 41 of the image processing device 11 generates three-dimensional data 52 of the biological tissue 60 by stacking and three-dimensionalizing the cross-sectional images 54 of the biological tissue 60 contained in the tomographic data 51 stored in the storage unit 42. As a three-dimensionalization method, any of a rendering method such as surface rendering or volume rendering, and various processes associated therewith, such as texture mapping including environment mapping, and bump mapping, is used. The control unit 41 stores the generated three-dimensional data 52 in the storage unit 42.

The control unit 41 of the image processing device 11 classifies the pixel group of the cross-sectional image 54 included in the tomographic data 51 acquired in step S101 into two or more classes. These two or more classes include a biological tissue class 80, a blood cell class 81, and a medical instrument class 82, and may further include an indwelling object class or a lesion class. Any classification method may be used, but in this embodiment, a method of classifying the pixel group of the cross-sectional image 54 using a trained model 56 as shown in FIG. 4 is used.

In step S103, the control unit 41 of the image processing device 11 causes the three-dimensional data 52 generated in step S102 to be displayed on the display 16 as a three-dimensional image 53. At this point, the control unit 41 may set the angle at which the three-dimensional image 53 is displayed to any angle.

Specifically, the control unit 41 of the image processing device 11 generates a three-dimensional image 53 from the three-dimensional data 52 stored in the storage unit 42. The three-dimensional image 53 includes a group of three-dimensional objects, such as a tissue object representing the biological tissue 60 in three-dimensional space and a medical instrument object representing the medical instrument 62 in the three-dimensional space. That is, the control unit 41 generates a three-dimensional object of the biological tissue 60 from the data of the biological tissue 60 stored in the storage unit 42, and generates a three-dimensional object of the medical instrument 62 from the data of the medical instrument 62 stored in the storage unit 42. The control unit 41 causes the generated three-dimensional image 53 to be displayed on the display 16 via the output unit 45.

In step S104, if the user performs a change operation to set the angle at which the three-dimensional image 53 is displayed, the process proceeds to step S105. If the user does not perform a change operation, the process proceeds to step S106.

In step S105, the control unit 41 of the image processing device 11 accepts an operation to set the angle at which the three-dimensional image 53 is to be displayed via the input unit 44. The control unit 41 adjusts the angle at which the three-dimensional image 53 is to be displayed to the set angle. Then, in step S103, the control unit 41 causes the display 16 to display the three-dimensional image 53 at the angle set in step S105.

Specifically, the control unit 41 of the image processing device 11 receives, via the input unit 44, an operation by the user to rotate the three-dimensional image 53 displayed on the display 16 using the keyboard 14, the mouse 15, or a touch screen provided integrally with the display 16. The control unit 41 interactively adjusts the angle at which the three-dimensional image 53 is displayed on the display 16 in response to the user's operation. Alternatively, the control unit 41 receives, via the input unit 44, an operation by the user to input a numerical value for the angle at which the three-dimensional image 53 is displayed using the keyboard 14, the mouse 15, or a touch screen provided integrally with the display 16. The control unit 41 adjusts the angle at which the three-dimensional image 53 is displayed on the display 16 to match the input numerical value.

If the tomographic data 51 is updated in step S106, the processes of steps S107 and S108 are executed. If the tomographic data 51 is not updated, the presence or absence of a change operation by the user is confirmed again in step S104.

In step S107, the control unit 41 of the image processing device 11 processes the signal input from the probe 20 to generate a new cross-sectional image 54 of the biological tissue 60, similar to the processing in step S101, thereby acquiring tomographic data 51 including at least one new cross-sectional image 54.

In step S108, the control unit 41 of the image processing device 11 updates the three-dimensional data 52 of the biological tissue 60 based on the tomographic data 51 acquired in step S107. That is, the control unit 41 updates the three-dimensional data 52 based on the tomographic data 51 acquired by the sensor 71. In step S108, it is preferable to update only the data of the location corresponding to the updated tomographic data 51. In that case, the amount of data processing when generating the three-dimensional data 52 can be reduced, and the real-time performance of the three-dimensional image 53 can be improved in the subsequent step S103.

Specifically, the control unit 41 of the image processing device 11 analyzes the multiple cross-sectional images 54 obtained at different times included in the tomographic data 51 stored in the storage unit 42 for each position in the moving direction of the ultrasonic transducer 25 as the sensor 71, and calculates the cross-sectional area of the lumen 61 in the multiple cross-sectional images 54. The multiple cross-sectional images 54 may include the cross-sectional images 54 obtained in step S101, the cross-sectional images 54 obtained in the previous step S107, or both. The calculation process may be omitted for the cross-sectional images 54 whose cross-sectional area has already been calculated in step S102. The control unit 41 selects one cross-sectional image 55 from the multiple cross-sectional images 54 according to the calculated cross-sectional area for each position in the moving direction of the ultrasonic transducer 25. More specifically, the control unit 41 selects the cross-sectional image 54 with the largest cross-sectional area of the lumen 61 from the multiple cross-sectional images 54 for each position in the moving direction of the ultrasonic transducer 25 as the cross-sectional image 55. Alternatively, the control unit 41 may select, as the cross-sectional image 55, the cross-sectional image 54 having the smallest cross-sectional area of the lumen 61 among the multiple cross-sectional images 54 for each position in the moving direction of the ultrasonic transducer 25. Alternatively, the control unit 41 may select, as the cross-sectional image 55, the cross-sectional image 54 having the median cross-sectional area of the lumen 61 among the multiple cross-sectional images 54 for each position in the moving direction of the ultrasonic transducer 25. The control unit 41 updates the three-dimensional data 52 of the biological tissue 60 from the selected image group obtained for the multiple positions in the moving direction of the ultrasonic transducer 25. More specifically, the control unit 41 updates the three-dimensional data 52 of the biological tissue 60 by stacking the selected image group including the cross-sectional image 55 selected for each position in the moving direction of the ultrasonic transducer 25 to make it three-dimensional. As a method for making it three-dimensional, a method similar to that of step S102 is used. The control unit 41 stores the updated three-dimensional data 52 in the storage unit 42.

7 shows a specific example of the operation of selecting the cross-sectional image 55 when the latest cross-sectional image 54 is acquired. In step S201 shown in this figure, the control unit 41 of the image processing device 11 adds the latest cross-sectional image 54 to the bucket B[i] corresponding to the position in the movement direction of the ultrasound transducer 25 when the latest cross-sectional image 54 is obtained. As shown in FIG. 3, the bucket B[i] already stores one or more other cross-sectional images 54 obtained when the ultrasound transducer 25 is in the same position before the latest cross-sectional image 54 is obtained. In step S202, the control unit 41 calculates the cross-sectional area of the lumen 61 in the multiple cross-sectional images 54 in the bucket B[i], including the latest cross-sectional image 54. In step S203, the control unit 41 selects the cross-sectional image 54 with the largest cross-sectional area of the lumen 61 as the cross-sectional image 55 from the multiple cross-sectional images 54 in the bucket B[i]. Alternatively, the control unit 41 may select the cross-sectional image 54 with the smallest cross-sectional area of the lumen 61 as the cross-sectional image 55 from the multiple cross-sectional images 54 in the bucket B[i]. Alternatively, the control unit 41 may select, as the cross-sectional image 55, the cross-sectional image 54 having the median cross-sectional area of the lumen 61 from among the multiple cross-sectional images 54 in the bucket B[i].

The control unit 41 of the image processing device 11 classifies the pixel group of the latest cross-sectional image 54 into two or more classes, similar to step S102. Therefore, in step S202, the control unit 41 calculates the number of pixels representing the lumen 61, such as the number of pixels classified into the blood cell class 81, as the cross-sectional area of the lumen 61, for each cross-sectional image 54 in bucket B[i].

After step S108, in step S103 again, the control unit 41 of the image processing device 11 causes the display 16 to display the three-dimensional data 52 updated in step S108 as a three-dimensional image 53.

FIG. 8 shows a modified example of the operation of selecting a cross-sectional image 55 when the latest cross-sectional image 54 is acquired. The processing of steps S211 and S212 shown in this figure is the same as the processing of steps S201 and S202 shown in FIG. 7, so the description will be omitted. In step S213, the control unit 41 compares the cross-sectional area of the lumen 61 in the multiple cross-sectional images 54 in the bucket B[i] with the cross-sectional area of the lumen 61 in the cross-sectional images 55 selected in the previous and next buckets B[i-4], ..., B[i-1] and B[i+1], ..., B[i+4] as shown in FIG. 3. In step S214, the control unit 41 selects, as the cross-sectional image 55, the cross-sectional image 54 having the smallest difference in the cross-sectional area of the lumen 61 from the cross-sectional image 55 selected in the buckets B[i-4], ..., B[i-1] and B[i+1], ..., B[i+4] among the multiple cross-sectional images 54 in the bucket B[i]. Alternatively, the control unit 41 may weight the cross-sectional images 55 selected in the buckets B[i-4], ..., B[i-1] and B[i+1], ..., B[i+4] according to the position difference in the moving direction, and then select the cross-sectional image 54 in which the total difference in the cross-sectional area of the lumen 61 between the cross-sectional images 55 selected in the buckets B[i-4], ..., B[i-1] and B[i+1], ..., B[i+4] is the smallest as the cross-sectional image 55 among the multiple cross-sectional images 54 in the bucket B[i]. When the position difference in the moving direction is j, it is possible to use 1/j ² as a weighting coefficient. In the example shown in FIG. 8, the cross-sectional images 55 selected in four buckets in front and behind are compared, but the cross-sectional images 55 selected in any number of buckets may be compared, or the cross-sectional images 55 selected in one or more buckets on either the front or rear side may be compared.

As described above, when each position in the movement direction of the sensor 71 is defined as a first position and one or more other positions are defined as a second position, the control unit 41 of the image processing device 11 may compare the cross-sectional area of the lumen 61 in the multiple cross-sectional images 54 acquired at the first position with the cross-sectional area of the lumen 61 in the selected image acquired at the second position. The control unit 41 may select one cross-sectional image 55 from the multiple cross-sectional images 54 according to the obtained comparison result. Specifically, the control unit 41 may select, as the cross-sectional image 55, the cross-sectional image 54 of the multiple cross-sectional images 54 acquired at the first position that has the smallest difference in the cross-sectional area of the lumen 61 from the selected image acquired at the second position. The second position may include at least one position before and after the first position in the movement direction of the sensor 71.

According to this modified example, even if the multiple cross-sectional images 54 acquired for the first position include a cross-sectional image 54 in which a very large area is erroneously detected as a region corresponding to the blood cell class 81, the selection of that cross-sectional image 54 can be avoided. Similarly, even if the multiple cross-sectional images 54 acquired for the first position include a cross-sectional image 54 in which a very small area is erroneously detected as a region corresponding to the blood cell class 81, the selection of that cross-sectional image 54 can be avoided. Therefore, it is possible to generate an image with less noise as the three-dimensional image 53. Therefore, it becomes easier for the surgeon to more accurately grasp the structure of the biological tissue 60 from the three-dimensional image 53.

The present disclosure is not limited to the above-described embodiments. For example, two or more blocks shown in the block diagram may be integrated, or one block may be divided. Two or more steps shown in the flowchart may be executed in parallel or in a different order, instead of being executed in chronological order as described, depending on the processing capabilities of the device executing each step, or as needed. Other modifications are possible without departing from the spirit of the present disclosure.

LIST OF SYMBOLS 10 Image display system 11 Image processing device 12 Cable 13 Drive unit 14 Keyboard 15 Mouse 16 Display 17 Connection terminal 18 Cart unit 20 Probe 21 Drive shaft 22 Hub 23 Sheath 24 Outer tube 25 Ultrasonic transducer 26 Relay connector 31 Scanner unit 32 Slide unit 33 Bottom cover 34 Probe connection section 35 Scanner motor 36 Socket 37 Probe clamp section 38 Slide motor 39 Switch group 41 Control section 42 Memory section 43 Communication section 44 Input section 45 Output section 51 Tomographic data 52 Three-dimensional data 53 Three-dimensional image 54 Cross-sectional image 55 Cross-sectional image 56 Trained model 57 Cross-sectional image 60 Biological tissue 61 Inner cavity 62 Medical instrument 70 Catheter 71 Sensor 80 Living tissue class 81 Blood cell class 82 Medical instrument class

Claims (10)

An image processing device including a control unit that acquires multiple cross-sectional images obtained at different times using a sensor for each position in the movement direction of the sensor moving through the cavity of the biological tissue, analyzes the multiple acquired cross-sectional images, calculates the cross-sectional area of the cavity in the multiple cross-sectional images, selects one cross-sectional image from the multiple cross-sectional images according to the calculated cross-sectional area, and generates a three-dimensional image representing the biological tissue from the selected images obtained for multiple positions in the movement direction of the sensor. The image processing device according to claim 1, wherein the control unit selects, for each position in the movement direction of the sensor, from among the multiple acquired cross-sectional images, a cross-sectional image having the largest cross-sectional area of the lumen. The image processing device according to claim 1, wherein the control unit selects, for each position in the movement direction of the sensor, from among the multiple acquired cross-sectional images, a cross-sectional image with the smallest cross-sectional area of the lumen. The image processing device according to claim 1, wherein the control unit selects, for each position in the movement direction of the sensor, from among the multiple acquired cross-sectional images, a cross-sectional image having a median cross-sectional area of the lumen. The image processing device according to claim 1, wherein the control unit, when each position in the movement direction of the sensor is defined as a first position and one or more other positions are defined as a second position, compares the cross-sectional area of the lumen in a plurality of cross-sectional images acquired for the first position with the cross-sectional area of the lumen in a selected image acquired for the second position, and selects one cross-sectional image from the plurality of cross-sectional images according to the obtained comparison result. The image processing device according to claim 5, wherein the control unit selects, from among the multiple cross-sectional images acquired for the first position, a cross-sectional image in which the difference between the cross-sectional area of the lumen and the cross-sectional area of the lumen in the selected image acquired for the second position is the smallest. The image processing device according to claim 5, wherein the second position includes at least one position before and after the first position in the movement direction of the sensor. An image processing device according to any one of claims 1 to 7,
and a display for displaying the three-dimensional image. acquiring a plurality of cross-sectional images obtained at different times using a sensor for each position in a moving direction of the sensor moving through a lumen of the biological tissue;
Analyzing the acquired cross-sectional images to calculate a cross-sectional area of the lumen in the cross-sectional images;
selecting one cross-sectional image from the plurality of cross-sectional images according to the calculated cross-sectional area;
generating a three-dimensional image representative of the biological tissue from a set of selected images obtained for a plurality of positions in a direction of movement of the sensor;
An image display method for displaying the generated three-dimensional image on a display. A process of acquiring a plurality of cross-sectional images obtained at different times using a sensor moving through a lumen of a biological tissue for each position in a moving direction of the sensor;
A process of analyzing the acquired multiple cross-sectional images and calculating a cross-sectional area of the lumen in the multiple cross-sectional images;
selecting one cross-sectional image from the plurality of cross-sectional images according to the calculated cross-sectional area;
and generating a three-dimensional image representing the biological tissue from a group of selected images obtained for a plurality of positions in the movement direction of the sensor.

Images