JP2017153953A - Ultrasonic diagnostic apparatus and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a display image on which a position and a travelling direction of an instrument based on a heart valve is visible.SOLUTION: An ultrasonic diagnostic apparatus includes an acquisition unit, a determination unit, a generation unit and a display control unit. The acquisition unit acquires three-dimensional medical image data of a region imaged using an ultrasonic probe and including the heart valve of a subject and a catheter inserted into the heart cavity of the subject. The determination part determines a travelling direction of a tip end part of the catheter by acquiring information on the position and the posture of the tip end part included in the three dimensional medical image data using at least one of shape information showing the shape of the tip end part of the catheter and reflection characteristic information showing ultrasonic reflection characteristics of the tip end part. The generation part generates a display image from the three-dimensional medical image data on the basis of the position and the travelling direction of the tip end part. The display control part displays the display image.SELECTED DRAWING: Figure 1

Description

  Embodiments described herein relate generally to an ultrasonic diagnostic apparatus and an image processing program.

  2. Description of the Related Art Conventionally, various techniques for supporting treatment by taking an image of a subject in cardiac surgery treatment have been proposed. For example, in the treatment of mitral regurgitation, a catheter is inserted from the femoral (femoral vein) vessel into the heart mitral valve, and the mitral valve is clamped with a clip-shaped device to reduce back flow. Treatment is taking place. In this treatment, for example, the position of an instrument (catheter) inserted toward the mitral valve is determined using a fluoroscopic image taken during surgery and a transesophageal echocardiogram (TEE) diagram. Techniques for displaying correctly have been proposed. In addition, for example, a technique for displaying an image for guiding the course of a catheter by aligning a CT (Computed Tomography) image taken before surgery and an image during surgery has been proposed.

  In addition, a technique for changing the visual field following the movement of the puncture needle has been proposed.

JP 2004-215701 A

  The problem to be solved by the present invention is to provide an ultrasonic diagnostic apparatus and an image processing program capable of generating a display image in which the position and the traveling direction of an instrument with respect to a heart valve can be visually recognized.

  The ultrasonic diagnostic apparatus according to the embodiment includes an acquisition unit, a determination unit, a generation unit, and a display control unit. The acquisition unit acquires three-dimensional medical image data of a region including a heart valve of the subject and a catheter inserted into the heart chamber of the subject, which is imaged using an ultrasonic probe. The determination unit includes at least one of shape information representing a shape of the distal end portion of the catheter and reflection characteristic information representing an ultrasonic reflection characteristic of the distal end portion, and is included in the three-dimensional medical image data. The traveling direction of the tip is determined by obtaining the position and orientation information of the tip. The generation unit generates a display image from the three-dimensional medical image data based on the position and the traveling direction of the tip. The display control unit displays the display image.

FIG. 1 is a diagram illustrating an example of an ultrasonic diagnostic apparatus according to the first embodiment. FIG. 2 is a diagram for explaining an example of information stored in the memory circuit according to the first embodiment. FIG. 3 is a diagram for explaining processing of the estimation function according to the first embodiment. FIG. 4 is a diagram illustrating an example of a display image generated by the generation function according to the first embodiment. FIG. 5 is a flowchart showing a processing procedure of the ultrasonic diagnostic apparatus according to the first embodiment. FIG. 6 is a diagram illustrating an example of a display image generated by the generation function according to Modification 1 of the first embodiment. FIG. 7 is a diagram illustrating an example of a display image generated by the generation function according to the third modification of the first embodiment. FIG. 8 is a block diagram illustrating an example of an ultrasonic diagnostic apparatus according to the second embodiment. FIG. 9 is a diagram illustrating an example of a display image displayed by the display control function according to the second embodiment. FIG. 10 is a block diagram illustrating an example of an ultrasonic diagnostic apparatus according to a modification of the second embodiment. FIG. 11 is a block diagram illustrating an example of an ultrasonic diagnostic apparatus according to a modification of the third embodiment. FIG. 12 is a block diagram illustrating an example of an ultrasonic diagnostic apparatus according to the fourth embodiment.

  Hereinafter, an ultrasonic diagnostic apparatus and an image processing program according to embodiments will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a diagram illustrating an example of an ultrasonic diagnostic apparatus 1 according to the first embodiment. As shown in FIG. 1, the ultrasonic diagnostic apparatus 1 according to the first embodiment includes an ultrasonic probe 101, an input apparatus 102, a display 103, and an apparatus main body 100. The ultrasonic probe 101, the input device 102, and the display 103 are connected to the apparatus main body 100 so as to communicate with each other. The subject P is not included in the configuration of the ultrasonic diagnostic apparatus 1.

  The ultrasonic probe 101 transmits and receives ultrasonic waves. For example, the ultrasonic probe 101 has a plurality of piezoelectric vibrators (not shown). The plurality of piezoelectric vibrators generate ultrasonic waves based on a drive signal supplied from a transmission / reception circuit 110 included in the apparatus main body 100 described later. In addition, the plurality of piezoelectric vibrators included in the ultrasonic probe 101 receives reflected waves from the subject P and converts them into electrical signals. Further, the ultrasonic probe 101 includes a matching layer (not shown) provided on the piezoelectric vibrator, a backing material (not shown) for preventing propagation of ultrasonic waves from the piezoelectric vibrator to the rear, and the like. The ultrasonic probe 101 is detachably connected to the apparatus main body 100.

  When ultrasonic waves are transmitted from the ultrasonic probe 101 to the subject P, the transmitted ultrasonic waves are successively reflected on the discontinuous surface of the acoustic impedance in the body tissue of the subject P. The reflected wave is received as a reflected wave signal by a plurality of piezoelectric vibrators included in the ultrasonic probe 101. The amplitude of the received reflected wave signal depends on the difference in acoustic impedance at the discontinuous surface where the ultrasonic wave is reflected. Note that the reflected wave signal when the transmitted ultrasonic pulse is reflected by the moving blood flow or the surface of the heart wall depends on the velocity component of the moving object in the ultrasonic transmission direction due to the Doppler effect. And undergoes a frequency shift.

  For example, the ultrasonic probe 101 according to the first embodiment is a transesophageal echocardiography (TEE) probe. An ultrasonic probe 101 that is a TEE probe is orally inserted into the body of the subject P and is brought into contact with an upper digestive tract such as an esophagus or stomach. The ultrasonic probe 101 picks up an arbitrary cross section or picks up three-dimensional ultrasonic image data (volume data) by mechanically rotating a plurality of piezoelectric vibrators. Alternatively, the ultrasonic probe 101 is a two-dimensional array ultrasonic probe that can ultrasonically scan the subject P in three dimensions by arranging a plurality of ultrasonic transducers in a matrix. Also good. The two-dimensional array ultrasonic probe is capable of electronically scanning the subject P in three dimensions by electronically focusing and transmitting / receiving ultrasonic waves. The three-dimensional ultrasonic image data is an example of three-dimensional medical image data.

  The input device 102 corresponds to a device such as a mouse, a keyboard, a button, a panel switch, a touch command screen, a foot switch, a trackball, and a joystick. The input device 102 receives various setting requests from an operator of the ultrasonic diagnostic apparatus 1 and transmits the received various setting requests to the apparatus main body 100.

  The display 103 displays a GUI (Graphical User Interface) for the operator of the ultrasonic diagnostic apparatus 1 to input various setting requests using the input device 102, ultrasonic image data generated in the apparatus main body 100, and the like. .

  The apparatus main body 100 generates ultrasonic image data based on the reflected wave signal received by the ultrasonic probe 101. Even if the ultrasonic image data generated by the apparatus main body 100 is two-dimensional ultrasonic image data generated based on a two-dimensional reflected wave signal, it is generated based on a three-dimensional reflected wave signal. It may be dimensional ultrasonic image data.

  As illustrated in FIG. 1, the apparatus body 100 includes a transmission / reception circuit 110, a signal processing circuit 120, an image processing circuit 130, an image memory 140, a storage circuit 150, and a processing circuit 160. The transmission / reception circuit 110, the signal processing circuit 120, the image processing circuit 130, the image memory 140, the storage circuit 150, and the processing circuit 160 are connected to be communicable with each other.

  The transmission / reception circuit 110 controls ultrasonic transmission / reception performed by the ultrasonic probe 101 based on an instruction from the processing circuit 160 described later. The transmission / reception circuit 110 includes a pulse generator (not shown), a transmission delay circuit (not shown), a pulser (not shown), and the like, and supplies a drive signal to the ultrasonic probe 101. The pulse generator repeatedly generates a rate pulse for forming a transmission ultrasonic wave at a predetermined repetition frequency (PRF: Pulse Repetition Frequency). Further, the transmission delay circuit generates a delay time for each piezoelectric vibrator necessary for focusing the ultrasonic wave generated from the ultrasonic probe 101 into a beam and determining transmission directivity. Give for each rate pulse. The pulser applies a drive signal (drive pulse) to the ultrasonic probe 101 at a timing based on the rate pulse. That is, the transmission delay circuit arbitrarily adjusts the transmission direction of the ultrasonic wave transmitted from the piezoelectric vibrator surface by changing the delay time given to each rate pulse.

  The transmission / reception circuit 110 can instantaneously change the transmission frequency, the transmission drive voltage, and the like in order to execute a predetermined scan sequence based on an instruction from the processing circuit 160 described later. Particularly, the change of the transmission drive voltage is realized by a linear amplifier type transmission circuit (not shown) capable of instantaneously switching the value or a mechanism for electrically switching a plurality of power supply units (not shown). .

  For example, the transmission / reception circuit 110 includes an amplifier circuit (not shown), an A / D (Analog / Digital) converter (not shown), an adder (not shown), and a phase detection circuit (not shown). And the reflected wave signal received by the ultrasonic probe 101 is subjected to various processes to generate reflected wave data. The amplifier circuit amplifies the reflected wave signal for each channel and performs gain correction processing. The A / D converter performs A / D conversion on the gain-corrected reflected wave signal and gives a delay time necessary for determining reception directivity to the digital data. The adder performs addition processing of the reflected wave signal processed by the A / D converter. By the addition processing of the adder, the reflection component from the direction corresponding to the reception directivity of the reflected wave signal is emphasized. The phase detection circuit converts the output signal of the adder into a baseband in-phase signal (I signal, I: In-phase) and a quadrature signal (Q signal, Q: Quadrature-phase). Then, the phase detection circuit sends the I signal and the Q signal (IQ signal) to the signal processing circuit 120 at the subsequent stage. Note that the data before processing by the phase detection circuit is also called an RF signal. Hereinafter, “IQ signal and RF signal” generated based on the reflected wave of the ultrasonic wave are collectively referred to as “reflected wave data”.

  The signal processing circuit 120 performs various types of signal processing on the reflected wave data generated from the reflected wave signal by the transmission / reception circuit 110. The signal processing circuit 120 performs logarithmic amplification, envelope detection processing, logarithmic compression, and the like on the reflected wave data (IQ signal) read from the buffer, and multipoint signal strength is expressed by brightness. Data (B-mode data) is generated.

  Further, the signal processing circuit 120 generates data (Doppler data) obtained by extracting motion information based on the Doppler effect of the moving body within the scanning range by performing frequency analysis on the reflected wave data. Specifically, the signal processing circuit 120 generates Doppler data in which an average speed, an average variance value, an average power value, and the like are estimated at each of a plurality of sample points as movement information of the moving body. Here, the moving body is, for example, a blood flow, a tissue such as a heart wall, or a contrast agent. The signal processing circuit 120 according to the present embodiment uses, as the blood flow motion information (blood flow information), an average blood flow velocity, an average blood flow dispersion value, an average blood flow power value, and the like for each of a plurality of sample points. Generate Doppler data estimated in.

  The image processing circuit 130 generates ultrasonic image data from various data generated by the signal processing circuit 120. The image processing circuit 130 generates two-dimensional B-mode image data in which the intensity of the reflected wave is represented by luminance from the two-dimensional B-mode data generated by the signal processing circuit 120. Further, the image processing circuit 130 generates two-dimensional Doppler image data in which blood flow information is visualized from the two-dimensional Doppler data generated by the signal processing circuit 120. The two-dimensional Doppler image data is velocity image data, distributed image data, power image data, or image data obtained by combining these. The image processing circuit 130 generates color Doppler image data in which blood flow information is displayed in color as Doppler image data, or generates Doppler image data in which one blood flow information is displayed in gray scale. The image processing circuit 130 is an example of an image generation unit.

  Here, the image processing circuit 130 generally converts (scan converts) a scanning line signal sequence of ultrasonic scanning into a scanning line signal sequence of a video format typified by a television to generate ultrasonic image data. To do. Specifically, the image processing circuit 130 generates ultrasonic image data by performing coordinate conversion in accordance with the ultrasonic scanning mode by the ultrasonic probe 101. Further, the image processing circuit 130 may perform various image processes in addition to the scan conversion. For example, the image processing circuit 130 uses a plurality of image frames after scan conversion to perform image processing (smoothing processing) for regenerating an average brightness image, or image processing (edge enhancement) using a differential filter in the image. Process). The image processing circuit 130 synthesizes character information, scales, body marks, and the like of various parameters with the ultrasonic image data.

  That is, the B-mode data and Doppler data are ultrasonic image data before the scan conversion process, and the data generated by the image processing circuit 130 is ultrasonic image data after the scan conversion process. The B-mode data and the Doppler data are also called raw data (Raw Data). The image processing circuit 130 generates two-dimensional ultrasonic image data from the two-dimensional ultrasonic image data before the scan conversion process.

  Further, the image processing circuit 130 generates three-dimensional B-mode image data by performing coordinate conversion on the three-dimensional B-mode data generated by the signal processing circuit 120. In addition, the image processing circuit 130 generates three-dimensional Doppler image data by performing coordinate conversion on the three-dimensional Doppler data generated by the signal processing circuit 120.

  Further, the image processing circuit 130 performs rendering processing on the volume data in order to generate various two-dimensional image data for displaying the volume data on the display 103. The rendering process performed by the image processing circuit 130 includes, for example, a process of generating MPR image data from volume data by performing a cross-section reconstruction method (MPR: Multi Planer Reconstruction). The rendering processing performed by the image processing circuit 130 includes, for example, volume rendering (VR) processing that generates two-dimensional image data reflecting three-dimensional information.

  The image memory 140 is a memory that stores ultrasonic image data generated by the image processing circuit 130. The image memory 140 can also store the signal processing circuit 120 and data generated by the signal processing circuit 120. The B mode data and Doppler data stored in the image memory 140 can be called by an operator after diagnosis, for example, and become ultrasonic image data via the image processing circuit 130. The image memory 140 can also store the reflected wave data output from the transmission / reception circuit 110.

  The storage circuit 150 stores a control program for performing ultrasonic transmission / reception, image processing, and display processing, diagnostic information (for example, patient ID, doctor's findings, etc.), various data such as a diagnostic protocol and various body marks. . The storage circuit 150 is also used for storing image data stored in the image memory 140 as necessary. The data stored in the storage circuit 150 can be transmitted to an external device via an interface (not shown). The storage circuit 150 can also store data transmitted from an external device via an interface (not shown).

  The processing circuit 160 controls the entire processing of the ultrasonic diagnostic apparatus 1. Specifically, the processing circuit 160 is based on various setting requests input from the operator via the input device 102, various control programs and various data read from the storage circuit 150, and the transmission / reception circuit 110 and the signal processing circuit 120. , And the processing of the image processing circuit 130 is controlled.

  Further, the processing circuit 160 controls the display 103 to display the ultrasonic image data stored in the image memory 140 and the storage circuit 150 as an ultrasonic image for display. For example, the processing circuit 160 displays the color Doppler image data generated by the image processing circuit 130 on the display 103 as a color Doppler image for display. Further, for example, the processing circuit 160 displays the B mode data generated by the image processing circuit 130 on the display 103 as a B mode image for display.

  The processing circuit 160 executes an acquisition function 161, an estimation function 162, a determination function 163, a generation function 164, and a display control function 165. The processing functions executed by the acquisition function 161, the estimation function 162, the determination function 163, the generation function 164, and the display control function 165 are recorded in the storage circuit 150 in the form of a program that can be executed by a computer, for example. The processing circuit 160 is a processor that realizes a function corresponding to each program by reading each program from the storage circuit 150 and executing the program. The processing circuit 160 in a state where each program is read from the storage circuit 150 has each function shown in the processing circuit 160 of FIG. Processing executed by the acquisition function 161, the estimation function 162, the determination function 163, the generation function 164, and the display control function 165 will be described later.

  In the present embodiment, the processing functions described above are described as being realized by a single processing circuit 160, but a processing circuit is configured by combining a plurality of independent processors, and each processor is programmed. The function may be realized by executing.

  The term “processor” used in the above description is, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an application specific integrated circuit (ASIC), a programmable logic device (for example, It means circuits such as a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA). The processor implements a function by reading and executing a program stored in the storage circuit. Instead of storing the program in the storage circuit 150, the program may be directly incorporated into the processor circuit. In this case, the processor realizes the function by reading and executing the program incorporated in the circuit. Note that each processor of the present embodiment is not limited to being configured as a single circuit for each processor, but may be configured as a single processor by combining a plurality of independent circuits to realize the function. Good. Furthermore, a plurality of components in each figure may be integrated into one processor to realize the function.

  In addition, in the following description, the case where the ultrasonic diagnostic apparatus 1 which concerns on this embodiment is utilized by the catheter treatment of mitral regurgitation as an example is demonstrated. Specifically, a case will be described in which the mitral valve that separates the left atrium and the left ventricle of the heart is applied to a treatment for reducing the back flow rate by fastening with a clip-shaped instrument inserted by a catheter. However, the embodiment is not limited to this, for example, an aortic valve that separates the left ventricle and the aorta, a tricuspid valve that separates the right atrium and the right ventricle, a pulmonary valve that separates the right ventricle and the pulmonary artery, etc. It is also applicable to the treatment of other heart valves.

  The storage circuit 150 stores information related to the catheter. For example, the storage circuit 150 stores shape information representing the shape of the distal end of the catheter and reflection characteristic information representing the reflection characteristic of the ultrasonic wave at the distal end of the catheter. That is, the storage circuit 150 stores factor information that is a factor of how the catheter is rendered in volume data that is ultrasonic image data. Note that the storage circuit 150 is an example of a storage unit. A catheter is an example of an instrument.

  FIG. 2 is a diagram for explaining an example of information stored in the storage circuit 150. For example, the storage circuit 150 stores information related to the distal end portion 10 of the catheter. Here, the distal end portion 10 of the catheter includes, for example, a clip portion 11 and a tubular portion 12. The clip part 11 is a part for fastening the mitral valve. Specifically, the clip part 11 improves the insufficiency of the mitral valve by bringing together two valves constituting the mitral valve. The tubular portion 12 is a portion operated by a catheter operator (operator) in order to insert the clip portion 11 attached to the distal end portion 10 up to the mitral valve. When the mitral valve is fastened, the clip part 11 is separated from the tubular part 12 and placed in the body of the subject P.

  For example, the storage circuit 150 stores shape information of the distal end portion 10 of the catheter and reflection characteristic information. The shape information is information representing the shape of the distal end portion 10 of the catheter. For example, the shape information is information on the three-dimensional shape of the clip portion 11 and the tubular portion 12 included in the distal end portion 10 of the catheter, and includes feature amounts relating to the slenderness and the sphericity. The reflection characteristic information is information representing the reflection characteristic of the ultrasonic wave at the distal end portion 10 of the catheter. For example, the reflection characteristic information includes the intensity of the reflected wave signal when the transmission ultrasonic wave from the ultrasonic probe 101 is reflected by the clip part 11 and the reflected wave signal when the transmission ultrasonic wave is reflected by the tubular part 12. Including strength. The intensity of the reflected wave signal varies depending on the composition (material) of the clip portion 11 and the tubular portion 12.

  The above description is merely an example. For example, the storage circuit 150 does not necessarily store both shape information and reflection characteristics, and may store only one of them. Moreover, although the application example to the catheter treatment using a clip was shown in FIG. 2, the embodiment is not limited to this. For example, it can be applied to catheter treatment using an artificial valve. In this case, the storage circuit 150 stores, for example, the shape information and reflection characteristic information of the catheter with the artificial valve attached to the tip. In other words, the tip of the catheter includes a clip or prosthetic valve that is placed in the mitral valve.

  Moreover, this embodiment is applicable not only to a clip and an artificial valve but when other instruments, such as an artificial annulus and an artificial chord, are used. An artificial annulus is an instrument for contracting an enlarged annulus embedded in the periphery of an annulus, for example, when the valve annulus becomes incomplete due to expansion. The artificial chord is a device for connecting the leaflet and the papillary muscle (or the vicinity of the apex) in the case where the insufficiency is caused by extension or cutting of the chordae.

  The acquisition function 161 acquires three-dimensional medical image data obtained by imaging the heart valve of the subject P and the instrument inserted into the body of the subject P using the ultrasonic probe 101. For example, the acquisition function 161 acquires volume data at the time when the mitral valve and the distal end portion 10 of the catheter are imaged from the image memory 140. The acquisition function 161 is an example of an acquisition unit. In other words, the acquisition function 161 acquires three-dimensional medical image data of a region including a heart valve of a subject and a catheter inserted into the heart chamber of the subject, which is imaged using an ultrasonic probe.

  For example, in catheter treatment of mitral regurgitation, the operator orally inserts a TEE probe into the body of the subject P and performs a three-dimensional scan of the region including the mitral valve. The reflected wave data collected by the three-dimensional scanning is generated as volume data by the image processing circuit 130 and stored in the image memory 140. The acquisition function 161 acquires volume data stored in the image memory 140.

  The above description of the acquisition function 161 is merely an example. For example, the acquisition function 161 may acquire volume data obtained by imaging other heart valves. For example, the acquisition function 161 may acquire volume data obtained by imaging the heart valve with an arbitrary ultrasonic probe other than the TEE probe. Further, the acquisition function 161 may acquire volume data generated as a moving image for each frame, or may acquire one volume data generated as a still image.

  The estimation function 162 estimates the position and orientation of the instrument included in the three-dimensional medical image data using the shape information and the reflection characteristic information. For example, the estimation function 162 estimates the position and posture of the catheter included in the volume data using the shape information and reflection characteristic information stored in the storage circuit 150. The estimation function 162 is an example of an estimation unit. In the following description, the position and orientation may be described as “position and orientation”.

  For example, the estimation function 162 estimates the position and posture of the instrument (catheter) included in the volume data in the coordinate system of the volume data acquired by the acquisition function 161. Here, the position of the instrument is, for example, “three-dimensional coordinates of the tip of the instrument” in the coordinate system of the volume data. The posture of the instrument is, for example, “a direction vector indicating the longitudinal direction of the instrument”.

  FIG. 3 is a diagram for explaining processing of the estimation function 162 according to the first embodiment. FIG. 3 illustrates a binarized image 20 in which volume data obtained by imaging a catheter is binarized.

  As shown in FIG. 3, the estimation function 162 estimates the position and posture of the catheter by image processing. For example, the estimation function 162 binarizes the volume data using the catheter reflection characteristic information stored in the storage circuit 150. Here, on the image generated from the ultrasound image data, the catheter inserted in the vicinity of the heart is rendered with a relatively high luminance with respect to the luminance of the heart region. That is, the estimation function 162 sets a luminance threshold value that can distinguish between the heart region and the catheter region based on the intensity of the reflected wave signal reflected by the distal end portion 10 of the catheter. Then, the estimation function 162 binarizes the volume data using the set threshold value, and generates the binarized image 20 shown in FIG. Note that the coordinate system of the binarized image 20 is the same as or associated with the coordinate system of the volume data.

  Subsequently, the estimation function 162 estimates the position and posture of the catheter from the binarized image 20 using the catheter shape information stored in the storage circuit 150. Specifically, the estimation function 162 calculates a feature amount related to a slenderness, a sphericity, and the like for a high luminance region included in the binarized image 20. Then, the estimation function 162 compares the calculated feature amount with each of the shape information of the clip portion 11 and the shape information of the tubular portion 12, whereby the region of the clip portion 11 and the region of the tubular portion 12 in the binarized image 20. Are extracted respectively. Then, the estimation function 162 calculates, for example, the three-dimensional coordinates of the center of gravity 21 of the region of the clip unit 11, and estimates the calculated three-dimensional coordinates as the position of the catheter. Further, the estimation function 162 calculates, for example, the major axis direction of the region of the clip portion 11 and the region of the tubular portion 12, and estimates a vector corresponding to the calculated major axis direction as the posture of the catheter.

  Thus, the estimation function 162 estimates the position and posture of the catheter. That is, the estimation function 162 estimates the position and posture of the catheter using factor information on how the catheter is drawn in the ultrasound image data.

  Note that the above description of the estimation function 162 is merely an example. For example, the position of the instrument may be a three-dimensional coordinate of the tip of the clip part 11. Further, the posture of the instrument may be a direction vector of the area of the clip portion 11. For example, the estimation function 162 does not necessarily need to use both shape information and reflection characteristics. For example, the estimation function 162 may estimate the position and posture of the catheter from the volume data using only the shape information of the catheter. That is, the estimation function 162 may estimate the position and posture of the catheter using at least one of shape information and reflection characteristic information.

  In addition, for example, the estimation function 162 may estimate the position and posture of the catheter using different features (compositions) at two locations separated in the long axis direction. For example, the estimation function 162 extracts the regions of the clip part 11 and the tubular part 12 from the binarized image 20 using the shape information of the clip part 11 and the tubular part 12, respectively. Then, the estimation function 162 calculates the center of gravity for each of the extracted regions of the clip part 11 and the tubular part 12. Then, the estimation function 162 may estimate a direction vector from the center of gravity of the tubular portion 12 toward the center of gravity of the clip portion 11 as the posture of the catheter. In addition, for example, the estimation function 162 may estimate the position and posture of the catheter by using different features at two locations separated in the short axis direction of the catheter (the direction orthogonal to the long axis direction of the catheter). For example, by disposing a composition (material) having characteristic reflection characteristics at positions that are asymmetric with respect to the short axis direction of the tubular portion 12, the position and posture based on the outer peripheral direction (circumferential direction) of the tubular portion 12. Can be estimated.

  The determination function 163 determines the traveling direction of the instrument based on the position and posture of the instrument. For example, the determination function 163 determines the traveling direction of the catheter based on the position and posture of the catheter. For example, the determination function 163 determines the direction of the vector corresponding to the posture of the catheter as the traveling direction of the catheter with the position of the catheter as the origin. For example, the vector corresponding to the posture of the catheter is a direction vector indicating the longitudinal direction of the catheter. The determination function 163 is an example of a determination unit. In other words, the determination function 163 is included in the three-dimensional medical image data using at least one of shape information representing the shape of the distal end portion of the catheter and reflection characteristic information representing the reflection characteristic of the ultrasound at the distal end portion. By obtaining the position and orientation information of the tip, the traveling direction of the tip is determined.

  In the example of FIG. 3, the determination function 163 determines the three-dimensional coordinates of the center of gravity 21 of the clip unit 11 as the origin. Then, the determination function 163 determines the direction of the vector corresponding to the posture of the catheter from the determined origin as the catheter traveling direction.

  Note that the above description of the determination function 163 is merely an example. For example, the determination function 163 calculates the center of gravity of the region of the clip portion 11 and the tubular portion 12 as the position of the catheter. And the determination function 163 may determine the advancing direction of a catheter by making the gravity center of the area | region of the calculated clip part 11 and the tubular part 12 into an origin. Further, for example, the determination function 163 calculates the direction vector from the origin to the center of gravity of the clip portion 11 as the posture of the catheter with the center of gravity of the tubular portion 12 as the origin. Then, the determination function 163 may determine the calculated direction vector as the traveling direction. In addition, since the catheter as a whole progresses while curving in the body, the traveling direction is preferably determined based on the position / posture of the distal end region of the catheter.

  The generation function 164 generates a display image from the three-dimensional medical image data based on the position and the traveling direction of the instrument. For example, the generation function 164 generates, as a display image, a volume rendering (VR) image having a position based on the position of the instrument as a viewpoint and a moving direction of the instrument as a line of sight. The generation function 164 is an example of a generation unit. In other words, the generation function 164 generates a display image from the three-dimensional medical image data based on the position of the tip and the traveling direction. The display image is an image that three-dimensionally represents the three-dimensional medical image data.

  FIG. 4 is a diagram illustrating an example of a display image generated by the generation function 164 according to the first embodiment. FIG. 4 illustrates a VR image 30 with the position of the distal end portion 10 of the catheter as a viewpoint and the direction of travel of the catheter as a line of sight. Since the distal end portion 10 of the catheter inserted into the blood vessel of the subject P is the viewpoint, the VR image 30 is rendered as if the blood vessel lumen or heart valve was seen from inside the blood vessel.

  As illustrated in FIG. 4, for example, the generation function 164 performs VR processing on the volume data acquired by the acquisition function 161 to generate a VR image 30. Here, the generation function 164 sets the viewpoint and line of sight of the VR processing based on the position and traveling direction of the catheter. For example, the generation function 164 sets the center of gravity 21 of the clip portion 11 estimated by the estimation function 162 as the position of the distal end portion 10 of the catheter as a viewpoint of VR processing. In addition, the generation function 164 sets the direction vector determined as the catheter traveling direction by the determination function 163 as a line of sight for VR processing. Then, the generation function 164 generates the VR image 30 that is a display image by executing the VR processing on the volume data using the set viewpoint and line of sight of the VR processing.

  Here, the VR image 30 is an image in which the position of the distal end portion 10 of the catheter is a viewpoint and the traveling direction of the catheter is a line of sight, so that it looks as if the blood vessel lumen in the traveling direction is viewed from the distal end portion 10 of the catheter. Image. When the mitral valve is positioned in the traveling direction, the mitral valve is depicted in the VR image 30 as shown in a region 31 of FIG. When the mitral valve is deviated from the advancing direction of the catheter, the position of the mitral valve is also drawn at a shifted position or not drawn at all.

  As described above, the generation function 164 generates a display image from the volume data based on the position and the traveling direction of the catheter. Note that the above description of the generation function 164 is merely an example. For example, the generation function 164 does not necessarily need to set the catheter position itself as the VR processing viewpoint. For example, the generation function 164 can set an arbitrary position based on the position of the catheter as a viewpoint of VR processing. For example, the generation function 164 may set a position away from the distal end of the catheter by a predetermined distance in the traveling direction as the viewpoint of VR processing. As a result, since the catheter itself is depicted in the VR image by using the position obliquely backward with respect to the moving direction of the catheter as a viewpoint, the positional relationship between the distal end of the catheter and the structure of the heart in the moving direction can be grasped. It becomes easy. Further, for example, if the catheter position itself is the viewpoint, the movement of the VR image may become severe when the catheter moves rapidly, and the visibility of the VR image may be significantly reduced. As a result, a stable VR image can be provided even when the movement of the catheter is severe. The display image generated by the generation function 164 is not limited to a VR image, and may be a surface rendering image related to the surface position of a biological tissue such as a mitral valve detected from a binarized image, for example. . That is, the display image is a volume rendering image or a surface rendering image. Further, these display images are volume rendering images or surface rendering images generated using a viewpoint based on the position of the tip and a line of sight based on the traveling direction of the tip.

  The display control function 165 displays a display image. For example, the display control function 165 causes the display 103 to display the VR image 30 generated by the generation function 164. The display control function 165 is an example of a display control unit. Here, in recent years, a technique is known in which a stereoscopic effect is enhanced by displaying a shadow behind a light source set in a VR image hitting a structure. In this embodiment, it is expected that additional information will be given to grasp the positional relationship by setting the light source so that the shadow of the catheter is reflected on the valve or myocardium. A VR image may be used.

  Note that the above description of the display control function 165 is merely an example. For example, in addition to the VR image 30, the display control function 165 may be displayed simultaneously with a cross-sectional image of an arbitrary cross-section of volume data, an X-ray fluoroscopic image captured during surgery, or the like. Note that other examples of display modes will be described later in other embodiments or modifications.

  FIG. 5 is a flowchart showing a processing procedure of the ultrasonic diagnostic apparatus 1. The processing procedure illustrated in FIG. 5 is started when, for example, an instruction to start imaging is received from the operator.

  In step S101, it is determined whether imaging has started. For example, the input device 102 receives an instruction to start imaging from the operator, and transmits the received instruction to the processing circuit 160 of the apparatus main body 100. When receiving the instruction transmitted from the input device 102, the processing circuit 160 determines that imaging has started (Yes at Step S101), and starts processing after Step S102. If imaging is not started (No at Step S101), the processing after Step S102 is not started, and each processing function of the processing circuit 160 is in a standby state.

  If step S101 is affirmed, in step S102, the acquisition function 161 acquires three-dimensional medical image data (volume data). For example, the acquisition function 161 acquires volume data obtained by imaging the mitral valve and the distal end portion 10 of the catheter from the image memory 140.

  In step S103, the estimation function 162 estimates the position and posture of the catheter. For example, the estimation function 162 estimates the position and posture of the catheter included in the volume data using the catheter shape information and reflection characteristic information stored in the storage circuit 150. Specifically, the estimation function 162 binarizes the volume data using the catheter reflection characteristic information, and generates the binarized image 20. And the estimation function 162 calculates the feature-value regarding slenderness, a sphericity, etc. about the high-intensity area | region contained in the binarized image 20, and compares with the shape information of a catheter, The catheter of the binarized image 20 Extract regions. Then, the estimation function 162 estimates the position and posture of the catheter based on the extracted three-dimensional coordinates of the catheter region.

  In step S104, the determination function 163 determines the traveling direction of the catheter. For example, the determination function 163 determines the advancing direction of the posture of the catheter based on the position and posture of the catheter estimated by the estimation function 162. Specifically, the determination function 163 determines the direction of the vector corresponding to the posture of the catheter as the traveling direction of the catheter with the position of the catheter as the origin.

  In step S105, the generation function 164 generates a display image based on the position and the traveling direction of the catheter. For example, the generation function 164 performs VR processing on the volume data with the position of the catheter as a viewpoint and the direction of movement of the catheter as a line of sight, and generates a VR image 30 as a display image.

  In step S106, the display control function 165 displays a display image. For example, the display control function 165 causes the display 103 to display the VR image 30 generated by the generation function 164.

  As described above, the ultrasonic diagnostic apparatus 1 generates and displays the VR image 30 with the catheter position as the viewpoint from the volume data and the catheter traveling direction. In FIG. 5, the processing procedure in the case of displaying the VR image 30 as a still image is illustrated, but it can also be applied to the case of displaying as a moving image. When displaying as a moving image, the processing circuit 160 repeatedly executes the processing of step S102 to step S106 until receiving an instruction to end imaging.

  As described above, in the ultrasound diagnostic apparatus 1, the acquisition function 161 acquires volume data obtained by imaging the mitral valve (heart valve) and the instrument (catheter). Then, the estimation function 162 estimates the position and orientation of the instrument included in the volume data using the instrument shape information and reflection characteristic information. And the determination function 163 determines the advancing direction of the attitude | position of an instrument based on the position and attitude | position of an instrument. And the production | generation function 164 produces | generates a display image based on the position and the advancing direction of an instrument. Then, the display control function 165 displays a display image. According to this, the ultrasonic diagnostic apparatus 1 according to the first embodiment can generate a display image in which the position of the instrument and the traveling direction with respect to the heart valve can be visually recognized.

  That is, the ultrasonic diagnostic apparatus 1 according to the first embodiment generates the VR image 30 by VR processing using the position of the distal end portion 10 of the catheter as a viewpoint and the line of sight of the traveling direction of the catheter. As a result, the ultrasonic diagnostic apparatus 1 can display the VR image 30 as if looking into the blood vessel lumen in the traveling direction from the distal end portion 10 of the catheter. As an example, when the ultrasound diagnostic apparatus 1 images in real time how the operator moves the catheter toward the mitral valve, an image as if traveling through the blood vessel lumen toward the mitral valve is obtained. It is possible to display. In addition, since the viewpoint and line of sight depicting the blood vessel lumen change according to the operation of the operator advancing and returning the catheter, the operator can intuitively grasp the movement of the catheter in the blood vessel lumen. It becomes possible. For this reason, when moving the catheter toward the mitral valve, the operator can operate the catheter as if the thread is passed through the hole of the sewing needle. As a result, the operator can easily place (clamp) the clip portion 11 attached to the distal end portion 10 of the catheter on the mitral valve, thereby improving the treatment accuracy while reducing the time required for the catheter treatment. Can be made.

  In addition, for example, the ultrasound diagnostic apparatus 1 according to the first embodiment uses factor information (shape information and reflection characteristic information) that is a factor of how an instrument is depicted in ultrasound image data. Estimate the position and orientation of the device in the image. Therefore, the ultrasonic diagnostic apparatus 1 can accurately estimate the position and orientation of the instrument even if the instrument itself has a property of bending in the subject, such as a catheter. Specifically, the ultrasonic diagnostic apparatus 1 uses the different features at two locations separated in the longitudinal direction of the instrument, such as the clip part 11 and the tubular part 12 included in the distal end part 10, to position the instrument. Estimate posture. For this reason, the ultrasonic diagnostic apparatus 1 can accurately determine the traveling direction of the instrument that is the line of sight of the VR process.

(Modification 1 of the first embodiment)
In the first embodiment, the case where the ultrasound diagnostic apparatus 1 displays the VR image 30 as a display image has been described, but the embodiment is not limited to this. For example, the ultrasonic diagnostic apparatus 1 may generate a cross-sectional image (MPR image).

  The generation function 164 generates a cross-sectional image along the traveling direction of the instrument as a display image. For example, the generation function 164 generates a cross-sectional image of two orthogonal cross sections that pass through the catheter traveling direction determined by the determination function 163.

  FIG. 6 is a diagram illustrating an example of a display image generated by the generation function 164 according to the first modification of the first embodiment. FIG. 6 illustrates a cross-sectional image 40 in a predetermined direction passing through the advancing direction of the catheter and a cross-sectional image 41 passing through the advancing direction and orthogonal to the cross-sectional image 40.

  As shown in FIG. 6, the generation function 164 generates a cross-sectional image 40 and a cross-sectional image 41. Here, since both the cross-sectional image 40 and the cross-sectional image 41 pass through the advancing direction of the catheter, the catheter 42 is depicted.

  Then, the display control function 165 causes the display 103 to display the cross-sectional image 40 and the cross-sectional image 41 generated by the generation function 164 as display images. In this case, the display control function 165 may simultaneously display the VR image 30, the cross-sectional image 40, and the cross-sectional image 41 on the display 103, or display the cross-sectional image 40 and the cross-sectional image 41 without displaying the VR image 30. It may be displayed.

  As described above, the ultrasonic diagnostic apparatus 1 generates and displays a cross-sectional image along the traveling direction of the instrument. According to this, since the instrument is depicted in the cross-sectional image, the ultrasonic diagnostic apparatus 1 can display the positional relationship between the instrument and a structure such as a heart valve in an easily understandable manner.

  Note that the descriptions of the generation function 164 and the display control function 165 are merely examples. For example, the generation function 164 does not necessarily generate an orthogonal two cross-section passing through the catheter traveling direction. Specifically, the cross-sectional image 40 and the cross-sectional image 41 generated by the generation function 164 are not limited to an orthogonal cross section, and may be a cross section that intersects at an arbitrary angle. Further, the cross-sectional image generated by the generation function 164 is not limited to two cross sections, and may be an arbitrary number of cross sections. In addition, for example, the generation function 164 is not limited to a cross-sectional image along the direction of travel of the instrument, and for example, an apical four-chamber (A4C) image, an apical two-chamber (A2C) image, and the like. It may be.

(Modification 2 of the first embodiment)
Further, for example, the ultrasonic diagnostic apparatus 1 may display an X-ray image captured in catheter treatment together with a display image.

  For example, in catheter treatment, the subject P is often seen through an X-ray diagnostic apparatus together with ultrasonic image data from a TEE probe. In this case, in the ultrasonic diagnostic apparatus 1, the display control function 165 displays an X-ray image generated based on the X-ray transmitted through the subject P at the same time as the display image. Specifically, the display control function 165 acquires a fluoroscopic image (X-ray image) photographed by the X-ray diagnostic apparatus, and displays the acquired fluoroscopic image on the display 103 simultaneously with the VR image 30 and the cross-sectional images 40 and 41. Let

  Thus, the ultrasonic diagnostic apparatus 1 displays an X-ray image together with a display image. Since doctors who perform catheter treatment are often accustomed to X-ray images, improvement of interpretation efficiency by doctors is expected.

(Modification 3 of the first embodiment)
In the first embodiment, the case where the ultrasound diagnostic apparatus 1 generates the first-person viewpoint VR image 30 has been described. However, the present invention is not limited to this, and the third-person viewpoint VR image can be generated. The first person viewpoint VR image and the third person viewpoint VR image may be switched. Note that the first-person viewpoint referred to here is a view from the position of the instrument, and generally the instrument itself is not drawn. On the other hand, the third person viewpoint is a view when viewed from a position different from the instrument, and the instrument itself is also depicted. This third person viewpoint is realized by taking a position behind the instrument as the viewpoint.

  The generation function 164 executes a VR process with a position behind the tip of the instrument as a viewpoint, and generates a VR image. For example, the generation function 164 executes the VR process using a position that is a predetermined distance rearward from the distal end portion 10 of the catheter as a viewpoint of the VR process.

  FIG. 7 is a diagram illustrating an example of a display image generated by the generation function 164 according to the third modification of the first embodiment. FIG. 7 illustrates a VR image 50 in which the direction of travel of the catheter is a line of sight and the position at a predetermined distance rearward from the distal end portion 10 of the catheter is the viewpoint.

  As shown in FIG. 7, the generation function 164 generates a VR image 50 by executing a VR process with the direction of travel of the catheter as a line of sight and a position at a predetermined distance rearward from the distal end portion 10 of the catheter. . As a result, the generation function 164 generates, for example, a VR image 50 in which the image 51 of the catheter tip is depicted.

  The generation function 164 switches between generating a first-person viewpoint VR image and generating a third-person viewpoint VR image in accordance with an instruction from the operator. In other words, the generation function 164 generates a VR image with the tip of the instrument as a viewpoint or a VR image with a position behind the tip of the instrument as a viewpoint according to an instruction from the operator.

  As described above, the ultrasonic diagnostic apparatus 1 may include a configuration capable of generating a third-person viewpoint VR image and a configuration capable of switching between a first-person viewpoint VR image and a third-person viewpoint VR image. According to this, for example, the operator can select a viewpoint that is easy to see and display a VR image.

(Second Embodiment)
For example, the ultrasonic diagnostic apparatus 1 can further display an image for guiding the instrument at a position and posture where the instrument is to be installed.

  FIG. 8 is a block diagram illustrating an example of the ultrasonic diagnostic apparatus 1 according to the second embodiment. The ultrasonic diagnostic apparatus 1 according to the second embodiment is different from the ultrasonic diagnostic apparatus 1 illustrated in FIG. 1 in that the processing circuit 160 further includes a valve position / posture estimation function 166. Therefore, in the second embodiment, the description will focus on the differences from the first embodiment, and the same functions as those in the configuration described in the first embodiment are the same as those in FIG. Reference numerals are assigned and description is omitted.

  The valve position / posture estimation function 166 estimates the position and posture where the device should be installed with respect to the position and posture of the heart valve. Note that the position and posture at which the device should be installed with respect to the position and posture of the heart valve are also referred to as “valve position and posture”.

  First, the estimation function 162 estimates the position and posture of the heart valve of the subject P in the three-dimensional medical image data. For example, the estimation function 162 extracts feature points included in the volume data using a supervised machine learning algorithm. Here, the supervised machine learning algorithm is constructed using a plurality of teacher images in which a plurality of feature points included in the heart valve are correctly arranged. Then, the estimation function 162 estimates the position and posture of the heart valve by comparing a model indicating a three-dimensional positional relationship among a plurality of feature points included in the heart valve with the extracted feature points.

  Then, the valve position / posture estimation function 166 estimates the position and posture (valve position / posture) at which the device should be installed with respect to the position and posture of the heart valve. For example, the valve position / posture estimation function 166 estimates the position / posture of the valve using a model of a three-dimensional positional relationship when the instrument is correctly installed on the heart valve. Specifically, the valve position / posture estimation function 166 has a three-dimensional positional relationship when the instrument is correctly installed on the heart valve with respect to the position and posture of the heart valve estimated by the estimation function 162. By fitting the model, the position and posture where the instrument should be installed is estimated.

  The display control function 165 is an image representing the direction in which the position and posture of the appliance estimated by the estimation function 162 approaches or moves away from the position and posture where the appliance estimated by the valve position / posture estimation function 166 is to be installed. Is displayed simultaneously with the display image.

  FIG. 9 is a diagram illustrating an example of a display image displayed by the display control function 165 according to the second embodiment. FIG. 9 illustrates an arrow image 60 indicating the direction in which the current position and orientation of the catheter approaches the position and orientation where the catheter is to be installed.

  As shown in FIG. 9, the display control function 165 uses the three-dimensional coordinates of the current catheter position and orientation and the three-dimensional coordinates of the position and orientation where the instrument is to be installed, A direction approaching the position and posture where the catheter is to be placed is determined. Then, the display control function 165 generates an arrow image 60 indicating the determined direction, and displays it together with the current VR image 30.

  As described above, the ultrasonic diagnostic apparatus 1 can display an image representing a direction approaching the position and orientation where the instrument (catheter) is to be installed. For this reason, the operator can grasp | ascertain intuitively the direction which should position a catheter, for example. In the example of FIG. 9, the case where the arrow image 60 represents the direction approaching the position and orientation where the catheter is to be installed has been described. The contents described in the first embodiment and the modifications can be applied to the second embodiment.

(Modification of the second embodiment)
Further, for example, the ultrasound diagnostic apparatus 1 may display the direction (movement direction) in which the instrument has actually moved together with the display image.

  FIG. 10 is a block diagram illustrating an example of the ultrasonic diagnostic apparatus 1 according to a modification of the second embodiment. The ultrasonic diagnostic apparatus 1 according to the modification of the second embodiment is different from the ultrasonic diagnostic apparatus 1 illustrated in FIG. 1 in that the processing circuit 160 further includes a moving direction determination function 167. . Therefore, in the modification of the second embodiment, the description will focus on the points that differ from the first embodiment, and the points having the same functions as the configuration described in the first embodiment will be described with reference to FIG. The same reference numerals are used and the description thereof is omitted.

  The movement direction determination function 167 determines the movement direction of the instrument based on the position of the instrument at different times. Then, the display control function 165 displays an image representing the movement direction of the instrument determined by the movement direction determination function 167 simultaneously with the display image.

  For example, the movement direction determination function 167 determines the movement direction of the instrument in the coordinate system of the volume data using the position of the instrument in each of the time-series volume data. Specifically, the moving direction determination function 167 is based on the three-dimensional coordinates of the catheter tip 10 in the volume data of the current frame and the three-dimensional coordinates of the catheter tip 10 in the volume data of the previous frame. The moving direction of the distal end portion 10 of the catheter from one frame before is determined. Then, the display control function 165 displays an image representing the movement direction of the instrument determined by the movement direction determination function 167 simultaneously with the display image. Note that an image representing the moving direction is the same as the arrow image 60 illustrated in FIG.

  Thus, the ultrasonic diagnostic apparatus 1 displays the direction in which the instrument has actually moved together with the display image. Therefore, for example, the operator can intuitively understand how the catheter that he / she moved actually moved in the display image.

  The above description is merely an example. For example, the display control function 165 may display an image (an arrow image 60 in FIG. 9) representing a direction approaching a position and orientation where the catheter is to be installed, together with an image representing the actual moving direction of the catheter. According to this, the operator can quickly grasp whether or not the direction in which the instrument is to be positioned matches the actual movement direction of the instrument.

(Third embodiment)
For example, the ultrasonic diagnostic apparatus 1 can further display the movement trajectory until the instrument is positioned at the position and posture where the instrument is to be installed based on the result of the preoperative simulation.

  The ultrasonic diagnostic apparatus 1 according to the third embodiment has the same configuration as the ultrasonic diagnostic apparatus 1 illustrated in FIG. 1, and the processing contents of the processing circuit 160 are partially different. Therefore, in the third embodiment, the description will focus on the points that differ from the first embodiment, and the illustration of the points having the same functions as the configuration described in the first embodiment is omitted.

  The acquisition function 161 acquires a trajectory of movement from the result of the preoperative simulation for the subject P until the instrument is placed on the heart valve of the subject P. Here, the preoperative simulation is a simulation executed prior to catheter treatment, and is performed using, for example, a CT (Computed Tomography) image taken in advance. As a result of the preoperative simulation, for example, a movement trajectory until the catheter is placed on the mitral valve of the subject P is output. The acquisition function 161 acquires the trajectory of movement until the catheter is placed on the mitral valve of the subject P from the result of the preoperative simulation. Note that this movement trajectory is information indicated by the coordinate system of the CT image used in the preoperative simulation.

  The display control function 165 aligns the movement trajectory with respect to the three-dimensional medical image data, and displays an image for guiding the path of the instrument. For example, the display control function 165 aligns the CT image used in the preoperative simulation and the volume data captured by the TEE probe, and converts the coordinate system of the movement trajectory into the coordinate system of the volume data. . Then, the display control function 165 generates an image for guiding the path of the instrument using the movement trajectory after the coordinate conversion. As an example, the display control function 165 superimposes and displays a line image (or a broken line image) representing a trajectory of movement on the VR image 30 shown in FIG. 4 as an image for guiding the path of the instrument. . As another example, the display control function 165 superimposes and displays a line image representing a trajectory of movement on the cross-sectional images 40 and 41 shown in FIG. 6 as an image for guiding the path of the instrument. .

  As described above, the ultrasound diagnostic apparatus 1 displays an image for guiding the path of the instrument. According to this, the operator can view the trajectory of movement until the instrument is positioned at the position and posture where the instrument is to be installed.

(Modification of the third embodiment)
For example, the ultrasonic diagnostic apparatus 1 can further determine and notify a degree appropriate for the placement of the instrument.

  FIG. 11 is a block diagram illustrating an example of an ultrasonic diagnostic apparatus 1 according to a modification of the third embodiment. The ultrasonic diagnostic apparatus 1 according to the modification of the third embodiment has the same configuration as the ultrasonic diagnostic apparatus 1 illustrated in FIG. 1, and the processing circuit 160 further includes a determination function 168 and a notification function 169. Is different. Therefore, in the modified example of the third embodiment, points different from the first embodiment will be mainly described, and the points having the same function as the configuration described in the first embodiment will be described with reference to FIG. The same reference numerals are used and the description thereof is omitted.

  The determination function 168 determines the degree of appropriateness of the indwelling position of the instrument based on a heart valve movement model obtained by preoperative simulation on the subject. Here, in the pre-operative simulation, for example, the motion locus of the heart valve may be calculated in advance as a motion model. This heart valve motion model is a model of the heart valve motion of the subject P that moves with the heartbeat. For example, the pattern of various states accompanying the heart motion, such as the heart valve being closed or opened. Is included.

  For example, the determination function 168 compares the volume data imaged by the TEE probe with the motion model of the heart valve by pattern recognition processing. Then, the determination function 168 calculates the likelihood (similarity) for the pattern in which the heart valve is closed.

  The notification function 169 notifies the determination result by the determination function 168. For example, the notification function 169 notifies the degree appropriate for placement of the instrument based on the likelihood for the pattern of the closed heart valve calculated by the determination function 168. For example, the notification function 169 causes the display 103 to display the calculated likelihood as a degree appropriate for the placement of the appliance. For example, the notification function 169 causes the display 103 to display a score (evaluation) corresponding to the calculated likelihood size. For example, the notification function 169 notifies a predetermined sound when the likelihood exceeds a threshold value.

  In this way, the ultrasonic diagnostic apparatus 1 can determine and notify a degree appropriate for placement of the instrument (catheter). According to this, the operator can grasp | ascertain a suitable timing for placement of an instrument, for example.

(Fourth embodiment)
For example, the ultrasonic diagnostic apparatus 1 can further display the operation direction of the instrument in a coordinate system that is easy for the operator to understand.

  FIG. 12 is a block diagram illustrating an example of the ultrasonic diagnostic apparatus 1 according to the fourth embodiment. The ultrasonic diagnostic apparatus 1 according to the fourth embodiment is different from the ultrasonic diagnostic apparatus 1 illustrated in FIG. 1 in that the processing circuit 160 further includes a detection function 170. Therefore, the fourth embodiment will be described with a focus on differences from the first embodiment, and the same functions as those in the first embodiment are the same as those in FIG. Reference numerals are assigned and description is omitted.

  The detection function 170 detects the position of the three-dimensional medical image data in the three-dimensional space. For example, in the fourth embodiment, an acceleration sensor such as a gyro sensor is attached to the ultrasonic probe 101 that is a TEE probe. Before the acceleration sensor attached to the ultrasonic probe 101 is inserted into the body of the subject P, three-dimensional coordinates in a three-dimensional space (for example, a global coordinate system) are registered. For example, the acceleration sensor registers the three-dimensional coordinates of the position of the mouth of the subject P immediately before being inserted into the esophagus of the subject P. Then, when the ultrasonic probe 101 is inserted into the esophagus of the subject P, the acceleration sensor detects a change in position due to the insertion. For example, the detection function 170 is based on the three-dimensional coordinates registered before the ultrasonic probe 101 is inserted into the body of the subject P and the change in the position detected by the acceleration sensor. Detect the three-dimensional coordinates of the current position.

  Further, for example, when the ultrasound probe 101 is depicted in an X-ray image obtained by imaging the subject P, the detection function 170 determines the position of the ultrasound probe 101 in the three-dimensional space based on the X-ray image. 3D coordinates are detected. Specifically, the detection function 170 is based on the position of the ultrasonic probe 101 in the X-ray image and the positions of the X-ray generator and the X-ray detector used for imaging the X-ray image in the three-dimensional space. Thus, the three-dimensional coordinates of the position of the ultrasonic probe 101 are detected.

  As described above, the detection function 170 detects the three-dimensional coordinates of the position of the ultrasonic probe 101 in the three-dimensional space using at least one of the information from the acceleration sensor and the X-ray image.

  The display control function 165 displays the operation direction of the instrument in the coordinate system of the subject P based on the position in the three-dimensional space detected by the detection function 170 and the position of the subject P with respect to the three-dimensional space. . Here, the position of the subject P with respect to the three-dimensional space is acquired from the position of the bed on which the subject P lies.

  Thereby, the display control function 165 displays the operation direction of the catheter in the coordinate system of the subject P. For example, the display control function 165 represents the operation direction of the catheter on the basis of the body part of the subject P such as the head side, the leg side, the abdomen side, the back side, the left arm side, and the right arm side of the subject. Can be displayed.

(Modification 1 of 4th Embodiment)
Further, for example, the ultrasonic diagnostic apparatus 1 according to the fourth embodiment may display the operation direction of the instrument in the operator's coordinate system.

  For example, the display control function 165 displays the operation direction of the instrument in the operator's coordinate system based on the position in the three-dimensional space detected by the detection function 170 and the position of the operator with respect to the three-dimensional space. Here, the position of the operator with respect to the three-dimensional space is acquired by photographing the room (operating room) in which the operator exists with a camera.

  As a result, the display control function 165 displays the operation direction of the catheter in the coordinate system of the operator. For example, the display control function 165 can display the operation direction of the catheter in expressions such as an upward direction, a downward direction, a right direction, and a left direction as viewed from the operator.

(Modification 2 of the fourth embodiment)
For example, the ultrasonic diagnostic apparatus 1 according to the fourth embodiment may display the operation direction of the instrument in the coordinate system of the X-ray image.

  For example, the display control function 165 displays the operation direction of the instrument in the coordinate system of the X-ray image based on the position in the three-dimensional space detected by the detection function 170 and the position of the X-ray image with respect to the three-dimensional space. . Here, the position of the X-ray image with respect to the three-dimensional space is acquired based on the positions of the X-ray generator and the X-ray detector used for photographing the X-ray image in the three-dimensional space.

  Thereby, the display control function 165 displays the operation direction of the catheter in the coordinate system of the X-ray image. For example, the display control function 165 can display the operation direction of the catheter in an expression such as upward, downward, rightward, leftward on the X-ray image.

(Other embodiments)
In addition to the above-described embodiment, various other forms may be implemented.

(Display after placement)
For example, in the above embodiment, the display until the clip portion 11 of the catheter is placed in the mitral valve has been described. However, the display after placement can be controlled.

  For example, the display control function 165 displays a cross-sectional image including the instrument when the instrument is indwelled. For example, the display control function 165 detects that the instrument has been placed. Specifically, the display control function 165 detects that the clip portion 11 has been placed in the binarized image 20 when the clip portion 11 and the tubular portion 12 of the catheter are separated from each other. When the display control function 165 detects that the clip portion 11 has been placed, the display control function 165 generates and displays cross-sectional images 40 and 41 along the traveling direction of the catheter immediately before placement.

  Thus, the ultrasonic diagnostic apparatus 1 can display a cross-sectional image including the instrument even when the instrument is indwelled.

  Further, each component of each illustrated apparatus is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or a part of the distribution / integration is functionally or physically distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured. Further, all or a part of each processing function performed in each device may be realized by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware by wired logic.

  In addition, among the processes described in the above embodiments and modifications, all or a part of the processes described as being automatically performed can be manually performed, or can be manually performed. All or part of the described processing can be automatically performed by a known method. In addition, the processing procedure, control procedure, specific name, and information including various data and parameters shown in the above-described document and drawings can be arbitrarily changed unless otherwise specified.

  In addition, the ultrasonic imaging method described in the above-described embodiments and modifications can be realized by executing a prepared ultrasonic imaging program on a computer such as a personal computer or a workstation. This ultrasonic imaging method can be distributed via a network such as the Internet. The ultrasonic imaging method may be executed by being recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, an MO, or a DVD, and being read from the recording medium by the computer. it can.

  According to at least one embodiment described above, it is possible to generate a display image in which the position of the instrument relative to the heart valve and the traveling direction can be visually recognized.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 1 Ultrasonic diagnostic apparatus 160 Processing circuit 161 Acquisition function 162 Estimation function 163 Determination function 164 Generation function 165 Display control function

Claims (19)

An acquisition unit for acquiring three-dimensional medical image data of a region including a heart valve of a subject and a catheter inserted into the heart chamber of the subject, imaged using an ultrasonic probe;
The position of the distal end portion included in the three-dimensional medical image data using at least one of shape information representing the shape of the distal end portion of the catheter and reflection characteristic information representing an ultrasonic reflection characteristic of the distal end portion And a determination unit that determines a traveling direction of the tip by obtaining posture information;
A generating unit that generates a display image from the three-dimensional medical image data based on the position and the traveling direction of the tip;
A display control unit for displaying the display image;
An ultrasonic diagnostic apparatus comprising: The tip includes a clip or an artificial valve that is placed in the mitral valve,
The ultrasonic diagnostic apparatus according to claim 1.   The ultrasonic diagnostic apparatus according to claim 1, wherein the display image is an image that three-dimensionally represents the three-dimensional medical image data.   The ultrasonic diagnostic apparatus according to claim 3, wherein the display image is a volume rendering image or a surface rendering image.   The ultrasonic diagnosis according to claim 4, wherein the display image is a volume rendering image or a surface rendering image generated using a viewpoint based on a position of the tip and a line of sight based on a traveling direction of the tip. apparatus. The generation unit generates a cross-sectional image along the traveling direction of the tip as the display image.
The ultrasonic diagnostic apparatus as described in any one of Claims 1-5. A valve position / posture estimation unit that estimates a position and posture where the catheter is to be installed with respect to the position and posture of the heart valve;
The ultrasonic diagnostic apparatus as described in any one of Claims 1-6. The display control unit is an image representing a direction in which the position and posture of the catheter estimated by the estimation unit approaches or moves away from a position and posture where the catheter should be installed estimated by the valve position and posture estimation unit. Is displayed simultaneously with the display image,
The ultrasonic diagnostic apparatus according to claim 7. A moving direction determining unit that determines the moving direction of the catheter based on the position of the catheter at different times;
The display control unit displays an image representing the movement direction of the catheter determined by the movement direction determination unit simultaneously with the display image;
The ultrasonic diagnostic apparatus as described in any one of Claims 1-8. The display control unit displays an X-ray image generated based on X-rays transmitted through the subject simultaneously with the display image;
The ultrasonic diagnostic apparatus as described in any one of Claims 1-9. The acquisition unit acquires a trajectory of movement until the catheter is installed on a heart valve of the subject from a result of preoperative simulation on the subject,
The display control unit performs alignment of the trajectory of the movement with respect to the three-dimensional medical image data, and displays an image for guiding the course of the catheter.
The ultrasonic diagnostic apparatus as described in any one of Claims 1-10. A determination unit that determines how appropriate the indwelling position of the distal end portion is based on a model of the movement of the heart valve obtained by preoperative simulation on the subject; and
An informing unit for informing a determination result by the determining unit;
Further comprising
The ultrasonic diagnostic apparatus as described in any one of Claims 1-11. A detector that detects a position of the three-dimensional medical image data in a three-dimensional space;
The display control unit displays an operation direction of the catheter in the coordinate system of the subject based on the position in the three-dimensional space detected by the detection unit and the position of the subject with respect to the three-dimensional space. ,
The ultrasonic diagnostic apparatus as described in any one of Claims 1-12. A detector that detects a position of the three-dimensional medical image data in a three-dimensional space;
The display control unit displays an operation direction of the catheter in the coordinate system of the operator based on the position in the three-dimensional space detected by the detection unit and the position of the operator with respect to the three-dimensional space. ,
The ultrasonic diagnostic apparatus as described in any one of Claims 1-12. A detector that detects a position of the three-dimensional medical image data in a three-dimensional space;
The display control unit is configured to operate the catheter in the X-ray image coordinate system based on the position in the three-dimensional space detected by the detection unit and the position of the X-ray image with respect to the three-dimensional space. To display,
The ultrasonic diagnostic apparatus according to claim 10. The generation unit generates a volume rendering image with the tip portion as a viewpoint or a volume rendering image with a position behind the tip portion as a viewpoint according to an instruction from an operator.
The ultrasonic diagnostic apparatus as described in any one of Claims 1-15. The three-dimensional medical image data is data imaged by transesophageal echocardiography, which is the ultrasonic probe.
The ultrasonic diagnostic apparatus as described in any one of Claims 1-16. The display control unit displays a cross-sectional image including the tip when the tip is placed.
The ultrasonic diagnostic apparatus as described in any one of Claims 1-17. Acquiring three-dimensional medical image data of a region including a heart valve of a subject and a catheter inserted into the heart chamber of the subject, imaged using an ultrasonic probe;
The position of the distal end portion included in the three-dimensional medical image data using at least one of shape information representing the shape of the distal end portion of the catheter and reflection characteristic information representing an ultrasonic reflection characteristic of the distal end portion And determining the traveling direction of the tip by obtaining posture information,
Generating a display image from the three-dimensional medical image data based on the position of the tip and the traveling direction;
Displaying the display image;
An image processing program that causes a computer to execute each process.

Images