WO2024034557A1

WO2024034557A1 - Puncturing technique assistance system, puncturing assistance image generation device, and puncturing assistance image generation method

Info

Publication number: WO2024034557A1
Application number: PCT/JP2023/028708
Authority: WO
Inventors: 登志一大島; 達夫中本
Original assignee: 学校法人立命館
Priority date: 2022-08-08
Filing date: 2023-08-07
Publication date: 2024-02-15
Also published as: JP2024023038A

Abstract

Provided is a puncturing technique assistance system that makes a puncturing technique easy. The puncturing technique assist system is used to assist puncturing techniques. The puncturing technique assist system includes a probe, a measurement device, and a puncturing assistance image generation device. The probe acquires a patient tomographic image which was freely specified. The measurement device measures a position and an attitude of a puncturing instrument used in the puncturing technique. The puncturing assistance image generation device generates, on the basis of the tomographic image and measurement data from the measurement device, assistance image data indicating a state of the puncturing, and displays the assistance image data on a display. The puncturing assistance image generation device: uses the measurement data to position the puncturing instrument, a fault plane of the probe, and a puncture target point on the body of the patient; indicates the positional relationships, obtained through the positioning, between a puncture needle of the puncturing instrument, the fault plane, and the puncture target point; and generates the assistance image data to be superimposed on the tomographic image.

Description

Puncture technique assistance system, puncture assistance image generation device, and puncture assistance image generation method

The present disclosure relates to a puncture technique assistance system, a puncture assistance image generation device, and a puncture assistance image generation method.

Conventionally, puncture techniques such as intravenous injection and thoracentesis have required highly skilled doctors. For example, both the blood vessel that is the target of puncture and the tip of the puncture needle are small in size. In addition, the puncture needle that has entered the blood vessel and the body cannot be visually recognized. Therefore, the puncture procedure is very difficult. Furthermore, since the condition of the puncture needle inside the body cannot be easily grasped visually, it has been difficult for young doctors to learn the technique. If more doctors can perform puncture procedures easily and in a short time, the burden on patients will be reduced, and patients will be able to receive necessary treatment quickly.

Non-Patent Document 1 proposes a method of measuring the position and orientation of an ultrasound probe using a polygon model constructed in advance from a plurality of ultrasound tomograms, and superimposing and displaying the ultrasound tomograms. .

The present disclosure provides a puncture technique assistance system, a puncture assistance image generation device, and a puncture assistance image generation method that realize assistance in a puncture procedure.

The puncturing technique assisting system of the present disclosure is a puncturing technique assisting system used to assist puncturing techniques. The puncture technique assistance system includes a probe, a measuring device, and a puncture assistance image generation device. The probe acquires an arbitrarily specified tomographic image of the patient. The measuring device measures the position and orientation of a puncture instrument used in a puncture procedure. The puncture auxiliary image generation device generates auxiliary image data indicating the state of puncture based on the tomographic image and measurement data from the measuring device, and displays the auxiliary image data on a display. The puncture auxiliary image generation device uses the measurement data to align the puncture device, the tomographic plane of the probe, and the puncture target point inside the patient's body, and the puncture needle of the puncture device, the tomographic plane, and the puncture target point obtained by the alignment. , generates auxiliary image data that indicates the positional relationship of the puncture target point and is superimposed on the tomographic image.

By using the puncture technique assistance system, puncture assistance image generation device, and puncture assistance image generation method of the present disclosure, it is possible to easily perform a puncture procedure in a short time.

According to the puncture technique assistance system, puncture assistance image generation device, and puncture assistance image generation method of the present disclosure, it is possible to perform a puncture procedure easily and in a short time.

FIG. 1 is a block diagram showing a puncture technique assistance system. FIG. 2 is a schematic diagram illustrating puncturing using a puncturing instrument and an ultrasonic probe. FIG. 2 is a schematic diagram showing the positional relationship of a measuring device, a mounting device, a puncture device, and an ultrasound probe. FIG. 2 is a block diagram showing a puncture auxiliary image generation device. FIG. 2 is a schematic diagram illustrating an input device of the puncture auxiliary image generation device. FIG. 2 is a schematic diagram illustrating a puncturing device and a cone showing first data. It is another schematic diagram explaining a puncture device and 1st data. FIG. 3 is a schematic diagram illustrating a puncture device and a first cone used to generate second data. FIG. 3 is a schematic diagram illustrating a puncturing device and a second cone used to generate second data. FIG. 3 is a schematic diagram illustrating a puncture device and a third cone used to generate second data. It is a schematic diagram explaining the positional relationship between the tip of a puncture needle and a puncture target point. FIG. 2 is a schematic diagram illustrating an example of the positional relationship between the ultrasound probe, the puncture needle, and the puncture target point when the tip of the puncture needle is far from the puncture target point. 8A is a schematic diagram illustrating the positional relationship between the tomographic plane, the puncture target point, and the first cone in the case of FIG. 8A. FIG. This is an example of auxiliary image data in the case of FIG. 8A. FIG. 7 is a schematic diagram illustrating another example of the positional relationship between the ultrasound probe, the puncture needle, and the puncture target point when the tip of the puncture needle approaches the puncture target point. 9A is a schematic diagram illustrating the positional relationship between the tomographic plane, the puncture target point, the first cone, and the second cone in the case of FIG. 9A. FIG. This is an example of auxiliary image data in the case of FIG. 9A. This is an example of auxiliary image data when the puncture needle is not perpendicular to the tomographic plane. FIG. 7 is a schematic diagram illustrating another example of the positional relationship between the ultrasound probe, the puncture needle, and the puncture target point when the tip of the puncture needle is the puncture target point. 10A is a schematic diagram illustrating the positional relationship between the tomographic plane, the puncture target point, the first cone, and the second cone in the case of FIG. 10A. FIG. This is an example of auxiliary image data in the case of FIG. 10A. FIG. 7 is a schematic diagram illustrating another example of the positional relationship between the ultrasound probe, the puncture needle, and the puncture target point after the tip of the puncture needle passes the puncture target point. FIG. 11A is a schematic diagram illustrating the positional relationship between the tomographic plane, the puncture target point, the first cone, and the second cone in the case of FIG. 11A. This is an example of auxiliary image data in the case of FIG. 11A. 12 is a flowchart illustrating a process for generating an auxiliary image for a puncture technique. 12A is a flowchart illustrating a part of the process in FIG. 12A in detail. 3 is a flowchart illustrating processing for generating position and orientation data.

Hereinafter, a puncture technique assistance system, a puncture assistance image generation device, and a puncture assistance image generation method according to the present disclosure will be described with reference to the drawings. Here, the puncture procedure assistance system can be used to assist a doctor in performing a puncture procedure on a patient.

"Puncture procedure" is an act in which a doctor inserts a puncture needle of a puncture device into a patient's body for treatment or diagnosis. In the following, the puncture assist system will be used in a puncture procedure in which a doctor inserts a puncture needle into a blood vessel using a puncture instrument while checking ultrasound tomographic images using an ultrasound device in order to inject a drug into a patient's blood vessel. Let's explain with an example. However, puncture techniques include lumbar puncture, thoracentesis, ascites puncture, etc. in addition to puncturing blood vessels. The puncture technique assistance system of the present disclosure can be used even in these cases. In the following description, the same configurations will be designated by the same reference numerals and the description will be omitted.

"Tomographic plane" refers to a plane inside the human body on which ultrasonic waves are transmitted by an ultrasonic device and an ultrasonic tomographic image is generated.

Note that in the following embodiments, an example will be described in which a tomographic image of a tomographic plane obtained using an ultrasound device is used. However, other tomographic images can be used instead of the tomographic images obtained using the ultrasound device. For example, a tomographic image obtained using an X-ray device may be used. Alternatively, for example, a tomographic image obtained using a magnetic device may be used.

FIG. 1 shows an example of a puncture procedure assistance system according to an embodiment. A puncture technique assistance system 1 according to the present disclosure includes a puncture assistance image generation device 10, a mounting device 30, and a measuring device 20. The mounting device 30 is worn by a user who is a doctor who performs a puncture procedure. The measuring device 20 measures the position and orientation of a specific object used in a puncture procedure. As an example is shown in FIG. 2, this puncture procedure assistance system 1 is used by a doctor who is a user to puncture a patient using an ultrasound device 50 and a puncture instrument 40. The example shown in FIG. 2 shows the positional relationship among the ultrasound probe 51 of the ultrasound device 50, the puncture device 40, and a part of the patient's body. At least the mounting device 30, the measuring device 20, the ultrasound device 50, and the puncture device 40 exist in the same space. However, the puncture auxiliary image generation device 10 may exist in another space as long as it can communicate with the other devices 20 to 40 via a network.

The puncture device 40 has a puncture needle 41 that is inserted into the patient's body. For example, the puncture device 40 is used for the purpose of inserting a puncture needle 41 into a patient's body and injecting a medical solution into the patient's body.

The ultrasound device 50 is used to obtain ultrasound tomographic images of a patient. Specifically, the ultrasound device 50 includes an ultrasound probe 51 and a processing device 52. The ultrasonic probe 51 generates and transmits ultrasonic waves into the patient's body, and receives ultrasonic waves reflected within the body. The processing device 52 uses the ultrasound received by the ultrasound probe 51 to generate an ultrasound tomographic image. Further, the ultrasound device 50 may have a display that displays ultrasound tomographic images. The user can grasp the internal state of the patient, the position of the puncture needle of the puncture instrument 40, etc. from the ultrasound tomographic image generated by the ultrasound device 50. The ultrasound device 50 transmits the ultrasound tomographic image generated by the processing device 52 to the puncture auxiliary image generation device 10.

The measuring device 20 measures the positions and postures of the mounting device 30, the puncture device 40, and the ultrasound probe 51. For example, the measuring device 20 is an optical tracker having a light projecting means 21, a light receiving means 22, and a processing device 23. The light projecting means 21 projects infrared light onto the reflector. The reflector is attached to the mounting device 30, the puncture instrument 40, and the ultrasound probe 51 as a marker. The light receiving means 22 receives the infrared light reflected by the reflector. The processing device 23 processes the infrared light received by the light receiving means 22 and measures the positions and postures of the mounting device 30, the puncture device 40, and the ultrasound probe 51. The measuring device 20 registers in advance the relationship between the shapes of the mounting device 30, the puncture instrument 40, and the ultrasonic probe 51 and the attachment positions of the plurality of reflectors attached to each as a target. Then, the measuring device 20 transmits the infrared light received by the light receiving means 22 to the puncturing auxiliary image generating device 10, the mounting device 30, the puncturing device 40, and the ultrasonic probe 51, which are measured by the processing device 23. The position and orientation of the device are transmitted as measurement data. Here, an example of a passive type is shown in which the measuring device 20 has a light projecting means 21 and a light receiving means 22, and the marker is a reflector, but the present invention is not limited to this. For example, the measuring device 20 may adopt an active type. Specifically, when the marker is a light-emitting means, the measuring device includes only a light-receiving means and does not include a light-emitting means, and measures the position of each marker by receiving the light emitted from the marker with the light-receiving means. This measures the position and orientation of the object.

The light projecting means 21 and the light receiving means 22 are configured as an integrated camera, for example. Here, by acquiring signals from a plurality of directions, the positions and postures of the mounting device 30, puncture instrument 40, and ultrasound probe 51 can be measured. Furthermore, since the positions and postures of the mounting device 30, the puncture device 40, and the ultrasound probe 51 are not fixed but vary, it is necessary to acquire images from multiple directions at the same time. Therefore, the measurement device 20 uses multiple cameras. . Therefore, the measuring device 20 includes multiple sets of light projecting means 21 and light receiving means 22. Specifically, the mounting device 30, the puncture instrument 40, and the ultrasound probe 51 whose positions and postures are to be measured are each equipped with a plurality of reflectors in advance. The light receiving means 22 receives signals emitted from the light projecting means 21 and reflected by the reflector, and measures their positions and orientations.

Here, as an example shown in FIG. 3, the plurality of sets of light projecting means 21 and light receiving means 22 are preferably installed above and in front of the user, specifically, in front of the user's head. In other words, it is preferable to install it not on the back side of the user but above the user's hand side. Furthermore, it is preferable that the plurality of reflectors 60 be attached to the upper part of the ultrasound probe 51, specifically, on the side opposite to the ultrasound transmission/reception side. In this way, by installing the light projecting means 21 and the light receiving means 22 above the user and attaching the reflector 60 above the ultrasound probe 51, it is possible not only to prevent the user from being disturbed during the puncturing procedure. It is possible to prevent the projection of light from the light projecting means 21 and the reception of light by the light receiving means 22 from being obstructed by the user's body or the like. In addition, such an arrangement makes it possible to ensure the transmission and reception of signals, so that the measuring device 20 can prevent a decrease in accuracy in position and orientation measurements. Further, in order to obtain the same effect, it is preferable that the plurality of reflectors 60 of the mounting device 30 be attached above, specifically, on the top side of the user's head when wearing the device. Furthermore, regarding the puncture device 40 as well, it is preferable that the plurality of reflectors 60 be attached above, specifically, on the distal side of the puncture needle 41 in the puncture device 40.

At this time, the positions and postures of the mounting device 30, puncture instrument 40, and ultrasound probe 51 are expressed using a world coordinate system expressed by the X-axis, Y-axis, and Z-axis, respectively. Thereby, the positional relationship between each device can also be specified. Various positions and orientations to be described later are also based on this world coordinate system, but since they are displayed on the display 32 of the mounting device 30 for the user to see, each position and orientation is based on the viewpoint of the user of the mounting device 30. They are also relatively transformed and drawn in the transformed position and orientation.

The puncture auxiliary image generation device 10 generates auxiliary image data indicating the state of puncture by the puncture instrument 40 based on the ultrasound tomographic image received from the ultrasound device 50 and the measurement data received from the measurement device 20. Furthermore, the puncture auxiliary image generation device 10 transmits the generated auxiliary image data to the mounting device 30 and displays it on the display 32. The configuration of the puncture auxiliary image generation device 10 will be described later using FIG. 4A. Here, the auxiliary image data includes first data showing the state of the puncture device 40 three-dimensionally in the surrounding environment image taken by the camera 31 of the mounting device 30, which will be described later, and first data showing the condition of the puncture device 40 in an ultrasound image. and second data two-dimensionally indicating the status of the instrument 40.

The mounting device 30 includes a camera 31 and a display 32. The camera 31 photographs the surrounding environment. The display 32 displays the surrounding environment image photographed by the camera 31 and the auxiliary image data received from the puncture auxiliary image generation device 10. For example, the mounting device 30 is a device that can be mounted on the user's head, and has a display that can display an image for puncturing assistance in conjunction with the direction of the head, such as a head-mounted display or smart glasses. It is. When using smart glasses, it is preferable that the smart glasses have a sense of stability on the user's head when worn, to the same extent as a head-mounted display, since instability will impede the puncturing procedure. The camera 31 is attached to match the user's line of sight. Therefore, the surrounding environment photographed by the camera 31 is an image aligned with the user's line of sight. Specifically, the surrounding environment photographed by the camera 31 is the surrounding environment that can be visually recognized by the user when the mounting device 30 is not worn. If the user is facing forward, the camera 31 will photograph the front of the user, and if the user is facing toward the user, the camera 31 will photograph the user's hand.

It is also preferable that the mounting device 30 includes separate displays 32 for the right eye and for the left eye, and each display 32 displays a stereoscopic image that takes into account actual parallax. In this case, the mounting device 30 may separately include a camera 31 that captures a left-eye image to be displayed on the right-eye display 32 and a camera 31 that captures a left-eye image to be displayed on the left-eye display 32. preferable.

As shown in FIG. 4A, the puncture auxiliary image generation device 10 is an information processing device such as a personal computer that includes a control device 11, a storage device 12, and a communication device 13. Furthermore, the puncture auxiliary image generation device 10 can include an input device 14 and an output device 15.

The control device 11 is a controller that controls the entire puncture auxiliary image generation device 10. For example, the control device 11 reads and executes a computer program stored in the storage device 12, thereby realizing various processes for generating auxiliary image data. The control device 11 may realize a predetermined function by cooperation between a general-purpose computer that is hardware and software, or may be a hardware circuit designed exclusively for realizing a predetermined function. In summary, the control device 11 can be realized by various processors such as a CPU, MPU, GPU, FPGA, DSP, and ASIC.

The storage device 12 is a recording medium that records various information. The storage device 12 is realized by, for example, a RAM, a ROM, a flash memory, an SSD (Solid State Drive), a hard disk drive, another storage device, or an appropriate combination thereof. The storage device 12 stores computer programs executed by the control device 11 and various data used to generate auxiliary image data.

The communication device 13 is a communication means according to the specifications of the external device to enable data communication with external devices (for example, the measuring device 20, the mounting device 30, the ultrasonic device 50, etc.). The above-mentioned data communication is wired and/or wireless data communication, and may be performed according to a communication specification designed specifically for the device or a known communication standard. For example, wired data communication is a communication controller of a semiconductor integrated circuit that operates in accordance with video signal standards such as HDMI (registered trademark), Ethernet (registered trademark) standards, and/or USB (registered trademark) standards, etc. This is done by using it as the device 13. In addition, wireless data communication is based on Bluetooth, which is a short-range wireless communication standard for digital devices, the IEEE802.11 standard for LAN (Local Area Network), and/or the fourth generation so-called 4G/5G for mobile communication. /This is performed by using, as the communication device 13, a communication controller of a semiconductor integrated circuit that operates in accordance with a fifth generation mobile communication system or the like.

The input device 14 may be an operation button, a keyboard, a touch panel, a microphone, etc., which are used for operations for generating and displaying auxiliary image data and inputting data. Since the puncture auxiliary image generation device 10 is used during a puncture procedure, it is not preferable for the user's hands to use input means such as a keyboard. Therefore, even a pedal that the user does not use during a puncture procedure can be used as the input device 14. Specifically, as the input device 14, pedals 141 to 143 as shown in FIG. 4B may be installed in advance at the user's feet. While performing the puncture procedure, the user presses the pedal with his/her foot to perform the desired operation input, thereby generating and displaying auxiliary image data with his or her feet while continuing the puncture procedure without using his/her hands. You can perform the necessary operations. Note that in the example shown in FIG. 4B, three pedals 141 to 143 are provided on a fan-shaped base, but the shape, number, installation method, etc. of the pedals used as the input device 14 are limited to this. Not done.

The output device 15 may be a display, a speaker, or the like that is used to output signals and data for generating and displaying auxiliary image data. The auxiliary image data generated by the puncture auxiliary image generation device 10 is displayed on the display 32 of the mounting device 30. The puncture auxiliary image generation device 10 may display the image displayed on the display 32 on a display, which is the output device 15, so that the image displayed on the display 32 can be viewed by a user other than the user wearing the mounting device 30. This allows, for example, a doctor in training who does not wear the mounting device 30 to easily understand the puncture technique by referring to the image displayed on the display 32 of the mounting device 30.

For example, upon receiving a request signal for generating and displaying auxiliary image data, the puncture auxiliary image generation device 10 executes a series of processes for generating auxiliary image data. Specifically, the puncture auxiliary image generation device 10 executes each process in the control device 11 as a process of generating auxiliary image data. This request for generation and display of auxiliary image data is operated, for example, by pressing the pedal 141 described using FIG. 4B. For example, by pressing the pedal 141, a mode is set in which generation and display of auxiliary image data is enabled, and auxiliary image data is continuously generated and displayed every frame until this mode is canceled. By pressing the pedal 141 again, the auxiliary image data generation and display mode is canceled and the auxiliary image data is not displayed.

The control device 11 acquires measurement data indicating the positions and postures of the mounting device 30, the puncture device 40, and the ultrasound probe 51 from the measurement device 20. Further, the control device 11 generates an ultrasound tomographic image from the ultrasound device 50 . The control device 11 uses the acquired measurement data and the ultrasonic tomographic image to generate auxiliary image data including first data and second data. The first data is information that three-dimensionally indicates the positional relationship between the puncture target point and the puncture needle 41, and is superimposed on the surrounding environment image. The second data is information that indicates the positional relationship between the puncture target point, the puncture needle 41, and the tomographic plane, and is superimposed on the ultrasound tomographic image.

The control device 11 transmits the generated auxiliary image data to the mounting device 30 and displays it on the display 32. At this time, the control device 11 superimposes the ultrasonic tomographic image on the real environment image including at least a part of the patient's body, the puncture device 40, and the ultrasonic probe 51 photographed by the camera 31, and It is preferable to display each image on the display 32 by superimposing auxiliary image data on the sonic tomographic image.

Note that when separate displays 32 are provided for the right eye and for the left eye, the control device 11 performs processing such as generation of auxiliary image data and display of each image data on the display 32 for the right eye in order to generate a stereoscopic image with parallax. It is preferable to perform this separately for the left eye and for the left eye.

The "information indicating the positional relationship" included in the auxiliary image data includes a "guide figure" indicating the positional relationship between the tip of the puncture needle 41 and the puncture target point. Here, the "guide figure" included in the first data is a puncture needle 41 whose apex is the puncture target point and is set by the user's instruction when the tip of the puncture needle 41 contacts the skin to be punctured. It is a cone whose base is a plane perpendicular to the puncture needle 41 with the tip of the puncture needle 41 as its center. In this embodiment, an example in which the cone used as the first data is a cone will be described. Further, the diameter of the bottom surface of the cone is not limited, and can be determined, for example, by the ratio to the height of the cone. Further, the "guide figure" included in the second data is represented by a cross section of a plurality of pyramids provided with the puncture needle 41 as an axis, with the tomographic plane as a reference. In this embodiment, an example will be described in which the plurality of cones used to generate the second data are also cones. Therefore, the cross section of the cone is circular. However, the pyramid is not limited to a cone, but may be a pyramid such as a square pyramid. Note that the tomographic plane is not necessarily obtained perpendicularly to the central axes of the plurality of cones, so it may be an ellipse instead of a perfect circle. Further, in the example described later, the cross section of the first cone is shown by a broken line, the cross section of the second cone is shown by a chain line, and the cross section of the third cone is shown by a chain double dot line.

FIGS. 5A and 5B show cones used as the first data. The cone serving as the first data is set with the puncture target point T as the apex, the puncture target point as the apex, the tip 411 of the puncture needle 41 as the center, and a plane perpendicular to the puncture needle 41 as the base. This cone itself is three-dimensionally superimposed on the surrounding environment image as first data. In the example shown in FIGS. 5A and 5B, the position of the puncture needle 41 is indicated by a solid line Ln, and the extension line of the puncture needle is indicated by a broken line Le. The first data may include these lines Ln and Le. By advancing the puncture needle toward the apex of the cone indicated as the first data in this manner, the tip of the puncture needle 41 can reach the puncture target point.

In FIG. 6A, a broken line shows an example of a cross section of the first cone used to generate the second data, specifically, a vertical cross section along the central axis of the first cone. The first cone has a central axis L of the puncture needle 41 as its central axis, and has an apex P1 located a first predetermined distance L1 behind the tip of the puncture needle 41, and when the tip of the puncture needle 41 approaches the puncture target point. , the cross section of the first cone in the tomographic plane is formed to be small. In other words, the apex P1 of the first cone is located a predetermined distance L1 from the tip of the puncture needle 41 toward the main body of the puncture instrument 40, and the bottom surface of the first cone is located in the puncturing direction. In the following, a cross section taken on a tomographic plane by ultrasound transmitted and received by the first conical ultrasound probe 51 will be referred to as a first cross section. For example, the height of each cone is adjusted as appropriate depending on the puncturing operation during use. More specifically, during puncturing, the length is preferably greater than the distance that the puncturing instrument 40 moves and the distance that the tip of the puncturing needle 41 moves.

FIG. 6B shows an example of a cross section of the second cone, specifically, a vertical cross section (dotted chain line) along the central axis of the second cone. Note that in FIG. 6B, a longitudinal section (broken line) of the first cone is also shown for reference. The second cone has the central axis L of the puncture needle 41 as its central axis, and has an apex P2 located a second predetermined distance L2 ahead from the tip of the puncture needle 41, and when the tip of the puncture needle 41 approaches the puncture target point. The cross section of the second cone at the fault plane is formed to be large. In other words, the apex P2 of the second cone is located on the central axis L of the puncture needle 41 at a predetermined distance L2 from the tip of the puncture needle 41 in the opposite direction to the main body of the puncture instrument 40, and The bottom surface exists on the main body side of the puncture device 40. Hereinafter, the cross section taken by the ultrasonic waves transmitted and received by the second conical ultrasonic probe 51 will be referred to as a second cross section. Note that the first cone and the second cone intersect with each other at a position perpendicular to the central axis at the tip of the puncture needle 41, with their respective cone side surfaces (the oblique sides in the figure) intersecting each other.

FIG. 6C shows an example of a cross section of the third cone, specifically, a vertical cross section (double-dashed line) along the central axis of the third cone. In FIG. 3C, a longitudinal section (dotted chain line) of the second cone is also shown for reference. The third cone has the central axis L of the puncture needle 41 as its central axis, the tip of the puncture needle 41 as the apex P3, and the puncture needle 41 moves away from the puncture target point after the tip of the puncture needle 41 passes the puncture target point. It is formed so that the cross section becomes large. Further, the third cone is formed such that the slope of the cone side surface is gentler than that of the second cone. In other words, the apex P3 of the third cone exists at the tip of the puncture needle 41, and the bottom surface of the third cone exists on the main body side of the puncture instrument 40. Further, since the third cone has a gentler oblique side than the second cone, the oblique side of the second cone and the oblique side of the third cone are closer to the main body of the puncture instrument 40 than the tip of the puncture needle 41. Cross on the sides. Further, the length specified at the intersection of the tip of the puncture needle 41, the oblique side of the second cone, and the oblique side of the third cone is defined as a third predetermined distance L3. Specifically, the second cone and the third cone are located at a position perpendicular to the central axis L at a position retracted a third predetermined distance from the tip of the puncture needle 41 toward the main body of the puncture device 40. The sides of the pyramid (the hypotenuses in the figure) intersect. Here, the cross section taken by the ultrasonic waves transmitted and received by the third conical ultrasonic probe 51 is referred to as the third cross section.

The puncture auxiliary image generation device 10 generates second data of auxiliary image data that includes the position of the puncture target point and the above-mentioned first to third cross sections as guide figures. In FIG. 7, "C" indicates the patient's skin surface, "U" indicates the ultrasonic wave U (position of the tomographic plane) transmitted and received by the ultrasound probe 51, and "T" indicates the puncture target point. At this time, (1) to (5) are respectively (1) the state before puncturing, (2) the state after puncturing where the tip of the puncturing needle 41 is away from the puncturing target point T, and (3) the state of the puncturing needle 41. A state in which the tip of the puncture needle 41 is close to the puncture target point T, (4) a state in which the tip of the puncture needle 41 has reached the puncture target point T, and (5) a state in which the tip of the puncture needle 41 has passed the puncture target point T are shown. An example of auxiliary image data in each of these states will be explained using FIGS. 8A to 11C.

FIG. 8A is a schematic diagram illustrating how a doctor punctures a blood vessel B of a patient while referring to an ultrasound tomographic image obtained by the ultrasound probe 51. In addition, in FIG. 8A, the puncture needle 41 puncturing the patient is shown by a broken line. FIG. 8A shows a state in which puncture has started ((2) in FIG. 7), and before the distance between the tip of the puncture needle 41 and the puncture target point T of the patient's blood vessel B reaches the second predetermined distance L2. Show the situation. Here, the puncture target point T is located on the blood vessel B where the tomographic plane defined by the path of the ultrasound wave U transmitted and received by the ultrasound probe 51 and the extension line of the puncture needle 41 intersect. By changing the inclination of the ultrasound probe 51 as shown in FIG. 8A, it is possible to change the angle of the transmitted and received ultrasound with respect to the patient's skin surface, and accordingly change the range of the ultrasound tomographic image. can be done. Therefore, before puncturing the patient with the puncture needle 41, the doctor changes the inclination of the ultrasound probe 51 and changes the inclination of the puncture instrument 40, as shown by the arrow in FIG. Also move. Thereby, the doctor can find the puncture target point T where the extension line of the puncture needle 41 and the blood vessel B intersect on the tomographic plane defined by the path of the ultrasound wave U.

In the state where the puncture has started as shown in FIG. 8A, the tip of the puncture needle 41 is distal from the puncture target point T, specifically, at a position farther than the predetermined distance L2. As described above using FIGS. 6A to 6C, the first cone is a conical shape that spreads toward the front side of the puncture, with the apex P1 located a predetermined distance L1 behind the tip of the puncture needle 41 as a reference. The second cone has a conical shape that spreads toward the rear of the puncture, with the apex P2 located a predetermined distance L2 ahead of the tip of the puncture needle 41 as a reference. The third cone has a conical shape that widens toward the rear of the puncture needle 41 with the tip of the puncture needle 41 as a reference. Therefore, as shown in FIG. 8B, in the tomographic plane defined by the ultrasonic waves U transmitted and received by the ultrasonic probe 51, only the first cross section is obtained by the first to third cones. In this case, as shown in FIG. 8C, the second data of the auxiliary image data includes only the outer periphery of the first cross section and the puncture target point T. Therefore, the distance between the tip of the puncture needle 41 and the puncture target point T is at least a predetermined distance L2 defined by the apex P2 of the second cone and the intersection of the hypotenuses of the first cone and the second cone. At this time, the second data includes only the puncture target point T and the first cross section. Then, as the distance between the puncture target point T and the tip of the puncture needle 41 becomes smaller, the size of the first cross section becomes smaller. Therefore, the user who confirms the second data, which is the guide figure, can grasp the degree of insertion of the puncture needle 41 according to the size of the first cross section.

FIG. 9A shows a state in which the puncture progresses and the tip of the puncture needle 41 approaches the puncture target point T within the second predetermined distance L2 ((3) in FIG. 7). Here, as described above using FIG. 6B, the second cone is a conical shape that spreads toward the rear side of the puncture needle 41 with reference to the apex P2 at a predetermined position forward from the tip of the puncture needle. Therefore, as shown in FIG. 9B, the cross sections obtained by the plurality of cones in the tomographic plane based on the ultrasonic wave U are a first cross section and a second cross section. When the tip of the puncture needle 41 approaches within the predetermined distance L2 from the puncture target point T in this way, a second cone exists within the first cone, as shown in FIG. 9B. Therefore, as shown in FIG. 9C, the second data of the auxiliary image data includes the outer circumference of the first cross section, the outer circumference of the second cross section existing inside the first cross section, and the puncture target point T. Therefore, the distance between the tip of the puncture needle 41 and the puncture target point T is within a predetermined distance L2 defined by the apex P2 of the second cone and the intersection of the hypotenuses of the first cone and the second cone. When , the second data includes the puncture target point T, the first cross section, and the second cross section located within the first cross section. As the distance between the puncture target point T and the tip of the puncture needle 41 becomes smaller, the size of the first cross section becomes smaller and the size of the second cross section becomes larger. Therefore, the user who confirms the second data, which is the guide figure, can grasp the degree of insertion of the puncture needle 41 according to the sizes of the first cross section and the second cross section. Note that if the puncture needle 41 is no longer perpendicular to the tomographic plane due to a change in the inclination of the ultrasound probe 51, the first cross section and the second cross section will not be a perfect circle, as shown in FIG. 9D. It becomes an ellipse. Therefore, by checking the second data, the user can grasp the degree to which the puncture needle 41 has reached the puncture target point T as well as the degree of inclination of the puncture needle 41.

FIG. 10A shows a state in which the puncture progresses further and the tip of the puncture needle 41 reaches the puncture target point T ((4) in FIG. 7). Here, as described above using FIG. 6B, the first cone and the second cone are arranged at a position perpendicular to the central axis at the tip of the puncture needle 41 so that their hypotenuses overlap. Therefore, when the puncture needle 41 is inserted perpendicularly to the tomographic plane based on the ultrasonic wave U and the tip of the puncture needle 41 reaches the puncture target point T as shown in FIG. The cross section shown is a state in which the first cross section and the second cross section overlap. Therefore, as shown in FIG. 10C, when the tip of the puncture needle 41 reaches the puncture target point T, the second data indicates that the first cross section and the second cross section overlap with the puncture target point T. include. In the second data of FIG. 10C, the state in which the first cross section and the second cross section overlap is shown by a solid line, but the display method is not limited as long as it is possible to understand that they overlap. The user who checks the second data of the auxiliary image data understands that the tip of the puncture needle 41 has reached the puncture target point T by understanding the state in which the first cross section and the second cross section overlap. be able to.

Additionally, the tip of the puncture needle 41 may pass through the puncture target point T. FIG. 11A shows a state in which the tip of the puncture needle 41 has passed through the puncture target point T on the blood vessel B ((5) in FIG. 7). In such a case, as shown in FIG. 11B, the cross sections obtained by the plurality of cones are the first cross section, the second cross section, and the third cross section on the tomographic plane based on the ultrasonic wave U. In this case, as shown in FIG. 11C, the second data of the auxiliary image data includes the puncture target point T, the outer periphery of the first cross section, the outer periphery of the third cross section located on the outer periphery of the first cross section, and the second data of the auxiliary image data. and the outer periphery of the cross section of 2. Here, as described above using FIG. 11C, the second cone and the third cone have different apex positions and different inclinations of the side surfaces of the cone. Therefore, there is a portion where the hypotenuses of the second cone and the third cone intersect. As a result, the second outer periphery is on the outside up to the intersection, but the third outer periphery is on the outside from the intersection. Therefore, when the tip of the puncture needle 41 passes through the puncture target point T, the second data includes the first cross section within the second cross section. Further, depending on the degree of insertion depth of the tip of the puncture needle 41, specifically, when the passage of the puncture target point T of the tip of the puncture needle 41 is shorter than the predetermined distance L3, the third cross section becomes the second cross section. This will be the inside of the cross section. When the passage of the puncture target point T of the tip of the puncture needle 41 is longer than the predetermined distance L3, the second cross section becomes inside the third cross section.

In the examples shown in FIGS. 8C, 9C, 10C, and 11C, the puncture target point T is shown as a cross, the first cross section is shown with a broken line, the second cross section is shown with a dashed line, and the third cross section is shown with a dashed double dotted line. Although the overlap between the first cross section and the second cross section is shown by a solid line, the present invention is not limited thereto. For example, if it is easier to understand the puncture target point T by a dot or a circle instead of a cross, the puncture target point T may be shown by a dot or a circle. For example, if it is easier to understand if all lines are shown as solid lines in different colors, each cross section may be shown as solid lines in different colors.

<How to generate auxiliary images for puncture technique>
A method for generating an auxiliary image for a puncture technique in the puncture auxiliary image generation device 10 according to the present disclosure will be described using the flowcharts shown in FIGS. 12A and 12B.

First, the puncture auxiliary image generation device 10 receives position and orientation data acquired by the measurement device 20 from the measurement device 20 (S001). The position and orientation data includes information indicating the positions and orientations of the puncture device 40, the ultrasound probe 51, and the mounting device 30. Note that acquisition of position and orientation data in the measuring device 20 will be described later using a flowchart shown in FIG.

Next, the puncture auxiliary image generation device 10, for example, acquires video data captured by the camera 31 of the mounting device 30 and displays it on the output device 15 (S002).

The puncture auxiliary image generation device 10 generates data for a calibration process (first calibration data) for the video data acquired in step S002 for a calibration process that specifies the tip of the puncture needle 41 in the puncture instrument 40 later. data). At the same time, the puncture auxiliary image generation device 10 receives data for a calibration process (second calibration data) for a calibration process that specifies the position of the tip of the ultrasound probe 51 (S003). The data for the calibration process (first data for calibration) is "difference correction data" between the position and orientation of the tip of the puncture needle 41 and the position and orientation of the puncture instrument 40 received in step S001. The data for calibration processing (data for second calibration) is "difference correction data" between the position and orientation of the tip of the ultrasound probe 51 and the measured position and orientation of the ultrasound probe 51 received in step S001. be. Specifically, the calibration processing data (first calibration data and second calibration data) received in step S003 is used for calibration processing, that is, the position and measurement of the tips of the puncture needle 41 and the ultrasound probe 51. This is difference correction data that corrects the deviation from the position that was specified. Accepts and saves the calibration processing data set by the user only for the first time. From the second time onward, the data for calibration processing saved at the first time is read out and accepted.

Subsequently, the puncture auxiliary image generation device 10 performs a calibration process for specifying the position of the tip of the puncture needle 41 in the puncture instrument 40, and a calibration process for specifying the position of the tip of the ultrasound probe 51 (S004). The calibration process uses the position and orientation of the puncture instrument 40 and the ultrasound probe 51 included in the position and orientation data received from the measuring device 20 in step S001, and the data for the calibration process received in step S003. Thereby, the tip of the puncture needle 41 and the tip of the ultrasound probe 51 are identified, and subsequent processing is executed.

Additionally, the puncture auxiliary image generation device 10 acquires an ultrasound tomographic image obtained by the ultrasound probe 51 (S005).

Thereafter, the puncture auxiliary image generation device 10 acquires video data of the real environment captured by the camera 31 mounted on the mounting device 30 (S006). The video data of the real environment acquired here includes the state of the real environment that would be visible if the user was not wearing the mounting device 30. Specifically, the video data of the real environment includes the skin near the affected area of the patient, as well as the puncture device 40 and the ultrasound probe 51.

Thereafter, the puncture auxiliary image generation device 10 causes the display 32 of the mounting device 30 to display the video data acquired in step S006 as a real environment video (S007). Here, if the video data captured by the camera 31 is wider than the actual field of view of the user wearing the mounting device 30, the puncture auxiliary image generation device 10, for example, trims the video data acquired in step S006. You may perform processing such as the following.

Furthermore, the puncture auxiliary image generation device 10 causes the display 32 of the mounting device 30 to display the ultrasound tomographic image acquired in step S005 (S008). At this time, the puncture auxiliary image generation device 10 superimposes the ultrasound tomographic image on the corresponding position on the real environment image displayed in step S007. Specifically, the puncture auxiliary image generation device 10 uses the position of the ultrasound probe 51 and the position of the mounting device 30 indicated by the position and orientation data received in step S001 to generate an ultrasound tomographic image on the real environment image. Determine the position.

The puncture auxiliary image generation device 10 generates auxiliary image data (S009). Details of the generation of auxiliary image data will be explained using the flowchart shown in FIG. 12B.

The puncture auxiliary image generation device 10 displays the auxiliary image data generated in step S009 on the display 32 of the mounting device 30 (S010). At this time, the puncture auxiliary image generation device 10 superimposes the first data of the auxiliary image data on the real environment video displayed in step S007. Furthermore, the puncture auxiliary image generation device 10 superimposes the second data of the auxiliary image data on the corresponding position on the ultrasound tomographic image displayed in step S008. Here, the puncture auxiliary image generation device 10 uses the position of the ultrasound probe 51 and the position of the mounting device 30 indicated by the position and orientation data received in step S001, and the position of the ultrasound tomographic image determined in step S010. , determine the position of auxiliary image data on the real environment image and on the ultrasound tomographic image.

Thereafter, while the puncturing procedure continues (YES in S011), the processes of steps S001 to S011 are repeated.

Note that each procedure of the puncture technique assistance method is not limited to the processing order shown in FIG. 12A. For example, the acquisition of the positions and postures of the puncture instrument 40, the ultrasound probe 51, the mounting device 30, etc. may be performed simultaneously. Further, for example, display of the real environment image, ultrasound tomographic image, and auxiliary image data on the display of the mounting device 30 may also be performed simultaneously.

Next, details of the process of generating auxiliary image data shown in step S009 of the flowchart of FIG. 12A will be described using the flowchart shown in FIG. 12B.

When the puncture auxiliary image generation device 10 receives a request for puncture target registration (YES in S101), it registers the intersection of the tomographic plane and the central axis of the puncture needle 41 as a "puncture target point" (S102). Here, the request for "puncture target registration" is generally set at the time when the user, such as a doctor performing the puncture procedure, punctures the skin with the puncture needle at the beginning of the puncture procedure. Furthermore, once a puncture target point is registered, the already registered puncture target point is used for frames of video data that are subsequently acquired. Specifically, registration of the puncture target is performed with the needle tip of the puncture needle 41 in contact with the patient's skin and with the needle tip of the puncture needle 41 directed toward the blood vessel that is the puncture target. The puncture auxiliary image generation device 10 guides the position of the "tomographic plane" where ultrasound waves are transmitted and received based on the position and orientation of the ultrasound probe 51 acquired in step S001. The image representing this tomographic plane is the ultrasound tomographic image acquired in step S005. Furthermore, the puncture auxiliary image generation device 10 guides the "central axis of the puncture needle" in three-dimensional space based on the position and orientation of the puncture instrument 40 acquired in step S001. Furthermore, the puncture auxiliary image generation device 10 registers the intersection of the thus derived "tomographic plane" and the "central axis of the puncture needle" as a "puncture target point." For example, the user can request registration of a puncture target point by pressing the pedal 143 shown in FIG. 4B.

Furthermore, when receiving the instruction to register a puncture target, the puncture auxiliary image generation device 10 registers the position of the tip of the puncture needle 41 as a "puncture starting point" (S103). The puncture auxiliary image generation device 10 determines the "puncture needle tip position" in the three-dimensional space based on the position and orientation of the puncture device 40 acquired in step S001 and the size of the puncture device 40 stored in advance in the storage device 12. ”.

The puncture auxiliary image generation device 10 draws the mark of the puncture target point registered in step S102 (S104). Note that if there is no instruction to register the puncture target (NO in S101), the puncture target point is registered in the previous frame, so the puncture target point is registered based on the puncture target point registered in the previous frame. Draw a mark.

The puncture auxiliary image generation device 10 generates a cone, which is part of the first data, whose axis is a line connecting the puncture start point registered in step S103 to the puncture target point registered in step S102, and whose apex is the puncture target point. is drawn (S105). This cone allows the direction of the puncture needle 41 to be indicated.

The puncture auxiliary image generation device 10 draws a line segment indicating the position of the puncture needle 41 as part of the first data (S106). For example, the puncture auxiliary image generation device 10 draws a line segment indicating the position of the puncture needle 41, as shown by line Ln in FIG. 5B. In relation to the drawing of the main line segment, in order to make the state of the puncture needle 41 visible, the line segment is drawn not as an intersection on a tomographic image but as a line segment for grasping the three-dimensional position of the puncture needle 41. The direction of the main line segment is made to coincide with the direction about the puncture needle 41 as the axis. The puncture auxiliary image generation device 10 guides the position of the puncture needle 41 based on the position and orientation of the puncture instrument 40 acquired in step S001.

The puncture auxiliary image generation device 10 draws a line segment indicating the extension of the puncture needle 41 as part of the first data (S107). For example, the puncture auxiliary image generation device 10 draws a line segment three-dimensionally indicating the position of the extension of the puncture needle 41, as shown by line Le in FIG. 5B. At this time, the puncture auxiliary image generation device 10 draws at least an extension line of the puncture needle 41 from the tip of the puncture needle 41 to the tomographic plane. Also, at this time, the puncture auxiliary image generation device 10 draws an extension line of the puncture needle 41 in a state where at least either the color or the line type is different from the puncture needle 41 drawn in step S106. The puncture auxiliary image generation device 10 guides the extension line of the puncture needle 41 based on the position and orientation of the puncture instrument 40 acquired in step S001.

The puncture auxiliary image generation device 10 draws an intersection mark at the intersection of the central axis of the puncture needle 41 and the tomographic plane as part of the first data (S108). The puncture auxiliary image generation device 10 guides the central axis of the puncture needle 41 based on the position and orientation of the puncture instrument 40 acquired in step S001. Furthermore, the puncture auxiliary image generation device 10 bases the tomographic plane on the position and orientation of the ultrasound probe 51 acquired in step S001. Since the tomographic plane changes in accordance with the inclination of the ultrasound probe 51, the position of the intersection between the central axis of the puncture needle 41 and the tomographic plane also changes in accordance with the change in the tomographic plane. In this way, the puncture auxiliary image generation device 10 generates the first data through the processes of steps S105 to S108.

The puncture auxiliary image generation device 10 generates, as part of the second data, a cross-section in a tomographic plane of a first cone that has its apex at a position a predetermined distance L1 behind the tip of the puncture needle 41 and opens toward the puncture direction. A certain first cross section is drawn (S109). This first cross section shows the depth of the puncture needle 41. Specifically, as the first cross section becomes smaller, the puncture depth of the puncture needle 41 becomes deeper.

The puncture auxiliary image generation device 10 generates, as part of the second data, a tomographic plane of a second cone that opens rearward from the puncture direction side and has its apex at a position a predetermined distance L2 ahead of the tip of the puncture needle 41. A second cross section is drawn (S110). This second cross section shows the degree to which the tip of the puncture needle 41 approaches the tomographic plane. Specifically, when the tip of the puncture needle 41 approaches the puncture target point T within a second predetermined distance L2, the second cross section is drawn, and when the puncture depth of the puncture needle 41 becomes deeper, The second cross section becomes larger.

The puncture auxiliary image generation device 10 generates, as part of the second data, a third cross section that is a cross section of a third cone that has the tip of the puncture needle 41 as its apex and opens rearward from the puncture direction side. is drawn (S111). This third cross section shows the degree to which the tip of the puncture needle 41 has passed through the tomographic plane. Specifically, when the tip of the puncture needle 41 passes the puncture target point T, a third cross section is drawn, and as the passing distance increases, the third cross section becomes larger. In this way, the second data is generated by the processing of steps S109 to S111.

<How to generate position and orientation data>
Next, an example of a method for generating position and orientation data in the measuring device 20 will be described using the flowchart shown in FIG. 13.

First, the measuring device 20 acquires image data using a plurality of light projecting means 21 and light receiving means 22 (S201). Here, markers used for specifying the positions and postures of the puncture device 40, the ultrasound probe 51, and the mounting device 30 are attached in advance to the puncture device 40, the ultrasound probe 51, and the mounting device 30. Further, the image data observed by the light receiving means 22 includes markers attached to the puncture device 40, the ultrasound probe 51, and the mounting device 30.

The measuring device 20 generates position and orientation data in the processing device 23 using the image data acquired in step S201 (S202). Specifically, the processing device 23 processes the markers attached to the puncture device 40, the ultrasound probe 51, and the mounting device 30 for each corresponding frame of a plurality of image data, that is, for each frame acquired at the same timing. Detect position and orientation. The processing device 23 also generates position and orientation data. The position and orientation data indicates the positions and orientations of the puncture instrument 40, the ultrasound probe 51, and the mounting device 30 using the unified coordinate system as described above. For example, when the wearing device 30 is a head-mounted display, the puncture auxiliary image generation device 10 can determine the position and posture of the head of the user who wears the wearing device 30. It is possible to determine which direction the user's head is facing.

The measurement device 20 transmits the position and orientation data generated in step S202 to the puncture auxiliary image generation device 10 (S005). The position and orientation data transmitted by the measuring device 20 in step S202 is received by the puncture auxiliary image generation device 10 in step S001.

Note that the generation and transmission of position and orientation data by the measuring device 20 shown in FIG. 13 are repeatedly and continuously performed, for example, while the puncturing auxiliary image generation device 10 assists the puncturing procedure. Specifically, the process shown in the flowchart shown in FIG. 13 continues to operate asynchronously with the process shown in the flowchart shown in FIG. 12A while the process shown in the flowchart shown in FIG. 12A is executed. Therefore, the puncture auxiliary image generation device 10 uses the position and orientation data continuously received from the measurement device 20 to acquire the positions and orientations of the puncture instrument 40, ultrasound probe 51, and mounting device 30 in real time, and performs puncture. It can be used to generate auxiliary images for procedures.

As described above, the puncture auxiliary image generation device 10 generates first data that three-dimensionally indicates the positional relationship between the puncture target point T and the puncture needle 41, which is generated according to the puncture target point T and the puncture start point. generate. In addition, the puncture auxiliary image generation device 10 displays first to third cross sections whose display states differ depending on the distance between the puncture needle 41 and the puncture target point T and the depth of the puncture needle 41, as well as the puncture target point T. generating second data containing the second data; Furthermore, the puncture auxiliary image generation device 10 displays the first data superimposed on the surrounding environment image, and displays the first data superimposed on the ultrasound tomographic image. In other words, the puncture technique assistance system 1 superimposes and displays auxiliary image data including first data and second data representing the puncture situation in addition to the ultrasonic tomographic image on the user's environmental video. , to realize mixed reality (MR). As a result, the user of the puncture procedure assistance system 1 can check the auxiliary image data, which is a guide figure that indicates the distance between the puncture target point T and the puncture needle 41, which cannot be fully grasped using ultrasound tomograms alone. The puncturing situation of the puncturing needle 41 can be grasped.

The puncture procedure assistance system, puncture assistance image generation device, and puncture assistance image generation method of the present disclosure are useful for assisting puncture procedures.

1 Puncture technique assistance system 10 Puncture assistance image generation device 11 Control device 12 Storage device 13 Communication device 14 Input device 15 Output device 20 Measuring device 21 Light projecting means 22 Light receiving means 23 Processing device 30 Mounting device 31 Camera 32 Display 40 Puncture instrument 41 Puncture needle 50 Ultrasonic device 51 Ultrasonic probe 52 Processing device

Claims

A puncture procedure assistance system used to assist puncture procedures,
a probe that acquires an arbitrarily specified tomographic image of a patient;
A measuring device that measures the position and posture of a puncture device used in a puncture procedure;
a puncture auxiliary image generation device that generates auxiliary image data indicating a puncture state based on the tomographic image and measurement data by the measuring device, and displays the auxiliary image data on a display;
Equipped with
The puncture auxiliary image generation device includes:
using the measurement data to align the puncture device, the tomographic plane of the probe, and the puncture target point within the patient's body;
Indicating the positional relationship between the puncture needle of the puncture device, the tomographic plane, and the puncture target point obtained by the alignment, and generating the auxiliary image data to be superimposed on the tomographic image;
Puncture procedure assistance system.
A puncture procedure assistance system used to assist puncture procedures,
an ultrasound probe that obtains ultrasound tomographic images of a patient;
A measuring device that measures the position and posture of a puncture device used in a puncture procedure;
a puncture auxiliary image generation device that generates auxiliary image data indicating a puncture state based on the ultrasound tomogram and measurement data by the measuring device, and displays the auxiliary image data on a display;
Equipped with
The puncture auxiliary image generation device includes:
using the measurement data to align the puncture device, the tomographic plane of the ultrasound probe, and the puncture target point within the patient's body;
Indicating the positional relationship between the puncture needle of the puncture device, the tomographic plane, and the puncture target point obtained by the alignment, and generating the auxiliary image data to be superimposed on the ultrasound tomographic image;
Puncture procedure assistance system.
further comprising a mounting device that has the display and a camera that captures images of the surrounding environment and is worn by a user performing a puncture procedure,
The measuring device is
measuring the position and orientation of the mounting device together with the ultrasound probe and the puncture device;
The puncture auxiliary image generation device includes:
3. The auxiliary image data is superimposed and displayed on a real environment image including the patient, the puncture device, and the ultrasound probe captured by the camera, at a position that takes into account the measurement results of the measurement device. puncture procedure assistance system.
A puncture auxiliary image generation device that includes a calculation device and generates auxiliary image data used to assist a puncture procedure,
The arithmetic device is
Receives measurement data from the measurement device that measures the position and posture of the ultrasound probe that acquires ultrasound tomographic images of the patient and the puncture instrument used in the puncture procedure,
Using the measurement data, aligning the puncture device, the tomographic plane of the ultrasound probe, and the puncture target point within the patient's body;
receiving an ultrasound tomographic image from the ultrasound probe;
generating the auxiliary image data to be superimposed on the ultrasound tomographic image, indicating the positional relationship between the puncture needle of the puncture device, the tomographic plane, and the puncture target point obtained by the alignment;
displaying the auxiliary image data on a display device on which the ultrasound tomographic image is displayed;
A puncture auxiliary image generation device that performs
The puncture auxiliary image generation device according to claim 4, wherein the information representing the positional relationship includes a guide figure indicating a distance between the tip of the puncture needle and the puncture target point located on the tomographic plane.
The puncture auxiliary image generation device according to claim 5, wherein the guide figure is represented by a cross section of a plurality of pyramids provided with the puncture needle as an axis, with the tomographic plane as a reference.
The plurality of cones are:
With the central axis of the puncture needle as the central axis, the apex is located a first predetermined distance backward from the tip of the puncture needle, and when the tip of the puncture needle approaches the puncture target point, the cross section on the tomographic plane a first cone that becomes smaller;
When the apex is located a second predetermined distance forward from the tip of the puncture needle with the central axis of the puncture needle as the central axis, and the tip of the puncture needle approaches the puncture target point, the cross section on the tomographic plane a second cone that grows larger;
The central axis of the puncture needle is the central axis, the tip of the puncture needle is the apex, and the slope of the side surface of the cone is gentler than that of the second cone, and when the tip of the puncture needle passes the puncture target point, , a third cone whose cross section on the tomographic plane becomes larger,
The information representing the positional relationship is
When the distance between the tip of the puncture needle and the puncture target point is at least the second predetermined distance, only the first cross section of the first cone is included;
When the distance between the tip of the puncture needle and the puncture target point is less than the second predetermined distance, a second cross section of the second cone is included in the first cross section,
When the tip of the puncture needle contacts the puncture target point, the first cross section and the second cross section are overlapped,
When the tip of the puncture needle passes the puncture target point, the first cross section is included in the second cross section, and the third conical section is included depending on the insertion depth of the tip of the puncture needle. The puncture auxiliary image generation device according to claim 6, wherein the third cross section of the body is displayed in a large size.
The puncture auxiliary image generation device according to claim 7, wherein the plurality of cones are cones or regular pyramids.
The puncture auxiliary image generation device according to claim 5, wherein the guide figure is a cone with the puncture target point as the apex and the tip of the puncture needle as the center of the bottom surface.
The puncture auxiliary image generation device according to claim 9, wherein the cone is a cone or a regular pyramid.