JP2023172310A - Image processing device, method, and program - Google Patents

Abstract

To provide an image processing device, method, and program capable of speedily and accurately positioning a three-dimensional image and a radiation image.SOLUTION: A processor acquires a three-dimensional image and a radiation image about an identical subject, performs rigid body positioning of the three-dimensional image and the radiation image, derives an error of a respiratory phase between the three-dimensional image and the radiation image subjected to the rigid body positioning, determines whether or not the error is less than a predetermined threshold, and when the determination is negative, performs non-rigid body positioning of the three-dimensional image and the radiation image.SELECTED DRAWING: Figure 3

Description

The present disclosure relates to an image processing device, method, and program.

An ultrasound endoscope with an endoscopic observation section and an ultrasound observation section at the tip is inserted into a lumen such as the digestive tract or bronchus of a subject, and endoscopic images of the inside of the lumen and the outside of the lumen wall are obtained. Ultrasonic images of lesions and other parts of the body are being taken. Biopsies are also performed in which tissue of a lesion located outside the lumen wall is collected using a treatment instrument such as forceps.

When performing a treatment using such an ultrasound endoscope, it is important to accurately reach the target position within the subject's body with the ultrasound endoscope. For this reason, by performing fluoroscopic imaging, in which radiation is continuously irradiated from a radiation source to the subject during the procedure and the resulting fluoroscopic images are displayed in real time, the position of the ultrasound endoscope and the human body structure can be adjusted. The relationship is being understood.

Here, since the fluoroscopic image includes overlapping anatomical structures such as organs, blood vessels, and bones within the subject, it is not easy to recognize lumens and lesions. For this reason, a three-dimensional image of the subject is acquired before the treatment using a CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, etc., the lesion position is identified in the three-dimensional image, and the three-dimensional image and fluoroscopic image are The position of a lesion is specified in a fluoroscopic image by performing position alignment with the fluoroscopic image (see, for example, Patent Document 1). Furthermore, various methods have been proposed for performing rigid body positioning or non-rigid body positioning of a three-dimensional image and a perspective image (see, for example, Patent Documents 2 and 3).

JP2020-137796A Japanese Patent Application Publication No. 2008-093443 Japanese Patent Application Publication No. 2014-135974

Rigid body positioning requires a small amount of calculation, so positioning can be performed relatively quickly, but the accuracy of positioning is not so high. On the other hand, non-rigid positioning has high positioning accuracy, but requires a large amount of calculation, so it takes time to process.

The present invention has been made in view of the above circumstances, and it is an object of the present invention to enable alignment of a three-dimensional image and a two-dimensional radiation image at high speed and with high precision.

An image processing device according to a first aspect of the present disclosure includes at least one processor, the processor acquires a three-dimensional image and a radiation image of the same subject,
Rigid alignment of the 3D image and radiographic image,
Derive the error in the respiratory phase between the rigidly aligned three-dimensional image and the radiographic image,
determining whether the error is less than a predetermined threshold;
If the determination is negative, non-rigid alignment between the three-dimensional image and the radiographic image is performed.

An image processing apparatus according to a second aspect of the present disclosure may be the image processing apparatus according to the first aspect, in which the three-dimensional image and the radiation image include a lung field region of the subject.

An image processing device according to a third aspect of the present disclosure is the image processing device according to the second aspect, wherein the processor extracts a lung field region from each of the three-dimensional image and the radiation image;
Non-rigid alignment between the three-dimensional image and the radiation image may be performed based on the extracted lung field region.

In the image processing apparatus according to a fourth aspect of the present disclosure, in the image processing apparatus according to the third aspect, when the determination is negative, the processor extracts a lung field region from each of the three-dimensional image and the radiation image. It may be something that does.

An image processing device according to a fifth aspect of the present disclosure is an image processing device according to the third aspect, in which the processor extracts a lung field region from each of the three-dimensional image and the radiation image before making the determination. It may be.

An image processing apparatus according to a sixth aspect of the present disclosure is an image processing apparatus according to any one of the second to fifth aspects, in which the processor includes a body axis of the subject of the diaphragm included in the three-dimensional image. The error in the respiratory phase may be derived based on the difference between the position in the direction and the position of the diaphragm included in the radiographic image in the body axis direction of the subject.

In the image processing apparatus according to a seventh aspect of the present disclosure, in the image processing apparatus according to the sixth aspect, the processor uses the learned model to detect the diaphragm in the body axis direction of the subject included in the three-dimensional image. The position and the position of the diaphragm included in the radiographic image in the body axis direction of the subject may be derived.

An image processing apparatus according to an eighth aspect of the present disclosure is an image processing apparatus according to any one of the second to fifth aspects, in which the processor is configured to project a three-dimensional image in a radiographic imaging direction. The error in the respiratory phase may be derived based on the difference between the area of the lung field region included in the image and the area of the lung field region included in the radiographic image.

In the image processing device according to a ninth aspect of the present disclosure, in the image processing device according to any one of the first to eighth aspects, when the determination is negative, the processor performs non-rigid alignment. The three-dimensional image and the radiographic image may be displayed in a superimposed manner, and if the determination is affirmative, the three-dimensional image and the radiographic image, which have been rigidly aligned, may be displayed in a superimposed manner.

The image processing method of the present disclosure acquires a three-dimensional image and a radiation image of the same subject,
Rigid alignment of the 3D image and radiographic image,
Derive the error in the respiratory phase between the rigidly aligned three-dimensional image and the radiographic image,
determining whether the error is less than a predetermined threshold;
If the determination is negative, non-rigid alignment between the three-dimensional image and the radiographic image is performed.

The image processing program of the present disclosure includes a procedure for acquiring a three-dimensional image and a radiation image of the same subject;
A procedure for rigid body alignment of a three-dimensional image and a radiographic image;
A procedure for deriving a respiratory phase error between a rigidly aligned three-dimensional image and a radiographic image;
a step of determining whether the error is less than a predetermined threshold;
If the determination is negative, the computer is caused to execute a procedure for performing non-rigid position alignment between the three-dimensional image and the radiographic image.

According to the present disclosure, a three-dimensional image and a radiation image can be aligned at high speed and with high precision.

A diagram showing a schematic configuration of a medical information system to which an image processing device according to a first embodiment of the present disclosure is applied. A diagram showing a schematic configuration of an image processing device according to the first embodiment Functional configuration diagram of the image processing device according to the first embodiment A diagram schematically showing processing performed by the image processing device according to the first embodiment Diagram to explain detection of differences in diaphragm position Diagram showing the display screen Flowchart showing processing performed in the first embodiment Flowchart showing processing performed in the second embodiment

Embodiments of the present disclosure will be described below with reference to the drawings. First, the configuration of a medical information system to which the image processing apparatus according to the first embodiment is applied will be described. FIG. 1 is a diagram showing a schematic configuration of a medical information system. In the medical information system shown in FIG. 1, a computer 1 including an image processing device according to the first embodiment, a three-dimensional image capturing device 2, a fluoroscopic image capturing device 3, and an image storage server 4 communicate with each other via a network 5. Connected as possible.

The computer 1 includes the image processing apparatus according to the first embodiment, and has the image processing program according to the first embodiment installed therein. The computer 1 is installed in a treatment room where a treatment is performed on a subject H, as will be described later. The computer 1 may be a workstation or personal computer directly operated by a medical professional who performs treatment, or may be a server computer connected thereto via a network. The image processing program is stored in a storage device of a server computer connected to a network or in a network storage in a state that can be accessed from the outside, and is downloaded and installed in the computer 1 used by the doctor according to a request. Alternatively, it is recorded and distributed on a recording medium such as a DVD (Digital Versatile Disc) or a CD-ROM (Compact Disc Read Only Memory), and installed on the computer 1 from the recording medium.

The three-dimensional image capturing device 2 is a device that generates a three-dimensional image representing the region to be diagnosed by photographing the region of the subject H. Specifically, the three-dimensional imaging device 2 is a device that generates a three-dimensional image representing the region to be diagnosed. (Positron Emission Tomography) equipment, etc. A three-dimensional image composed of a plurality of tomographic images generated by the three-dimensional image capturing device 2 is transmitted to the image storage server 4 and stored. In this embodiment, the treatment target region of the subject H is the lungs, the three-dimensional image capturing device 2 is a CT device, and as will be described later, the chest of the subject H is By photographing, a CT image including the chest of the subject H is obtained in advance as a three-dimensional image and stored in the image storage server 4.

The fluoroscopic image capturing device 3 includes a C-arm 3A, an X-ray source 3B, and an X-ray detector 3C. The X-ray source 3B and the X-ray detector 3C are respectively attached to both ends of the C-arm 3A. In the fluoroscopic image photographing device 3, the C-arm 3A is configured to be rotatable and movable so that the subject H can be photographed from any direction. Then, the fluoroscopic image capturing device 3 continuously irradiates the subject H with X-rays at a predetermined frame rate during the treatment of the subject H, as will be described later. X-ray images of the subject H are sequentially acquired by sequentially performing fluoroscopic imaging using the X-ray detector 3C. In the following description, the sequentially acquired X-ray images will be referred to as fluoroscopic images. A fluoroscopic image is an example of a radiographic image according to the present disclosure.

The image storage server 4 is a computer that stores and manages various data, and includes a large-capacity external storage device and database management software. The image storage server 4 communicates with other devices via a wired or wireless network 5 and sends and receives image data and the like. Specifically, various data including the three-dimensional image acquired by the three-dimensional image capturing device 2 and the image data of the fluoroscopic image acquired by the fluoroscopic image capturing device 3 are acquired via a network, and are stored in a large-capacity external storage device, etc. Save and manage the information on a recording medium. Note that the storage format of image data and the communication between each device via the network 5 are based on a protocol such as DICOM (Digital Imaging and Communication in Medicine).

In this embodiment, while performing fluoroscopic imaging of the subject H, a part of a lesion such as a pulmonary nodule existing in the lungs of the subject H, which has been detected in advance using the three-dimensional image V0, is sampled. A biopsy procedure will be performed to determine the presence of disease. For this reason, the fluoroscopic image capturing device 3 is placed in a treatment room for performing treatment. Further, an ultrasonic endoscope device 6 is installed in the treatment room. The ultrasonic endoscope device 6 includes an endoscope 6A having an ultrasonic probe and a treatment tool such as forceps attached to its tip. In this embodiment, in order to perform a biopsy of a lesion, the operator inserts the endoscope 6A into the bronchus of the subject H, photographs a fluoroscopic image of the subject H using the fluoroscopic image capturing device 3, and While displaying the image and the endoscopic image taken by the endoscope 6A in real time, confirm the tip position of the endoscope 6A within the subject H in the fluoroscopic image, and move the endoscope 6A to the target lesion position. move the tip of the

Here, lung lesions such as pulmonary nodules occur outside the bronchi rather than inside the bronchi. Therefore, after moving the tip of the endoscope 6A to the target position, the operator takes an ultrasound image of the outside of the bronchi with an ultrasound probe, displays the ultrasound image, and identifies the location of the lesion in the ultrasound image. While checking, perform a procedure to collect a part of the lesion using a treatment tool such as forceps.

Next, an image processing apparatus according to the first embodiment will be described. FIG. 2 is a diagram showing the hardware configuration of the image processing apparatus according to the first embodiment. As shown in FIG. 2, the image processing device 10 includes a CPU (Central Processing Unit) 11, a nonvolatile storage 13, and a memory 16 as a temporary storage area. The image processing device 10 also includes a display 14 such as a liquid crystal display, an input device 15 such as a keyboard and a mouse, and a network I/F (InterFace) 17 connected to the network 5. The CPU 11, storage 13, display 14, input device 15, memory 16, and network I/F 17 are connected to the bus 18. Note that the CPU 11 is an example of a processor in the present disclosure.

The storage 13 is realized by an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, or the like. An image processing program 12 is stored in a storage 13 as a storage medium. The CPU 11 reads the image processing program 12 from the storage 13, loads it into the memory 16, and executes the loaded image processing program 12.

Next, the functional configuration of the image processing device according to the first embodiment will be explained. FIG. 3 is a diagram showing the functional configuration of the image processing device according to the first embodiment. FIG. 4 is a diagram schematically showing processing performed by the image processing apparatus according to the first embodiment. As shown in FIG. 3, the image processing device 10 includes an image acquisition section 21, a first alignment section 22, an error derivation section 23, a determination section 24, an extraction section 25, a second alignment section 26, and a display control section 27. . Then, when the CPU 11 executes the image processing program 12, the CPU 11 executes the image acquisition section 21, the first alignment section 22, the error derivation section 23, the determination section 24, the extraction section 25, the second alignment section 26, and the display. It functions as a control section 27.

The image acquisition unit 21 acquires a three-dimensional image V0 of the subject H from the image storage server 4 according to instructions from the input device 15 by the operator. Further, the image acquisition unit 21 sequentially acquires fluoroscopic images T0 acquired by the fluoroscopic image capturing device 3 during treatment of the subject H.

The first alignment unit 22 aligns the sequentially acquired perspective images T0 and three-dimensional images V0. Here, the fluoroscopic image T0 is a two-dimensional radiation image. Therefore, the first alignment unit 22 aligns the two-dimensional image and the three-dimensional image. In this embodiment, the first positioning unit 22 first projects the three-dimensional image V0 in the same direction as the photographing direction of the fluoroscopic image T0 to derive a two-dimensional pseudo fluoroscopic image VT0. Then, the first alignment unit 22 rigidly aligns the two-dimensional pseudo perspective image VT0 with the perspective image T0. Any method such as affine transformation can be used as a method for rigid body alignment. For example, when affine transformation is used, the first alignment unit 22 applies The amount of parallel movement and the amount of rotation are derived as rigid body alignment information. The first alignment unit 22 then derives a modified three-dimensional image V1 by transforming the three-dimensional image V0 using the rigid body alignment information.

The error deriving unit 23 derives a respiratory phase error D0 between the rigidly aligned three-dimensional image V0, that is, the deformed three-dimensional image V1, and the perspective image T0. For this purpose, the error deriving unit 23 calculates the position of the diaphragm in the body axis direction of the subject H included in the deformed three-dimensional image V1 derived by the first alignment unit 22, and the position of the diaphragm in the body axis direction of the subject H included in the fluoroscopic image T0. The difference from the position of H in the body axis direction is derived.

Specifically, as shown in FIG. 5, the error derivation unit 23 projects the deformed three-dimensional image V1 in the same direction as the photographing direction of the fluoroscopic image T0 to derive a two-dimensional pseudo-deformed fluoroscopic image VT1. Then, the error deriving unit 23 aligns the bone portions in the pseudo-deformed fluoroscopic image VT1 and the fluoroscopic image T0, and derives the difference in the position of the diaphragm after the alignment. As the difference in the position of the diaphragm, a representative value of the difference between the position of the diaphragm 31 in the pseudo-deformed fluoroscopic image VT1 and the position of the diaphragm 32 in the fluoroscopic image T0 can be used. Representative values include maximum values, minimum values, average values, intermediate values, and the like.

The error deriving unit 23 calculates the difference between the position of the diaphragm in the body axis direction of the subject H included in the deformed three-dimensional image V1 and the position of the diaphragm in the body axis direction of the subject H included in the fluoroscopic image T0. The learned model to be derived may be used to derive the difference in the position of the diaphragm. Such a trained model is constructed by learning a neural network using training data of a three-dimensional image and a perspective image in which the difference in the position of the diaphragm in the body axis direction is known.

The determining unit 24 determines whether the error D0 derived by the error deriving unit 23 is less than a predetermined threshold Th1.

If the determination by the determination unit 24 is negative, the extraction unit 25 extracts the lung field region from the three-dimensional image V0 and the fluoroscopic image T0. The lung field region is the soft tissue of the lungs included in the three-dimensional image V0 and the fluoroscopic image T0. In the present embodiment, the extraction unit 25 extracts a lung field region from the three-dimensional image V0 and the fluoroscopic image T0 using a well-known computer aided image diagnosis (CAD) algorithm.

If the determination by the determination unit 24 is negative, the second alignment unit 26 performs non-rigid alignment between the three-dimensional image V0 and the perspective image T0. Specifically, the 3D image V0 and the fluoroscopic image are transformed by transforming the 3D image V0 so that the lung field region extracted from the 3D image V0 matches the lung field region extracted from the fluoroscopic image T0. Non-rigid alignment is performed with T0, and non-rigid alignment information representing the amount of deformation of the three-dimensional image V0 with respect to the perspective image T0 is derived. The second alignment unit 26 then deforms the three-dimensional image V0 using the non-rigid alignment information to derive a modified three-dimensional image V2.

Note that non-rigid positioning is performed by using, for example, functions such as B-splines and thin plate splines to determine the correspondence between the lung field region in the pseudo-deformed fluoroscopic image VT1 and the lung field region in the fluoroscopic image T0 for the deformed three-dimensional image V1. A method based on nonlinear transformation of points can be used, but is not limited thereto. Any method can be used, such as a method that performs non-rigid registration using a trained model.

If the determination by the determination unit 24 is negative, the display control unit 27 displays the modified three-dimensional image V2 in a superimposed manner on the perspective image T0, and displays the superimposed image on the display 14. On the other hand, if the determination by the determination unit 24 is affirmative, that is, if the error D0 is less than the threshold value Th1, the display control unit 27 displays the deformed three-dimensional image V1 in a superimposed manner on the perspective image T0, and The image is displayed on the display 14. In this case, the display control unit 27 projects the deformed three-dimensional images V1 and V2 in the same direction as the photographing direction of the fluoroscopic image T0 to derive two-dimensional pseudo-deformed fluoroscopic images VT1 and VT2, and The modified perspective images VT1 and VT2 are displayed in a superimposed manner. In addition, in FIG. 4, when the determination as to whether the error D0 is less than the threshold value Th1 is affirmative, by pointing the broken line arrow toward the modified 3-dimensional image V1, the modified 3-dimensional image V1 and the perspective This indicates that the image is displayed superimposed on the image T0.

FIG. 6 is a diagram showing a display screen of superimposed images. As shown in FIG. 6, a superimposed image 41 in which a modified three-dimensional image V1 or a modified three-dimensional image V2 is superimposed on a perspective image T0 is displayed on the display screen 40.

Next, the processing performed in the first embodiment will be explained. FIG. 7 is a flowchart showing the processing performed in the first embodiment. First, the image acquisition unit 21 acquires the three-dimensional image V0 from the image storage server 4 (step ST1), and then the image acquisition unit 21 acquires the perspective image T0 (step ST2). Then, the first alignment unit 22 rigidly aligns the three-dimensional image V0 and the perspective image T0 (step ST3). Subsequently, the error deriving unit 23 derives an error D0 in the respiratory phase between the rigidly aligned three-dimensional image, that is, the deformed three-dimensional image V1 and the fluoroscopic image T0 (step ST4), and the determining unit 24 calculates the error D0. It is determined whether or not is less than a predetermined threshold Th1 (step ST5).

If step ST5 is negative, the extraction unit 25 extracts the lung field region from the three-dimensional image V0 and the fluoroscopic image T0 (step ST6), and the second positioning unit 26 extracts the three-dimensional image V0 based on the lung field region. Non-rigid positioning is performed between the image T0 and the perspective image T0 (step ST7). Then, the display control unit 27 displays the non-rigidly aligned three-dimensional image, that is, the deformed three-dimensional image V2 and the perspective image T0 in a superimposed manner (step ST8), and returns to step ST2.

On the other hand, if step ST5 is affirmed, the display control unit 27 displays the rigidly aligned three-dimensional image, that is, the deformed three-dimensional image V1 and the perspective image T0 in a superimposed manner (step ST9), and returns to step ST2. .

In this way, in this embodiment, the three-dimensional image V0 and the fluoroscopic image T0 are aligned by rigid body alignment, which requires a relatively small amount of calculation, and only when the respiratory phase error D0 is equal to or greater than the threshold Th1. , the three-dimensional image V0 and the perspective image T0 are aligned using non-rigid body alignment which requires a relatively large amount of calculation. Therefore, the three-dimensional image V0 and the perspective image T0 can be aligned with high accuracy compared to the case where only rigid body alignment is performed. Furthermore, compared to the case where only non-rigid alignment is performed, the amount of calculation can be reduced and alignment between the three-dimensional image V0 and the perspective image T0 can be performed at high speed. Therefore, the three-dimensional image V0 and the two-dimensional perspective image T0 can be aligned at high speed and with high accuracy.

Next, a second embodiment of the present disclosure will be described. Note that the functional configuration of the image processing apparatus according to the second embodiment is the same as the functional configuration of the image processing apparatus according to the first embodiment shown in FIG. 3, so a detailed explanation will be omitted here. In the first embodiment, when the determination by the determination unit 24 is negative, the lung field region is extracted from the three-dimensional image V0 and the fluoroscopic image T0, and non-rigid positioning is performed. The second embodiment differs from the first embodiment in that the lung field region is extracted from the three-dimensional image V0 and the fluoroscopic image T0 before the determination by the determination unit 24.

Next, the processing performed in the second embodiment will be explained. FIG. 8 is a flowchart showing the processing performed in the second embodiment. First, the image acquisition unit 21 acquires the three-dimensional image V0 from the image storage server 4 (step ST11), and then the image acquisition unit 21 acquires the perspective image T0 (step ST12). Subsequently, the extraction unit 25 extracts a lung field region from the three-dimensional image V0 and the fluoroscopic image T0 (step ST13). Then, the first alignment unit 22 rigidly aligns the three-dimensional image V0 and the perspective image T0 (step ST14). Subsequently, the error deriving unit 23 derives the error D0 of the respiratory phase between the rigidly aligned three-dimensional image, that is, the deformed three-dimensional image V1, and the fluoroscopic image T0 (step ST15), and the determining unit 24 derives the error D0 It is determined whether or not is less than a predetermined threshold Th1 (step ST16). Note that the process in step ST13 may be performed after the process in step ST14 and before the process in step ST16. Further, the processing may be performed in parallel with the processing in steps ST14 and ST15.

If step ST16 is negative, the second alignment unit 26 performs non-rigid alignment between the three-dimensional image V0 and the fluoroscopic image T0 based on the lung field region (step ST17). Then, the display control unit 27 displays the non-rigidly aligned three-dimensional image, that is, the deformed three-dimensional image V2 and the perspective image T0 in a superimposed manner (step ST18), and returns to step ST12.

On the other hand, if step ST16 is affirmed, the display control unit 27 displays the rigidly aligned three-dimensional image, that is, the deformed three-dimensional image V1 and the perspective image T0 in a superimposed manner (step ST19), and returns to step ST12. .

As described above, in the second embodiment, since the lung field region is extracted from the three-dimensional image V0 and the fluoroscopic image T0 before the determination by the determination unit 24, if the determination by the determination unit 24 is negative, Non-rigid alignment can be performed immediately.

In each of the above embodiments, the error deriving unit 23 derives the difference in the position of the diaphragm between the modified three-dimensional image V1 and the fluoroscopic image T0 as the error D0, but the present invention is not limited to this. Here, the area of the lung field region is different between the exhalation phase and the inhalation phase. For this reason, the error deriving unit 23 may extract lung field regions from each of the modified three-dimensional image V1 and the fluoroscopic image T0, and derive the difference in area of the extracted lung field regions as the error D0.

Further, in each of the embodiments described above, a process is described in which a lung lesion is collected using a bronchoscope inserted into a bronchus, but the present invention is not limited to this. For example, the image processing apparatus according to this embodiment can be applied to the case where an ultrasonic endoscope is inserted into a digestive organ such as the stomach to perform a biopsy of a tissue such as the pancreas or liver.

Furthermore, in each of the above embodiments, various processes such as the image acquisition section 21, the first alignment section 22, the error derivation section 23, the determination section 24, the extraction section 25, the second alignment section 26, and the display control section 27 are performed. As the hardware structure of the processing unit that executes, the following various processors can be used. As mentioned above, the various processors mentioned above include the CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as circuits such as FPGA (Field Programmable Gate Array) after manufacturing. A programmable logic device (PLD), which is a processor whose configuration can be changed, and a dedicated electrical device, which is a processor with a circuit configuration specifically designed to execute a specific process, such as an ASIC (Application Specific Integrated Circuit) Includes circuits, etc.

One processing unit may be composed of one of these various types of processors, or a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). ). Further, the plurality of processing units may be configured with one processor.

As an example of configuring a plurality of processing units with one processor, firstly, as typified by computers such as a client and a server, one processor is configured with a combination of one or more CPUs and software, There is a form in which this processor functions as a plurality of processing units. Second, there are processors that use a single IC (Integrated Circuit) chip to implement the functions of an entire system including multiple processing units, as typified by System On Chip (SoC). be. In this way, various processing units are configured using one or more of the various processors described above as a hardware structure.

Furthermore, as the hardware structure of these various processors, more specifically, an electric circuit (Circuitry) that is a combination of circuit elements such as semiconductor elements can be used.

1 Computer 2 Three-dimensional image capturing device 3 Fluoroscopic image capturing device 3A Arm 3B X-ray source 3C X-ray detector 4 Image storage server 5 Network 6 Ultrasonic endoscope device 6A Endoscope 10 Image processing device 11 CPU
12 Image processing program 13 Storage 14 Display 15 Input device 16 Memory 21 Image acquisition unit 22 First alignment unit 23 Error derivation unit 24 Judgment unit 25 Extraction unit 26 Second alignment unit 27 Display control unit 31, 32 Diaphragm 40 Display screen 41 Superimposed image T0 Perspective image V0 Three-dimensional image V1, V2 Deformed three-dimensional image VT0 Pseudo-transparent image VT1 Pseudo-deformed transparent image

Claims (11)

comprising at least one processor;
The processor includes:
Acquire three-dimensional images and radiographic images of the same subject,
rigid body alignment of the three-dimensional image and the radiation image;
Deriving a respiratory phase error between the rigidly aligned three-dimensional image and the radiographic image,
determining whether the error is less than a predetermined threshold;
If the determination is negative, the image processing device performs non-rigid alignment of the three-dimensional image and the radiation image. The image processing apparatus according to claim 1, wherein the three-dimensional image and the radiation image include a lung field region of the subject. The processor extracts the lung field region from each of the three-dimensional image and the radiation image,
The image processing device according to claim 2 , wherein non-rigid positioning of the three-dimensional image and the radiation image is performed based on the extracted lung field region. The image processing apparatus according to claim 3, wherein the processor extracts a lung field region from each of the three-dimensional image and the radiation image when the determination is negative. The image processing device according to claim 3, wherein the processor extracts a lung field region from each of the three-dimensional image and the radiation image before making the determination. The processor determines the respiratory phase based on a difference between a position of the diaphragm included in the three-dimensional image in the body axis direction of the subject and a position of the diaphragm included in the radiographic image in the body axis direction of the subject. The image processing apparatus according to any one of claims 2 to 5, which derives an error of. The processor uses the trained model to determine the position of the diaphragm included in the three-dimensional image in the body axis direction of the subject, and the position of the diaphragm included in the radiographic image in the body axis direction of the subject. The image processing device according to claim 6, wherein the image processing device is derived. The processor determines the respiratory phase based on a difference between an area of a lung field region included in a projection image obtained by projecting the three-dimensional image in the imaging direction of the radiographic image and an area of a lung field region included in the radiographic image. The image processing apparatus according to any one of claims 2 to 5, which derives an error of. When the determination is negative, the processor superimposes and displays the non-rigidly aligned three-dimensional image and the radiation image, and when the determination is affirmative, the processor displays the rigidly aligned three-dimensional image and the radiation image. The image processing device according to claim 1, wherein the image processing device displays a radiation image in a superimposed manner. Acquire three-dimensional images and radiographic images of the same subject,
rigid body alignment of the three-dimensional image and the radiation image;
Deriving a respiratory phase error between the rigidly aligned three-dimensional image and the radiographic image,
determining whether the error is less than a predetermined threshold;
If the determination is negative, the image processing method performs non-rigid alignment of the three-dimensional image and the radiation image. A procedure for obtaining a three-dimensional image and a radiographic image of the same subject;
a step of rigidly aligning the three-dimensional image and the radiation image;
a step of deriving a respiratory phase error between the rigidly aligned three-dimensional image and the radiographic image;
a step of determining whether the error is less than a predetermined threshold;
If the determination is negative, the image processing program causes a computer to execute a procedure for non-rigid positioning of the three-dimensional image and the radiation image.