JP6510301B2 - MEDICAL SUPPORT SYSTEM, MEDICAL SUPPORT METHOD, IMAGE PROCESSING APPARATUS, AND CONTROL METHOD AND CONTROL PROGRAM THEREOF - Google Patents

Description

  The present invention relates to a technology for supporting medical treatment by displaying a three-dimensional image in a patient's body.

  In the above technical field, Patent Document 1 discloses a technique for displaying a tomographic image of CT or MRI whose cross section coincides with an ultrasonic image. Further, Patent Document 2 discloses a technique for displaying the display position of an ultrasonic image on a tomographic image of CT or MRI. Further, Patent Document 3 discloses a technique for displaying a three-dimensional image of CT or MRI whose cross section substantially coincides with that of a three-dimensional ultrasonic image.

JP 2008-188193 A JP, 2012-223416, A JP, 2013-063342, A

  However, although the technique described in the above document aligns the ultrasonic image and the CT or MRI image and displays the same cross-sectional image side by side so as to be referable, it recognizes the state of the part in real time inside the patient's living body It was difficult to do.

  An object of the present invention is to provide a technique for solving the above-mentioned problems.

In order to achieve the above object, the medical support system according to the present invention is
A part model generation unit that generates data of a three-dimensional part model based on a tomographic image group of a patient;
First feature point extraction means for extracting position information of a plurality of first feature points having predetermined features from the three-dimensional part model;
The ultrasound data acquired for the patient using the ultrasound probe and the position and orientation data of the ultrasound probe for which the ultrasound data is acquired are stored in association with each other, and the ultrasound data, the position, and the position Ultrasonic model generation means for generating data of a three-dimensional ultrasonic model using orientation data;
Second feature point extraction means for extracting position information of the plurality of second feature points having the predetermined feature from the three-dimensional ultrasound model;
Deformation means for deforming the three-dimensional part model so that the plurality of first feature points and the plurality of second feature points overlap;
Display means for displaying a projection image of the three-dimensional part model deformed by the deformation means;
Equipped with
The deformation means is
Division area generation means for dividing the three-dimensional part model into a plurality of three-dimensional areas each including at least one of the first feature points;
Each of the plurality of three-dimensional regions is position-adjusted such that the plurality of first feature points and the plurality of second feature points overlap, and a division region connection is made to connect the plurality of position-adjusted plurality of three-dimensional regions. Means,
Have
The display means, that displays a projection image of the divided plurality of three-dimensional area region connecting means is connected as a projection image of the three-dimensional site model.

In order to achieve the above object, the medical support method according to the present invention is
A part model generation step of generating data of a three-dimensional part model based on a tomographic image group of a patient;
A first feature point extraction step of extracting position information of a plurality of first feature points having predetermined features from the three-dimensional part model;
An accumulation step of correlating the ultrasound data acquired for the patient using the ultrasound probe with the position and orientation data of the ultrasound probe for which the ultrasound data is acquired, and accumulating the data in an accumulation unit;
An ultrasound model generation step of generating data of a three-dimensional ultrasound model using the ultrasound data and the position and orientation data;
A second feature point extraction step of extracting position information of the plurality of second feature points having the predetermined feature from the three-dimensional ultrasound model;
Deforming the three-dimensional part model so that the plurality of first feature points and the plurality of second feature points overlap;
A display step of displaying a projection image of the three-dimensional part model deformed in the deformation step;
Only including,
The deformation step is
A division area generation step of dividing the three-dimensional part model into a plurality of three-dimensional areas each including at least one of the first feature points;
Each of the plurality of three-dimensional regions is position-adjusted such that the plurality of first feature points and the plurality of second feature points overlap, and a division region connection is made to connect the plurality of position-adjusted plurality of three-dimensional regions. Step and
Including
In the display step, projection images of the plurality of three-dimensional regions connected in the division region connection step are displayed as projection images of the three-dimensional part model .

In order to achieve the above object, an image processing apparatus according to the present invention is
Receiving means for receiving data of a three-dimensional part model generated based on a group of tomographic images of the patient;
First feature point extraction means for extracting position information of a plurality of first feature points having predetermined features from the three-dimensional part model;
Storage means for correlating and storing ultrasound data acquired for the patient using an ultrasound probe and position and orientation data of the ultrasound probe for which the ultrasound data is acquired;
Ultrasonic model generation means for generating data of a three-dimensional ultrasonic model using the ultrasonic data and the position and orientation data;
Second feature point extraction means for extracting position information of the plurality of second feature points having the predetermined feature from the three-dimensional ultrasound model;
Deformation means for deforming the three-dimensional part model so that the plurality of first feature points and the plurality of second feature points overlap;
Transmission means for transmitting the projection image of the three-dimensional part model deformed by the deformation means to display means;
Equipped with
The deformation means is
Division area generation means for dividing the three-dimensional part model into a plurality of three-dimensional areas each including at least one of the first feature points;
Each of the plurality of three-dimensional regions is position-adjusted such that the plurality of first feature points and the plurality of second feature points overlap, and a division region connection is made to connect the plurality of position-adjusted plurality of three-dimensional regions. Means,
Have
It said transmission means that sends the projected image of the division of the plurality of regions connecting means are linked three-dimensional region on the display means as a projection image of the three-dimensional site model.

In order to achieve the above object, a control method of an image processing apparatus according to the present invention is:
Receiving data of a three-dimensional part model generated based on a group of tomographic images of the patient;
A first feature point extraction step of extracting position information of a plurality of first feature points having predetermined features from the three-dimensional part model;
An accumulation step of correlating the ultrasound data acquired for the patient using the ultrasound probe with the position and orientation data of the ultrasound probe for which the ultrasound data is acquired, and accumulating the data in an accumulation unit;
An ultrasound model generation step of generating data of a three-dimensional ultrasound model using the ultrasound data and the position and orientation data;
A second feature point extraction step of extracting position information of the plurality of second feature points having the predetermined feature from the three-dimensional ultrasound model;
Deforming the three-dimensional part model so that the plurality of first feature points and the plurality of second feature points overlap;
A transmitting step of transmitting a projection image of the three-dimensional part model deformed in the deformation step to display means;
Only including,
The deformation step is
A division area generation step of dividing the three-dimensional part model into a plurality of three-dimensional areas each including at least one of the first feature points;
Each of the plurality of three-dimensional regions is position-adjusted such that the plurality of first feature points and the plurality of second feature points overlap, and a division region connection is made to connect the plurality of position-adjusted plurality of three-dimensional regions. Step and
Including
In the transmission step, projection images of the plurality of three-dimensional regions connected in the division region connection step are transmitted to the display means as a projection image of the three-dimensional part model .

In order to achieve the above object, a control program of an image processing apparatus according to the present invention is:
Receiving data of a three-dimensional part model generated based on a group of tomographic images of the patient;
A first feature point extraction step of extracting position information of a plurality of first feature points having predetermined features from the three-dimensional part model;
An accumulation step of correlating the ultrasound data acquired for the patient using the ultrasound probe with the position and orientation data of the ultrasound probe for which the ultrasound data is acquired, and accumulating the data in an accumulation unit;
An ultrasound model generation step of generating data of a three-dimensional ultrasound model using the ultrasound data and the position and orientation data;
A second feature point extraction step of extracting position information of the plurality of second feature points having the predetermined feature from the three-dimensional ultrasound model;
Deforming the three-dimensional part model so that the plurality of first feature points and the plurality of second feature points overlap;
A transmitting step of transmitting a projection image of the three-dimensional part model deformed in the deformation step to display means;
A control program of an image processing apparatus which causes a computer to execute
In the deformation step,
A division area generation step of dividing the three-dimensional part model into a plurality of three-dimensional areas each including at least one of the first feature points;
Each of the plurality of three-dimensional regions is position-adjusted such that the plurality of first feature points and the plurality of second feature points overlap, and a division region connection is made to connect the plurality of position-adjusted plurality of three-dimensional regions. Step and
On your computer,
In the transmitting step, the computer is made to transmit the projection images of the plurality of three-dimensional regions connected in the division region connecting step to the display means as a projection image of the three-dimensional part model .

  According to the present invention, it is possible to easily recognize the state of a site in real time inside a patient's living body.

It is a block diagram showing composition of a medical support system concerning a 1st embodiment of the present invention. It is a figure showing processing outline of a medical support system concerning a 2nd embodiment of the present invention. It is a figure which shows the outline | summary of the extraction process of the 1st feature point group which concerns on 2nd Embodiment of this invention, and a 2nd feature point group. It is a block diagram showing composition of a medical support system concerning a 2nd embodiment of the present invention. It is a sequence diagram which shows the operation | movement procedure of the medical support system which concerns on 2nd Embodiment of this invention. It is a block diagram showing functional composition of an image processing device concerning a 2nd embodiment of the present invention. It is a figure which shows the structure of the division area table which concerns on 2nd Embodiment of this invention. It is a figure which shows the structure of the ultrasound image table which concerns on 2nd Embodiment of this invention. It is a figure which shows the structure of the 2nd feature point table which concerns on 2nd Embodiment of this invention. It is a block diagram showing functional composition of a display picture generation part concerning a 2nd embodiment of the present invention. It is a figure explaining the process of the division | segmentation area | region connection part which concerns on 2nd Embodiment of this invention. It is a figure which shows the structure of the division area connection table which concerns on 2nd Embodiment of this invention, and the example of the information for alignment. It is a figure which shows the structure of the display data production | generation table which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the hardware constitutions of the image processing apparatus which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the process sequence of the image processing apparatus which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the procedure of the feature point extraction process which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the procedure of the area | region division processing which concerns on 2nd Embodiment of this invention. It is a block diagram showing processing outline of a medical support system concerning a 3rd embodiment of the present invention. It is a block diagram showing the composition of the medical support system concerning a 3rd embodiment of the present invention. It is a block diagram showing functional composition of an image processing device concerning a 3rd embodiment of the present invention. It is a block diagram showing composition of a medical support system concerning a 4th embodiment of the present invention. It is a block diagram showing functional composition of an image processing device concerning a 4th embodiment of the present invention. It is a block diagram showing the composition of the medical support system concerning a 5th embodiment of the present invention. It is a figure showing an example of various composition of a medical support system of the present invention.

  Hereinafter, embodiments of the present invention will be exemplarily described in detail with reference to the drawings. However, the components described in the following embodiments are merely examples, and the scope of the present invention is not limited to them. As used herein, the term "site" refers to the whole or a part of the inside of a living body, for example, from a wide range such as the lower abdomen, visceral organs such as lung, liver, stomach, etc. It is a term that expresses a narrow range such as lymphatic line and a part of nerve.

First Embodiment
A medical support system 100 according to a first embodiment of the present invention will be described with reference to FIG. The medical support system 100 is a system that supports medical care by displaying a three-dimensional site model in the patient's body.

  As shown in FIG. 1, the medical support system 100 includes a part model generation unit 110, a first feature point extraction unit 120, an ultrasound model generation unit 130, a second feature point extraction unit 140, and a deformation unit 150. , And the display unit 160. The part model generation unit 110 generates data of the three-dimensional part model 111 based on the tomographic image group of the patient 101. The first feature point extraction unit 120 extracts position information of a plurality of first feature points having predetermined features from the three-dimensional part model 111. The ultrasound model generation unit 130 associates and stores the ultrasound data 104 acquired for the patient 101 using the ultrasound probe 103 and the position and orientation data of the ultrasound probe 103 for which the ultrasound data 104 is acquired. Then, using the ultrasound data 104 and the position and orientation data, data of the three-dimensional ultrasound model 131 is generated. The second feature point extraction unit 140 extracts position information of a plurality of second feature points having predetermined features from the three-dimensional ultrasound model 131. The deformation unit 150 deforms the three-dimensional part model 111 so that the plurality of first feature points and the plurality of second feature points overlap. The display unit 160 displays the projection image 160 of the three-dimensional part model deformed by the deformation unit 150.

  Note that the deformation unit 150 divides the three-dimensional region model 111 into a plurality of three-dimensional regions each including at least one first feature point, a plurality of first feature points, and a plurality of first feature points. The display unit 160 has a divided area connection unit 152 which adjusts the positions of the plurality of three-dimensional areas so that the two feature points overlap and connects the plurality of position-adjusted three-dimensional areas 153 with one another. A plurality of three-dimensional areas 161 connected by the divided area connection unit 152 are displayed as a three-dimensional part model.

  According to the present embodiment, the three-dimensional region model based on the tomographic image group is converted and displayed corresponding to the three-dimensional ultrasonic image acquired by the ultrasonic probe in real time inside the patient's living body. The state of the site can be easily recognized.

Second Embodiment
Next, a medical support system according to a second embodiment of the present invention will be described. The medical support system according to the present embodiment extracts the first feature point of the three-dimensional blood vessel model included in the target region of the three-dimensional region model data generated from the CT image or the MRI image, Divide into regions containing feature points. In addition, second feature points of the three-dimensional blood vessel model included in the target area of the ultrasound image data acquired by the ultrasound probe are extracted. Then, the three-dimensional part model data is converted to deform the three-dimensional part model by adjusting and connecting the divided areas so that the three-dimensional position of the first feature point matches the three-dimensional position of the second feature point. And display the deformed three-dimensional site model so as to overlap the patient's body. Detailed deformation of the three-dimensional part model may be further performed by non-rigid deformation or the like.

  Also, in order to display so as to overlap with the living body of the patient, an optical marker fixed to the living body of the patient is imaged, and three-dimensional site model data of position, shape and size based on the position and orientation of the optical marker is generated Do. The optical marker is provided as a magnetic sensor with an optical marker. The precise position and orientation of the ultrasonic probe are detected from the magnetic sensor fixed to the ultrasonic probe and the magnetic sensor fixed to the living body. The three-dimensional site model is displayed on an optical see-through head mounted display (hereinafter, HMD: Head Mounted Display). In the present embodiment, an example in which the optical marker or the magnetic sensor is fixed to the patient's living body is shown. However, when the laparotomy is performed during the operation, the optical marker or the magnetic sensor is directly fixed to the surgery target organ It is also possible to calculate a more accurate position.

<< medical support system >>
The configuration and operation of the medical support system of the present embodiment will be described with reference to FIGS. 2A, 2B, 2C, and 3.

(System overview)
FIG. 2A is a diagram showing an outline of a medical support system 200 according to the present embodiment.

  In the medical support system 200, for example, a three-dimensional part model or a three-dimensional part represented by three-dimensional model data of STL (Standard Triangulated Language) format generated from a CT image 210 which is one of computed tomography images. From a three-dimensional blood vessel model or the like extracted from the model, a first feature point group 240 arranged on three-dimensional coordinates representing the features of the model is extracted. The first feature point group 240 is not limited as long as it represents the characteristic position of the living tissue in the three-dimensional model data. For example, the number of branches, the branch vector, or the entire pattern at the branch point of the blood vessel A unique point may be taken as the first feature point. Next, division processing 260 is performed from the first feature point group 240 into a region including at least one, preferably at least three adjacent first feature points. It is preferable that the first feature points be located at the division plane or division point of the area because the division areas are easily connected.

  On the other hand, in the examination room or operating room 230, from the time-series two-dimensional ultrasonic image group 220 acquired in real time by the ultrasonic probe 203 whose position has been determined using a position detection instrument (for example, a magnetic sensor) , Generate a three-dimensional blood vessel model from the three-dimensional ultrasound image, and extract a second feature point group 250 arranged on three-dimensional coordinates representing the feature of the model. The second feature point group 250 is not limited as long as it represents the characteristic position of the living tissue in the three-dimensional ultrasonic image data, but the number of branches or the number of branches at the blood vessel branch point is the same as the first feature point. It is desirable that the branching vector or the entire pattern have a characteristic point as the second feature point. Here, the instrument for detecting the position of the ultrasonic probe 203 is preferably a magnetic sensor that is not blocked rather than an optical marker that is blocked by a doctor or the like during medical examination or surgery.

  Then, in the medical support system 200, the three-dimensional matching between the first feature point and the second feature point is performed, and the division area is adjusted so that the first feature point group 240 overlaps the second feature point group 250. The transformation of the three-dimensional model data is carried out by being connected. The deformation of the three-dimensional model data may further include non-rigid deformation of the divided region and the three-dimensional part model after connection.

  Next, in the medical support system 200, based on the positional relationship between the living body and the three-dimensional part model based on the three-dimensional image data, which is measured using the position detection instrument fixed on the living body 201 in real time, The three-dimensional part model 270 deformed corresponding to the ultrasonic image of time is adjusted to be inside the living body and displayed three-dimensionally. Preferably, the display unit for three-dimensional display of the three-dimensional part model 270 is an optical see-through HMD, and the doctor visually observes the actual living body ahead of the three-dimensional part model on the display screen of the HMD. Here, as a tool for position detection on a living body, a magnetic sensor with an optical marker for obtaining the positional relationship between the living body and the display of the three-dimensional part model with high accuracy is desirable.

  FIG. 2B is a diagram showing an outline of extraction processing of the first feature point group 240 and the second feature point group 250 according to the present embodiment. Although extraction of the first feature point group 240 is described as a representative in FIG. 2B, extraction of the second feature point group 250 is the same.

  First, a center line (core line) 242 of each blood vessel image is extracted from the acquired three-dimensional blood vessel image 241. Next, feature points 243 such as branch points and inflection points of the plurality of core lines 242 are extracted and used in the division processing 260.

(System configuration)
FIG. 2C is a block diagram showing the configuration of the medical support system 200 according to the present embodiment.

  The medical support system 200 includes a CT apparatus 211, a three-dimensional model generation apparatus 212, an image processing apparatus 280, and a medical examination / surgical instrument group arranged in a medical examination room or an operating room 230.

  The CT apparatus 211 acquires computer tomography images of the patient 201 in advance. The three-dimensional CT image generated for the acquired target organ is desirably converted to three-dimensional data such as STL format. Although there is no limitation on the range of the three-dimensional data, it is desirable to be the organ unit of interest of the patient 201. The three-dimensional model generation device 212 generates a three-dimensional region model including, for example, a three-dimensional blood vessel model from the group of computed tomography images from the CT device 211 and passes it to the image processing device 280. In the image processing apparatus 280 shown in FIG. 2C, the CT apparatus 211 obtains a three-dimensional site model based on a group of computed tomography images. However, even if it is an MRI apparatus, a three-dimensional site model based on another computed tomography apparatus It may be Further, the data of the three-dimensional region model is not limited to the STL format as long as it represents the configuration of a living tissue inside the living body of the patient 201 or the configuration of an organ or an affected part. In the present embodiment, although the case where the image processing device 280 acquires the data of the three-dimensional part model generated by the external three-dimensional model generation device 212 is shown, the image processing device 280 is a three-dimensional model generation device The configuration may include the functions of 212.

  On the other hand, as a medical examination / surgical instrument group of the medical examination room and the operating room 230, an ultrasonic probe 203, a magnetic sensor 231, a magnetic sensor 293 with an optical marker, a web camera 294 and an HMD 295 are provided. The web camera 294 and the HMD 295 are integrated as an optical see-through HMD. The ultrasound probe 203 acquires ultrasound images from the patient 201 during real time examination or surgery. The magnetic sensor 231 is disposed on the ultrasonic probe 203 to detect the position of the ultrasonic probe 203. The magnetic sensor with magnetic marker 293 is disposed on the patient 201 and detects the position of the optical marker with the magnetic sensor, and the optical marker is used to determine the positional relationship between the living body of the patient 201 and the Web camera 294. The web camera 294 captures an optical marker of the magnetic sensor 293 with an optical marker, and determines the positional relationship between the living body of the patient 201 and the web camera 294. A magnetic sensor may be attached to the HMD in order to measure the positional relationship between the living body of the patient 201 and the web camera 294. The HMD 295 displays a three-dimensional model represented by three-dimensional data deformed to match the ultrasound image at the position of the patient 201 viewed by the doctor 202.

  The image processing device 280 deforms the three-dimensional region model represented by the three-dimensional data so as to match the ultrasound image acquired by the ultrasound probe 203 whose position and orientation have been determined by the detection signal of the magnetic sensor 231. Such deformation includes division processing for dividing a three-dimensional region model into regions based on feature points such as branch points of blood vessels, and connection processing for connecting division regions so as to match feature points of an ultrasound image. . Next, the image processing device 280 detects the position and orientation of the patient 201, and the positions of the patient 201 and the HMD 295, based on the detection signal of the magnetic sensor of the magnetic sensor 293 with an optical marker and the image data of the Web camera 294 imaging the optical marker. Calculate the relationship. The positional relationship between the web camera 294 and the HMD 295 is fixed as an integral HMD. Then, the image processing device 280 generates three-dimensional display data so that the deformed three-dimensional region model is at an accurate position inside the living body of the patient 201, and causes the HMD 295 to perform three-dimensional display. The merit of fixing the sensor to the living body in this way is that the three-dimensional position can be accurately estimated even when the living body moves.

(Operation procedure)
FIG. 3 is a sequence diagram showing an operation procedure of the medical support system 200 according to the present embodiment.

  The CT apparatus 211 acquires a computer tomography image group of the patient 201 in step S301. In step S302, the three-dimensional model generation device 212 generates three-dimensional region model data for extracting a region such as a liver or a blood vessel thereof from the computed tomography image group and displaying it three-dimensionally, and transmits it to the image processing device 280 . The image processing device 280 receives the three-dimensional part model data from the three-dimensional model generation device 212 in step S303. In step S305, the image processing device 280 extracts a first feature point group based on a three-dimensional region model such as a three-dimensional blood vessel model. Then, in step S307, the image processing device 280 divides and holds the three-dimensional region model into a three-dimensional region including at least one first feature point.

  In step S309, the magnetic sensor 231 acquires the position and orientation of the ultrasonic probe 203 and transmits it to the image processing device 280, and the ultrasonic probe 203 acquires the ultrasonic signal at that time and transmits it to the image processing device 280. In step S311, the image processing device 280 generates two-dimensional data of ultrasonic waves based on a plurality of sequential position information and ultrasonic signals, and then generates, for example, three-dimensional ultrasonic model data indicating a blood vessel model. Do. Next, in step S313, the image processing device 280 extracts a second feature point group based on three-dimensional ultrasound model data indicating a blood vessel model.

  Furthermore, in step S315, the image processing device 280 performs position conversion on the divided area generated from the three-dimensional part model so that the first feature point group matches the second feature point group. Then, the image processing device 280 connects the divided areas obtained by the position conversion. In order to accurately connect the divided areas, it is desirable to include the first feature point on the surface, ridge line or vertex of the divided area.

  The magnetic sensor of the magnetic sensor 293 with the optical marker acquires the position and orientation of the living body of the patient 201 placed in step S317, and transmits the acquired position and orientation to the image processing device 280. In step S319, the web camera 294 captures an optical marker of the magnetic sensor 293 with an optical marker, and transmits imaging data of the optical marker to the image processing apparatus 280. In step S 321, the image processing device 280 calculates the positional relationship (distance and direction) between the living body and the HMD 295 as the display unit from the imaging data of the optical marker.

  The position of the ultrasonic probe 203 can be detected more accurately by the magnetic sensor 293 with an optical marker and the magnetic sensor 231 of the ultrasonic probe 203. That is, although the accurate position can be detected by arranging the optical marker on the ultrasonic probe 203, it is a condition that the optical marker does not hide from the field of view of the camera. In the present embodiment, the magnetic sensor 231 is disposed on the ultrasonic probe 203 so that the ultrasonic probe 203 can be freely moved. However, since the accuracy is reduced as compared with the optical marker, by disposing the magnetic sensor 293 with the optical marker on the living body 201, the accuracy is secured. Therefore, when the magnetic sensor is attached to the HMD 295, the positional relationship between the HMD 295 and the ultrasonic probe 203 can be detected, and therefore, imaging by a camera is not necessary.

  Then, in step S 323, the image processing device 280 generates display data to display the three-dimensional region model deformed by the connection of divided regions as if it is inside the living body of the patient 201, and transmits it to the HMD 295. In step S325, the HMD 295 displays the three-dimensional site model as if it is inside the living body of the patient 201 based on the received display data.

<< Functional Configuration of Image Processing Device >>
FIG. 4 is a block diagram showing a functional configuration of the image processing apparatus 280 according to the present embodiment.

  The image processing device 280 includes a three-dimensional part model reception unit 411, a first feature point extraction unit 412, and a divided region generation unit 424. The image processing apparatus 280 further includes a communication control unit 413, a magnetic signal reception unit 414, an ultrasonic probe position detection unit 415, an ultrasonic signal reception unit 416, an ultrasonic model data generation unit 417, and a second feature. And a point extraction unit 418. Furthermore, the image processing device 280 includes a magnetic signal reception unit 419, a patient position detection unit 420, an optical marker image reception unit 421, an HMD position calculation unit 422, a display image generation unit 423, and a display image transmission unit 425. And. The display image generation unit 423 includes a divided area matching unit 431, a divided area connection unit 432, and an HMD image generation unit 433.

  The three-dimensional region model reception unit 411 receives a three-dimensional region model such as a blood vessel model generated from the computer tomography image group in the external three-dimensional model generation device 212. The reception of the three-dimensional part model may be via a storage medium or via the communication control unit 413. When the image processing device 280 executes generation of three-dimensional data such as STL data from a tomographic image, the image processing device 280 includes an STL data generation unit. The first feature point extraction unit 412 extracts a first feature point from a three-dimensional part model such as a blood vessel model. The divided area generation unit 424 divides the three-dimensional part model represented by the STL data or the like into three-dimensional small areas including the adjacent first feature points. Each divided area includes at least one first feature point. Preferably, at least three first feature points are included in each split region for position conversion by matching with the second feature points of the ultrasound image of the split region. Furthermore, in order to facilitate connection of divided regions, it is desirable to divide the first feature points so that they are on the faces, edges, and vertices of each divided region.

  The communication control unit 413 exchanges information with the instruments included in the medical examination / surgical instrument group 290. Communication with the medical examination / operational instrument group in the examination room or the operation room 230 may be wired or wireless, but it is desirable to be wireless communication. The magnetic signal reception unit 414 receives the magnetic detection signal from the magnetic sensor 231 disposed in the ultrasonic probe 203 via the communication control unit 413. The ultrasonic probe position detection unit 415 detects the position and orientation of the ultrasonic probe 203 based on the magnetic detection signal from the magnetic sensor 231. In detail, detection of the position and orientation by the magnetic sensor 231 based on the magnetic detection signal is performed by a controller in a position detection system including a magnetic field generator (not shown), a controller unit (not shown) and the magnetic sensor 231. The direction and coordinates are calculated by the unit and transmitted. When the magnetic sensor 231 has a storage unit for detecting the position and the direction from the magnetic detection signal, data (for example, coordinates and a vector) indicating the position and the direction may be received from the magnetic sensor 231. The ultrasound signal reception unit 416 receives an ultrasound signal from the ultrasound probe 203 via the communication control unit 413. When the ultrasound probe 203 has a function of generating ultrasound image data from an ultrasound signal, the ultrasound signal reception unit 416 may receive ultrasound image data.

  The ultrasound model data generation unit 417 has three-dimensional coordinates of the ultrasound signal received by the ultrasound signal reception unit 416, corresponding to the position and orientation of the ultrasound probe 203 detected by the ultrasound probe position detection unit 415. Generate ultrasound model data. The ultrasound model data generation unit 417 connects the cross-sectional images acquired by the ultrasound probe 203 three-dimensionally to generate three-dimensional ultrasound model data. In the present embodiment, a structure such as a blood vessel is extracted from each cross-sectional image acquired by the ultrasonic probe 203, and they are connected three-dimensionally to generate three-dimensional ultrasonic model data. The second feature point extraction unit 418 extracts, for example, a second feature point such as branching of a blood vessel from the three-dimensional ultrasonic model data such as the blood vessel model generated by the ultrasonic model data generation unit 417.

  The magnetic signal reception unit 419 receives the magnetic detection signal from the optical sensor with magnetic marker 293 via the communication control unit 413. When the optical sensor with magnetic marker 293 has a storage unit for detecting the position and orientation from the magnetic detection signal, data (for example, coordinates and vectors) indicating the position and orientation may be received. The patient position detection unit 420 detects the position and orientation of the patient 201 based on the magnetic detection signal from the optical sensor equipped magnetic sensor 293. In detail, detection of the position and orientation by the magnetic sensor with magnetic marker 293 based on the magnetic detection signal is a position detection system including a magnetic field generator (not shown), a controller unit (not shown), and a magnetic sensor 293. The controller unit is configured to calculate and transmit the direction and coordinates.

  The optical marker image reception unit 421 receives a photographed image of the optical marker of the optical sensor with a magnetic marker 293 captured by the Web camera 294 installed in the HMD 295 via the communication control unit 413. The HMD position calculation unit 422 calculates the positional relationship between the HMD 295 and the patient 201 based on the captured image of the optical marker received by the optical marker image reception unit 421. As described above, if the magnetic sensor for position determination is arranged in the HMD 295, the calculation of the positional relationship between the HMD 295 and the patient 201 based on the captured image of the optical marker may not be necessary.

  The divided region matching unit 431 of the display image generation unit 423 matches the first feature point included in each divided region generated by the divided region generation unit 424 with the second feature point extracted by the second feature point extraction unit 418. Then, each divided area is moved and rotated so that the feature points of each other match. Then, the divided area connection unit 432 connects the divided areas by the first feature points, and generates a three-dimensional part model represented by STL data or the like, which is deformed so as to match the three-dimensional ultrasonic image. Do. Next, the HDM image generation unit 433 performs three-dimensional display at the HMD position calculated by the HMD position calculation unit 422 so that the three-dimensional part model deformed at the position of the patient 201 detected by the patient position detection unit 420 overlaps. Generate image data of That is, when observed from the measured position of the HMD 295, a three-dimensional image is generated which calculates how the three-dimensional part model overlapping the living body of the patient 201 looks like. The display image transmission unit 425 transmits the image data overlapping the position of the patient 201 generated by the HMD image generation unit 433 to the HMD 295 via the communication control unit 413 for display.

(Divided area table)
FIG. 5 is a diagram showing the configuration of the divided area table 500 according to the present embodiment. The divided region table 500 holds a first feature point extracted from a part model data such as a blood vessel model in a three-dimensional format from a CT image which is one of computed tomography images, and a divided region including the first feature point. It is a table. The configuration of the divided area table 500 is not limited to that shown in FIG. Further, FIG. 5 shows an example in which the branch location of the blood vessel is used as the first feature point, but it is not limited to this and it may be a location representing a unique feature.

  The divided area table 500 is generated from CT data of a plurality of slices, the received three-dimensional part model data 501 such as STL format, the three-dimensional blood vessel model data 502 included in the three-dimensional part model data 501, and the core line of the blood vessel. The extracted blood vessel core 503 is stored. In this example, feature points are extracted based on the blood vessel core line 503 in order to consider abstraction and universal features, but if the features possessed by the feature points are invariant to a matchable degree, the thickness of the blood vessel You may consider the ratio of The received three-dimensional region model data 501 may be an STL file from which only blood vessels are extracted.

  The divided area table 500 uses the pattern of the blood vessel core 503 as the first feature point data 504 of the present example, using the branch points as feature points, the branch ID, the branch feature data, and the three-dimensional position data of the feature points. Remember. As the branching feature data, at least one of the number of branches 511, a branching vector 512 indicating the start point and direction of each branch, a branching pattern 513 in which the entire branches are patterned, and others, for example, the blood flow direction 514 is stored. The branched feature data may be classified in advance as a feature ID 515. The first feature point data 504 is not limited to FIG. It is possible to add data that increases the reliability in matching and remove data that decreases the reliability.

  The divided area table 500 stores divided area IDs 505 for identifying divided areas including the first feature points, and divided area data 506. Here, the divided area data 506 includes blood vessel data other than the first feature point, shape data of the divided area, centroid data, and the like. Although FIG. 5 illustrates the case where the divided area includes the three first feature points or the four first feature points, the number of the first feature points included in the divided area is one example. There is no limitation. Further, by generating adjacent divided areas so as to have the first feature point common to the face, ridge line, and vertex, connection processing becomes easy.

(Ultrasound image table)
FIG. 6A is a diagram showing the configuration of an ultrasound image table 610 according to the present embodiment. The ultrasound image table 610 is used to generate three-dimensional ultrasound model data in consideration of the position and orientation of the ultrasound probe 291 from the ultrasound signal received from the ultrasound probe 291.

  The ultrasound image table 610 corresponds to ultrasound image data ID 611, patient information 612 including a patient ID and an affected part, acquisition date (time stamp) 613, and acquisition parameters 614 corresponding to the format of the ultrasound probe 203 and And store the received ultrasound image data 615 processed by the acquisition parameter 614.

  In addition, the ultrasonic image table 610 is a three-dimensional position of the ultrasonic probe indicating the position and the direction of the ultrasonic probe 203 detected from the magnetic sensor relative signal 616 from the magnetic sensor 231 and the magnetic sensor 293 and the magnetic sensor relative signal 616. Data 617 is stored. Here, the position and direction of the ultrasonic probe 203 are detected by the magnetic sensor relative signal 616 from the magnetic sensor 231 and the magnetic sensor 293 only when the position information of the ultrasonic probe 203 by the magnetic sensor 231 moves the living body. Sometimes the three-dimensional position shifts. Therefore, the position and orientation of the ultrasonic probe 203 are recorded as the relative position and orientation from the magnetic sensor 293 attached to the living body.

  Then, the ultrasound image table 610 stores three-dimensional ultrasound image data 618 in which the ultrasound image data 615 received corresponding to the position and orientation of the ultrasound probe 203 is three-dimensionally coordinate-transformed. Further, three-dimensional ultrasound model data 619 of an organ unit or an affected area unit of a patient connected to the three-dimensional ultrasound image data 618 corresponding to each scan is generated and stored.

(2nd feature point table)
FIG. 6B is a diagram showing the configuration of the second feature point table 620 according to the present embodiment. The second feature point table 620 is a table that holds feature points extracted from the ultrasound image data 618. The configuration of the second feature point table 620 is not limited to FIG. 6B. Further, FIG. 6B shows an example in which a branch point of a blood vessel is used as a feature point, but the present invention is not limited to this and it may be a place representing a unique feature, but the feature is common to the first feature point. desirable.

  The second feature point table 620 stores blood vessel site extraction image data 621 in association with ultrasound image data ID 611 and ultrasound image data 618 of the ultrasound image table 610. The blood vessel site extraction is performed by binarization and contour extraction of the ultrasonic image data 618, selection of an annular contour, and the like.

  Further, the second feature point table 620 stores blood vessel model data 622 extracted by connecting the blood vessel site extraction image data 621 and a blood vessel core line 623 from which the blood vessel core line is extracted. In this example, the feature points are extracted by the blood vessel core line 623 in order to consider abstract and universal features, but if the features possessed by the feature points are invariant to the extent that matching is possible, The ratio may be considered.

  Then, the second feature point table 620 uses the pattern of the blood vessel core 623 as the second feature point data 624 of this example, with the branch point as the feature point, the branch ID, the branch feature data, and the three-dimensional position of the feature point. Store the data. The branching feature data is the same as that of FIG. 5, so the detailed description will be omitted.

<< Display image generation part >>
FIG. 7A is a block diagram showing a functional configuration of the display image generation unit 423 according to the present embodiment.

  The display image generation unit 423 includes a three-dimensional part model acquisition unit 701, a divided area acquisition unit 702, a three-dimensional part model arrangement unit 703, a divided area arrangement unit 704, a second feature point data acquisition unit 705, and division. And a region matching unit 431. Here, the three-dimensional part model arrangement unit 703 and the divided area arrangement unit 704 correspond to the divided area connection unit 432. Further, the display image generation unit 423 includes a patient position data acquisition unit 707, a patient position model generation unit 708, an HMD position data acquisition unit 709, and an HMD image generation unit 433.

  The three-dimensional region model acquisition unit 701 acquires a three-dimensional region model including the three-dimensional blood vessel model received by the three-dimensional region model reception unit 411. The divided area acquisition unit 702 acquires the divided area generated by the divided area generation unit 424. The three-dimensional part model arrangement unit 703 aligns and arranges the three-dimensional part model data acquired by the three-dimensional part model acquisition unit 701 in association with the arrangement of the divided regions. The divided area arrangement unit 704 aligns and arranges the divided areas so that the divided areas match the three-dimensional ultrasound image by matching the first feature points and the second feature points included in the divided areas. The second feature point data acquisition unit 705 acquires second feature point data extracted from the second feature point extraction unit 418. The divided area matching unit 431 matches the first feature point and the second feature point included in the divided area, so that the first feature point data of the position-adjusted divided area matches the second feature point data, The three-dimensional part model placement unit 703 and the divided area placement unit 704 are deformed by position adjustment. Note that the deformation due to the position adjustment includes parallel movement, rotational movement or size change in which the shape of the divided area is maintained. Furthermore, a shape change may be included to match feature points after arrangement of divided regions.

  The patient position data acquisition unit 707 acquires the patient position data detected from the patient position detection unit 420. The patient position model generation unit 708 generates a three-dimensional model overlapping the patient position from the three-dimensional region model adjusted so as to be matched with the ultrasound image in the divided area connection unit 432. The HMD position data acquisition unit 709 acquires the HMD position data calculated from the HMD position calculation unit 422. The HMD image generation unit 710 generates, from the model generated by the model generation unit 708 for the patient position, an HMD image to be three-dimensionally displayed on the HMD 295 so as to be superimposed on the living body of the patient. It is output to the image transmission unit 425.

(Process of display image generation unit)
FIG. 7B is a diagram for explaining the process 720 of the divided area linking unit 432 according to the present embodiment. In addition, the process 720 of the division | segmentation area | region connection part 432 is not limited to FIG. 7B. Any process can be applied as long as the three-dimensional part model is deformed by adjusting the position of the divided area such that the first feature point included in the divided area matches the second feature point.

  The process 720 of the divided area linking unit 432 is based on the first set of feature points and the first set of divided areas from the set 721 of divided area data divided from the initial three-dimensional part model and the first feature point data included in the divided areas. Matching with two feature points includes conversion into three-dimensional part model data 722 in which divided areas are combined by resizing, three-dimensional position conversion, or direction conversion. Further, the processing 720 of the divided area linking unit 432 includes conversion of the three-dimensional part model data 722 obtained by combining the divided areas into size adjustment data 723 aligned with the position of the living body.

  The conversion from the set 721 of divided area data and first feature point data included in the divided area to three-dimensional part model data 722 in which divided areas are combined is performed by divided area matching 724 with second feature point data 726. It is realized by repeating the alignment and connection of the area. For example, by the least-square distance between the first feature point and the second feature point having the same feature, the position is adjusted to the divided area of the size or direction closest to the three-dimensional ultrasonic model, but other methods can be adopted It is.

  Next, the conversion from the three-dimensional part model data 722 obtained by combining the divided areas to the size adjustment data 723 matched to the position of the living body is performed so that the divided areas are superimposed on the living body position 727 acquired by the patient position data acquiring unit 707. And change the size of the three-dimensional site model data 722 combined (725).

  FIG. 7C is a diagram showing a configuration of the divided area linking table 730 according to this embodiment and an example 740 of alignment information.

  The divided region linking table 730 is used in the divided region linking unit 432 to link the divided regions to generate a three-dimensional region model that matches the three-dimensional ultrasound image.

  The divided area linking table 730 stores the divided area ID 505 of the divided area table 500 of FIG. 5, the divided area data 506, and the first feature point data 504. The divided area linking table 730 stores the current value 731 of the size, direction, and position of the changed divided area, and the changed position 732 of the first feature point data changed based on the current value 731. Then, the divided area connection table 730 stores the connected position 734 of the divided area based on the second feature point data 733 that matches the change position 732 of the first feature point data and the second feature point data 733 that matches. .

  Further, an example of the alignment information 740 is a position to be particularly targeted for alignment in the alignment target site (organ), and determines the arrangement position of the marker or sensor, and the scanning position by the ultrasonic probe. Used for

  The example of the alignment information 740 stores the alignment target position 742 and the condition 743 of the target position in association with the alignment target portion 741.

  Note that non-rigid deformation that compensates for the match may be combined during position adjustment of the divided area and the connection process, or after the connection process. In that case, even in the method of obtaining an optimal solution by non-rigid deformation using a conversion matrix based on material parameters, the difference in the distance between the first feature point and the second feature point having the same feature is corrected. May be used. As the material parameter, a general non-rigid parameter of internal organs may be used, but it is desirable to measure and use the non-rigid parameter for each patient or affected part.

  Furthermore, other methods may be adopted to match the feature points included in the non-rigid material. For example, a method may be used in which deformation of a non-rigid material is stored as learning information and used. When the three-dimensional model is an organ unit such as liver, stomach or pancreas, it is possible to learn many variations of these organs and express patient-specific deformation with less parameters.

  Alternatively, control points including feature points may be arranged in a fine grid in a three-dimensional region, and non-rigid deformation may be performed between the control points. Alternatively, region-based alignment can be performed by estimating spatially smooth irregular deformation (fluctuation) with high accuracy using non-rigid deformation (FFD: Free-form deformation) using B-Spline function. The method of performing super resolution used may be used.

  Also, as non-rigid deformation, affine transformation or transformation using a high-order polynomial can be applied. The match evaluation of the three-dimensional model includes residual sum of squared differences (SSD), normalized cross correlation (NCC), mutual information or normalized mutual information (MI: Mutual information) may be considered.

  In addition, it is also practiced to superimpose an image having anatomical information obtained by MRI or X-ray CT with an image having functional information such as PET or SPECT. It is applicable.

(Display data generation table)
FIG. 8 is a view showing the configuration of a display data generation table 800 according to the present embodiment. The display data generation table 800 is a table used to display on the HMD 295 a three-dimensional region model that matches ultrasound image data acquired in real time from the patient 201 so as to overlap the patient 201.

  The display data generation table 800 stores ultrasonic probe position information 802, patient position information 803, and HMD position information 804 in association with a time stamp 801 indicating an actual time. The ultrasonic probe position information 802 includes detection data of the magnetic sensor 292 and position data of the ultrasonic probe 291. The patient position information 803 includes detection data of the optical sensor with magnetic marker 293 and position data of the patient 201. The HMD position information 804 includes image data of the optical marker of the magnetic sensor with optical marker 293 and HMD position data.

  Then, the display data generation table 800 stores the HMD display information 805 generated with reference to the position information. The HMD display information 805 includes three-dimensional part model data generated with reference to the position information, HMD image data for the right eye, and HMD image data for the left eye.

<< Hardware Configuration of Image Processing Device >>
FIG. 9 is a block diagram showing the hardware configuration of the image processing apparatus 280 according to this embodiment.

  In FIG. 9, a CPU 910 is a processor for arithmetic control, and implements a functional configuration unit of the image processing apparatus 280 in FIG. 4 by executing a program. The ROM 920 stores fixed data and programs such as initial data and programs. In addition, the communication control unit 413 communicates with the examination / operational instrument group of the examination room and the operation room 230 and other devices via a network. Note that the CPU 910 is not limited to one, and may be a plurality of CPUs, or may include a GPU for image processing. Further, it is desirable that the communication control unit 413 have a CPU independent of the CPU 910, and write or read transmission / reception data in the area of the RAM 940. In addition, it is desirable to provide a DMAC for transferring data between the RAM 940 and the storage 950 (not shown). Furthermore, it is desirable that the input / output interface 960 have a CPU independent of the CPU 910 and write or read input / output data in the area of the RAM 940. Therefore, the CPU 910 recognizes that the data has been received or transferred to the RAM 940 and processes the data. Further, the CPU 910 prepares the processing result in the RAM 940, and leaves the subsequent transmission or transfer to the communication control unit 413, the DMAC, or the input / output interface 960.

  A RAM 940 is a random access memory used by the CPU 910 as a work area for temporary storage. In the RAM 940, an area for storing data necessary for realizing the present embodiment is secured. The three-dimensional region model data 501 is three-dimensional region model data including a three-dimensional blood vessel model in the three-dimensional STL format generated and received from CT data. The first feature point data 504 is data of a first feature point extracted from the three-dimensional part model data 501. The divided area data 506 is data of a divided area divided from the three-dimensional part model data 501, including the first feature point data 504. The ultrasound model data 619 is three-dimensional image data including a three-dimensional blood vessel image generated from an ultrasound signal detected in real time by the ultrasound probe 203. The second feature point data 624 is data of a second feature point extracted from the ultrasound model data 619. The divided area matching table 730 is a table shown in FIG. 7C for the divided area matching unit 431 to connect the divided areas by matching the first feature point and the second feature point included in the divided area. The HMD position data 804 is position data of the HMD 295 calculated based on the image of the optical marker captured by the web camera 294. The display image data 805 is data generated for three-dimensional display on the HMD 295 by superimposing a three-dimensional part model based on position-adjusted and connected divided areas on the patient 201. Each data of the RAM 940 may be stored as a table as shown in FIG. 5, FIG. 6A, FIG. 6B, FIG. 7C and FIG.

  The storage 950 stores a database, various parameters, or the following data or program necessary for realizing the present embodiment. The feature point extraction algorithm 951 is an algorithm for extracting the first feature point or the second feature point. The area division algorithm 952 is an algorithm for dividing the three-dimensional part model into small areas including the first feature points. The divided area matching algorithm 953 is an algorithm for matching the divided area including the first feature point with the second feature point.

  The storage 950 stores the following programs. The image processing program 955 is a program that controls the entire image processing apparatus 280. The feature point extraction module 956 is a module for extracting the first feature point or the second feature point according to the feature point extraction algorithm 951. The area dividing module 957 is a module for generating a divided area including the first feature point according to the area dividing algorithm 952. The divided area matching module 958 is a module for adjusting and connecting the divided areas so that the first feature point included in the divided area matches the second feature point according to the divided area matching algorithm 953. The display image generation module 959 is a module that generates a display image to be three-dimensionally displayed on the HMD 295 so that the STL data representing the deformed three-dimensional part model is inside the living body of the patient 201.

  The input / output interface 960 interfaces input / output data with input / output devices. A display unit 961 and an operation unit 962 are connected to the input / output interface 960, if necessary. When the medical examination / surgical instrument group is connected by wire, the ultrasonic probe 203 with the magnetic sensor 231 included in the medical examination / surgical instrument group, the magnetic sensor 293 with an optical marker, the web camera 294, and the HMD 295 have an input / output interface 960. Connected to In addition, if necessary, a voice input / output unit or a GPS position determination unit may be connected to the input / output interface 960.

  In the RAM 940 and the storage 950 of FIG. 9, programs and data related to general-purpose functions of the image processing apparatus 280 and other realizable functions are not shown.

Processing Procedure of Image Processing Device
FIG. 10 is a flowchart showing the processing procedure of the image processing apparatus 280 according to the present embodiment. This flowchart is executed by the CPU 910 in FIG. 9 using the RAM 940, and implements the functional components in FIG. 4 and FIG. 7A.

  In step S1001, the image processing device 280 receives three-dimensional region model data including a three-dimensional blood vessel model from the external three-dimensional model generation device 212. When STL data is generated from CT data in the image processing apparatus 280, the CT data is received and the STL data is generated, and then three-dimensional part model data is generated. In step S1003, the image processing device 280 executes a first feature point extraction process based on the three-dimensional part model data. In step S1005, the image processing device 280 executes area division processing of a three-dimensional part model represented, for example, in the STL format into small areas including first feature points.

  The image processing device 280 acquires ultrasonic image data from the ultrasonic probe 203 and the magnetic sensor position (ultrasound probe position) from the magnetic sensor 231 and the magnetic sensor 293 in step S1007. Then, in step S1009, the image processing device 280 accumulates the position of each ultrasonic probe 203 and the ultrasonic image data, and generates three-dimensional ultrasonic model data. In step 1011, the image processing device 280 executes a second feature point extraction process based on the three-dimensional ultrasound model data.

  In step S1013, the image processing device 280 performs matching between the first feature point and the second feature point included in the divided area, and determines whether the first feature point of the divided area matches the second feature point. Do. If the first feature point of the divided area does not match the second feature point, the image processing device 280 adjusts the position of the divided area in step S1015 to adjust the first feature point. In the case of divided area data in the STL format, conversion of STL data is performed. Thus, matching between the first feature point and the second feature point of the divided area is repeated.

  If the first feature point of the divided area matches the second feature point, the image processing apparatus 280 links the divided area with another divided area in step S1017. Then, in step S1019, the image processing apparatus 280 determines whether or not connection of the small areas divided in step S1005 is completed. If connection of divided areas is not completed, the image processing apparatus 280 returns to step S1013 and continues position adjustment and connection processing of divided areas for the next divided area.

  Since the three-dimensional part model corresponding to the three-dimensional ultrasonic model is generated when the connection of the divided areas is completed, the image processing apparatus 280 generates the magnetic marker 293 with the optical marker from the magnetic sensor 293 disposed on the patient 201 in step S1021. The sensor position (patient position) is acquired and adjusted so as to display the three-dimensional part model deformed by the division processing and the connection processing inside the living body of the patient 201. Next, in step S 1023, the image processing device 280 acquires an optical marker image of the magnetic sensor with optical marker 293 from the web camera 294. Then, the image processing device 280 calculates the positional relationship between the patient 201 and the HMD 295 from the optical marker image, and the display data of the HMD 295 so that the three-dimensional region model deformed by the division process and the connection process overlaps with the patient's view. Generate and output

  In step S1025, the image processing device 280 determines whether or not the display of the three-dimensional part model inside the living body of the patient 201 is ended. If the display of the three-dimensional region model is not completed, the image processing device 280 returns to step S1021 and repeats the display of the three-dimensional region model aligned with the ultrasound image inside the living body of the patient 201 in real time. Then, for example, when the shape of the liver is deformed, another scan using ultrasound is performed to perform new alignment. That is, if the shape of the liver does not change, the observation of the liver continues without scanning by ultrasound.

(Feature point extraction processing)
FIG. 11 is a flowchart showing the procedure of feature point extraction processing (S1003 / S1011) according to the present embodiment.

  In step S1111, the image processing device 280 acquires target three-dimensional model data. In the case of the first feature point extraction process (S1003), the target three-dimensional model data is three-dimensional part model data, and in the case of the second feature point extraction process (S1011), the target three-dimensional model data is It is ultrasonic model data. In step S1113, the image processing device 280 extracts blood vessel model data from the acquired three-dimensional model data. In the case of the first feature point extraction process (S1003), if blood vessel model data is obtained in advance, step S1113 is not necessary.

  In step S1115, the image processing device 280 extracts the blood vessel core line based on the blood vessel model data, and generates an abstracted blood vessel pattern. Next, in step S1117, the image processing device 280 extracts a branch point of the blood vessel as a feature point based on the pattern of the blood vessel core. Then, in step S1119, the image processing device 280 stores the set of the feature data of the branch point and its three-dimensional position in a comparable manner.

(Area division processing)
FIG. 12 is a flowchart showing the procedure of area division processing (S1005) according to the present embodiment.

  In step S1201, in this example, the image processing device 280 divides the three-dimensional region model into regions including at least three branch points as first feature points in the blood vessel core line. Next, in step S1203, the image processing apparatus 280 assigns an identifier (ID) to each divided area. Then, in step S1205, the image processing apparatus 280 stores the first feature point set in association with the divided area ID.

  In the present embodiment, the display of the deformed three-dimensional part model on the optical see-through HMD is shown, but the deformed three-dimensional part model is superimposed on the living body image of the patient captured by the web camera 294 It may be displayed on a video through HMD, a monitor, or a portable terminal.

  According to the present embodiment, since the three-dimensional part model deformed corresponding to the three-dimensional ultrasonic model is superimposed and displayed on the living body of the patient, the state of the part in real time inside the living body of the patient can be It can be recognized at the correct position.

Third Embodiment
Next, a medical support system according to a third embodiment of the present invention will be described. As compared with the second embodiment, the medical support system according to the present embodiment deforms a three-dimensional region model obtained from a CT image or the like corresponding to an ultrasonic image and superimposes it on an internal image observed by a laparoscope Differs in that it is displayed. The other configurations and operations are similar to those of the second embodiment, and therefore the same configurations and operations are denoted by the same reference numerals and the detailed description thereof is omitted.

<< Composition of medical support system >>
The configuration and operation of the medical support system of the present embodiment will be described with reference to FIGS. 13A and 13B.

(System overview)
FIG. 13A is a block diagram showing an outline of processing of the medical support system 1300 according to the present embodiment. In FIG. 13A, the same components as in FIG. 2A are assigned the same reference numerals and descriptions thereof will be omitted.

  In the medical support system 1300, a rod-shaped ultrasonic probe 1303 is inserted into the body, and an ultrasonic image around the site observed by the laparoscope 1391 is acquired. Also, using the position detection instrument fixed to the laparoscope 1391 for observing the inside of the living body in the operating room 1330 in real time, a three-dimensional image deformed corresponding to the observation image of the laparoscope 1391 and the real time ultrasonic image The positional relationship with the part model is measured. Then, based on the measured positional relationship, the three-dimensional region model is adjusted and superimposed on the observation image of the laparoscope 1391 to display a three-dimensional display 1370. In FIG. 13A, the blood vessel model 1371 is clearly illustrated as a site model superimposed on the stippled liver image 1372 which is an observation image of the laparoscope 1391 in the three-dimensional display 1370. The internal results are displayed superimposed. The display unit that performs three-dimensional display of the three-dimensional region model by superimposing it on the observation image of the laparoscope 1391 is preferably a portable terminal, a monitor, or a video see-through HMD. Here, as a position detection instrument, a magnetic sensor fixed to the tip of the laparoscope 1391 is used. Further, in order to obtain the positional relationship between the observation image and the display of the three-dimensional part model with high accuracy, it is desirable to fix the optical marker to the organ.

(System configuration)
FIG. 13B is a block diagram showing the configuration of a medical support system 1300 according to the present embodiment. In FIG. 13B, the same components as in FIG. 2B are assigned the same reference numerals and descriptions thereof will be omitted.

  The medical support system 1300 includes a CT apparatus 211, a three-dimensional model generation apparatus 212, an image processing apparatus 1380, and a medical examination / surgical instrument group of a medical examination room and an operating room 1330.

  The medical examination / surgical instrument group in the examination room or operation room 1330 includes a rod-shaped ultrasonic probe 1303, a magnetic sensor 1331 fixed to the ultrasonic probe, a laparoscope 1391 and a magnetic sensor 1392 fixed to the laparoscope It has a monitor 1370 and a magnetic sensor 1393 fixed to a living body or a target organ. The magnetic sensor 1331 and the magnetic sensor 1393 enable stable position detection of the ultrasonic probe 1303. The laparoscope 1391 acquires in vivo images from the patient 201 during real time examination or surgery. The magnetic sensor 1392 is disposed at the distal end position of the laparoscope 1391 to detect the observation position of the laparoscope 1391. The magnetic sensor 1392 and the magnetic sensor 1393 allow stable position detection of the laparoscope 1391.

  The image processing apparatus 1380 divides the three-dimensional region model represented by the three-dimensional data into regions, and matches the three-dimensional ultrasonic image acquired by the ultrasonic probe 1303 whose position and orientation are determined by the detection signal of the magnetic sensor 1331. It is deformed by connecting to. Then, the image processing device 1380 generates three-dimensional display data in which the deformed three-dimensional region model is superimposed on the observation image acquired by the laparoscope 1391 and causes the monitor 1370 to display the data. In addition, a laparoscopic treatment tool 1373 operated by the operator 202 is observed by the laparoscope 1391 and displayed on the monitor 1370.

  Although only the operator 202 is illustrated in FIG. 13B, in actual laparoscopic surgery, it may be operated by a plurality of persons such as a surgeon operating an ultrasound probe and a scopist operating an laparoscope. become. Alternatively, a three-dimensional region model matched with the observation image acquired by the laparoscope 1391 may be superimposed and displayed on the video see-through HMD.

<< Functional Configuration of Image Processing Device >>
FIG. 14 is a block diagram showing the functional configuration of the image processing apparatus 1380 according to this embodiment. In FIG. 14, the same functional components as in FIG. 4 will be assigned the same reference numerals and descriptions thereof will be omitted.

  The magnetic signal reception unit 1419 receives, via the communication control unit 413, a magnetic detection signal from the magnetic sensor 1392 disposed in the laparoscope 1391. The laparoscopic position detection unit 1420 detects the position and orientation of the laparoscope 1391 based on the magnetic detection signal from the magnetic sensor 1392 and the magnetic detection signal from the magnetic sensor 1393. When the magnetic sensor 1392 has a storage unit for detecting the position and orientation from the magnetic detection signal, data (for example, coordinates and vectors) indicating the position and orientation may be received. The laparoscopic image reception unit 1421 receives an observation image from the laparoscope 1391 via the communication control unit 413. When the laparoscope 1391 has a function of generating laparoscopic image data, the laparoscopic image reception unit 1421 may directly receive laparoscopic image data. The magnetic signal receiving unit 1422 receives, via the communication control unit 413, a magnetic detection signal from the magnetic sensor 1393 disposed in the living body 201 or a target organ in the living body.

  The generation model superimposing unit 1433 of the display image generation unit 1423 matches the real-time three-dimensional ultrasound model by dividing the three-dimensional part model represented by the STL format or the like into regions and linking them so that the feature points correspond. An image is generated by superimposing the three-dimensional region model deformed as described above on the observation image acquired by the laparoscope 1391. Then, the generated model superimposing unit 1433 transmits the generated superimposed image from the display image transmitting unit 425 to the monitor 1370, the portable terminal, or the video see-through HMD and causes the display to be displayed.

  According to the present embodiment, since the three-dimensional part model deformed corresponding to the three-dimensional ultrasonic image is superimposed and displayed on the in-vivo image acquired by the laparoscope, the part in real time inside the patient's in-vivo The state can be accurately recognized with the image inside the living body.

Fourth Embodiment
Next, a medical support system according to a fourth embodiment of the present invention will be described. The medical support system according to the present embodiment is different from the second embodiment in that a three-dimensional part model deformed corresponding to a three-dimensional ultrasonic model is displayed on a sheet display on a living body of a patient. The other configurations and operations are similar to those of the second embodiment, and therefore the same configurations and operations are denoted by the same reference numerals and the detailed description thereof is omitted.

<< Composition of medical support system >>
FIG. 15 is a block diagram showing the configuration of a medical support system 1500 according to the present embodiment. In FIG. 15, the same components as in FIG. 2B or FIG. 13B carry the same reference numerals, and the description thereof is omitted.

  The medical support system 1500 includes a CT apparatus 211, a three-dimensional model generation apparatus 212, an image processing apparatus 1580, and a medical examination / surgical instrument group of a medical examination room or an operating room 1530.

  The medical examination / surgical instrument group in the medical examination room or operating room 1530 includes an ultrasonic probe 203, a magnetic sensor 231 fixed to the ultrasonic probe, a magnetic sensor 293 with an optical marker, a video camera 1594, and a sheet display 1595. Have. The video camera 1594 captures an optical marker of the optical sensor with a magnetic marker 293 disposed on the sheet display 1595. The sheet display 1595 disposed on the surface of the patient 201 displays the three-dimensional model represented by the three-dimensional data deformed to match the three-dimensional ultrasonic model at an accurate position inside the living body.

  The image processing device 1580 matches the three-dimensional region model represented by the three-dimensional data with the three-dimensional ultrasonic model obtained by accumulating the ultrasonic signal from the ultrasonic probe 203 and the positional information of the ultrasonic probe 203. To be transformed. Then, the image processing device 1580 generates three-dimensional display data so that the deformed three-dimensional region model is at an accurate position inside the living body of the patient 201, and causes the sheet display 1595 to perform three-dimensional display.

  In the present embodiment, since the sheet display 1595 is the same as the living body surface of the patient 201, the image processing device 1580 may detect the position and orientation of the patient 201 and the sheet display 1595 from the captured image of the optical marker. There is no need to calculate the positional relationship between the and the sheet display 1595.

<< Functional Configuration of Image Processing Device >>
FIG. 16 is a block diagram showing the functional configuration of the image processing apparatus 1580 according to this embodiment. In FIG. 16, the same functional components as in FIG. 4 or FIG. 14 are assigned the same reference numerals and descriptions thereof will be omitted.

  The sheet position calculation unit 1622 calculates an accurate position of the sheet display 1595 based on the captured image of the optical marker received by the optical marker image reception unit 421.

  The sheet image generation unit 1633 of the display image generation unit 1623 calculates the sheet display calculated by the sheet position calculation unit 1622 so that the three-dimensional part model by the SLT data adjusted to the position of the patient 201 detected by the patient position detection unit 420 overlaps. Generate a display image to be displayed three-dimensionally at the position. The display image transmission unit 1625 transmits the image data overlapping the position of the patient 201 generated by the sheet image generation unit 1633 to be displayed on the sheet display 1595 via the communication control unit 413.

  The configuration and operation of the sheet image generation unit 1633 are the same as those of the second embodiment except that the display destination of the three-dimensional part model is changed from the HMD 295 to the sheet display 1595, and thus the detailed description is omitted here. However, in this example, since the sheet display 1595 is non-rigidly deformed, a sheet-like non-rigid deformation operation is required, and the existing technology can be used.

  According to the present embodiment, since the deformed three-dimensional part model is displayed on the sheet display on the patient's living body, the state of the part in the patient's living body in real time is normally corrected without considering the line of sight or the like. It can be recognized in position.

Fifth Embodiment
Next, a medical support system according to a fifth embodiment of the present invention will be described. The medical support system according to the present embodiment is different from the third embodiment in that an image obtained by superimposing the deformed three-dimensional region model on the observation image of the laparoscope 1391 is displayed on the communication terminal via the network. . The other configurations and operations are similar to those of the third embodiment, and therefore the same configurations and operations are denoted by the same reference numerals and the detailed description thereof will be omitted.

<< Composition of medical support system >>
FIG. 17 is a block diagram showing the configuration of a medical support system 1700 according to the present embodiment. In FIG. 17, the same components as those in FIGS. 2B, 13B, and 15 are assigned the same reference numerals and descriptions thereof will be omitted.

  In FIG. 17, the image processing device 1780 displays a three-dimensional region model by superimposing on the observation image of the laparoscope 1391 as in the image processing device of FIG. 13B. At the same time, the image processing apparatus 1780 generates an image in which the three-dimensional region model is superimposed on the observation image captured by the laparoscope 1391 and distributes the image to another external doctor or user via the network 1790. A notebook PC 1791, a portable terminal 1792, such as a smartphone or a tablet, a monitor 179i, a server 179n, or the like may be connected to the delivery destination.

  According to this embodiment, other doctors and third parties can monitor the observation image of the laparoscope inside the living body of the patient outside the doctor's office or the operating room, and can support more accurate medical care.

[Other embodiments]
In the above embodiment, only a limited configuration example of the medical support system of the present invention is shown. FIG. 18 is a view showing various configuration examples 1800 of the medical support system of the present invention. Although the present invention is not limited to the example of FIG. 18, the scope of application of the present invention is apparent from FIG.

  In the configuration example 1800 of FIG. 18, a three-dimensional region model 1801 deformed and displayed, a three-dimensional ultrasound model 1802 used for alignment of the three-dimensional region model 1801, and a region model matching method (alignment method) An object 1804 on which the three-dimensional part model 1801 is superimposed, and a display unit 1805 are shown. These are independently combined to achieve more preferable alignment and superposition.

  Furthermore, although the present invention has been described with reference to the embodiment, the present invention is not limited to the above embodiment. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. Also included within the scope of the present invention are systems or devices that combine the different features included in each embodiment.

  Furthermore, the present invention may be applied to a system configured of a plurality of devices or to a single device. Furthermore, the present invention is also applicable to the case where a control program for realizing the functions of the embodiments is supplied to a system or apparatus directly or remotely. Therefore, in order to realize the functions of the present invention on a computer, a program installed on the computer, a medium storing the program, and a WWW (World Wide Web) server for downloading the program are also included in the scope of the present invention. . In particular, a non-transitory computer readable medium storing a program that causes a computer to execute at least the processing steps included in the above-described embodiment is included in the scope of the present invention.

Claims (14)

A part model generation unit that generates data of a three-dimensional part model based on a tomographic image group of a patient;
First feature point extraction means for extracting position information of a plurality of first feature points having predetermined features from the three-dimensional part model;
The ultrasound data acquired for the patient using the ultrasound probe and the position and orientation data of the ultrasound probe for which the ultrasound data is acquired are stored in association with each other, and the ultrasound data, the position, and the position Ultrasonic model generation means for generating data of a three-dimensional ultrasonic model using orientation data;
Second feature point extraction means for extracting position information of the plurality of second feature points having the predetermined feature from the three-dimensional ultrasound model;
Deformation means for deforming the three-dimensional part model so that the plurality of first feature points and the plurality of second feature points overlap;
Display means for displaying a projection image of the three-dimensional part model deformed by the deformation means;
Equipped with
The deformation means is
Division area generation means for dividing the three-dimensional part model into a plurality of three-dimensional areas each including at least one of the first feature points;
Each of the plurality of three-dimensional regions is position-adjusted such that the plurality of first feature points and the plurality of second feature points overlap, and a division region connection is made to connect the plurality of position-adjusted plurality of three-dimensional regions. Means,
Have
The display means, the medical support system that displays a projection image of the divided plurality of three-dimensional area region connecting means is connected as a projection image of the three-dimensional site model. The plurality of first feature points are feature points of a three-dimensional blood vessel model included in a focused portion of the three-dimensional portion model, and the plurality of second feature points are positioned at the focused portion of the three-dimensional ultrasound model. It is a feature point of the included three-dimensional blood vessel model,
The medical support system according to claim 1, wherein the display means displays the three-dimensional blood vessel model as the three-dimensional region model. Imaging means for imaging the living body of the patient and a first position detection instrument for detecting the position and direction of the living body or the position and direction of the living body image;
First adjustment means for adjusting the position, orientation and size of the three-dimensional part model deformed by the deformation means based on the position and orientation of the first position detection instrument imaged;
And further
The medical support system according to claim 1 or 2 , wherein the display means displays the projection image of the deformed and adjusted three-dimensional part model so as to overlap the living body or the living body image of the patient. The medical support system according to claim 3 , wherein the first position detection instrument is a magnetic sensor or an optical marker with an optical marker. A laparoscope fixed with a second position detection instrument for imaging the inside of the patient's living body;
A second adjustment for adjusting the position, orientation and size of the three-dimensional part model deformed by the deformation means based on the position and orientation of the second position detection instrument and the position and orientation of the ultrasonic probe Means,
And further
The medical support system according to claim 1 or 2 , wherein the display means displays the projection image of the deformed and adjusted three-dimensional region model so as to overlap the in-vivo image captured by the laparoscope. The medical support system according to claim 5 , wherein the second position detection instrument is a magnetic sensor. The medical support system according to any one of claims 1 to 6 , further comprising ultrasonic probe position detection means for detecting the position and orientation of the third position detection instrument fixed to the ultrasonic probe. The medical support system according to claim 7 , wherein the third position detection instrument is an optical marker or a magnetic sensor. The display means is an optical see-through head mounted display for displaying the projected image of the deformed and adjusted three-dimensional part model and the living body of the patient so as to overlap each other, or the deformed and adjusted three-dimensional part model The medical support system according to any one of claims 1 to 8 , wherein the medical support system is a video see-through head-mounted display that displays the projection image of and the living body image of the patient in an overlapping manner. The sheet display according to any one of claims 1 to 8 , wherein the display means is a sheet display for displaying the projection image of the deformed and adjusted three-dimensional part model and the living body or the living body image of the patient in an overlapping manner. Medical support system. A part model generation step of generating data of a three-dimensional part model based on a tomographic image group of a patient;
A first feature point extraction step of extracting position information of a plurality of first feature points having predetermined features from the three-dimensional part model;
An accumulation step of correlating the ultrasound data acquired for the patient using the ultrasound probe with the position and orientation data of the ultrasound probe for which the ultrasound data is acquired, and accumulating the data in an accumulation unit;
An ultrasound model generation step of generating data of a three-dimensional ultrasound model using the ultrasound data and the position and orientation data;
A second feature point extraction step of extracting position information of the plurality of second feature points having the predetermined feature from the three-dimensional ultrasound model;
Deforming the three-dimensional part model so that the plurality of first feature points and the plurality of second feature points overlap;
A display step of displaying a projection image of the three-dimensional part model deformed in the deformation step;
Only including,
The deformation step is
A division area generation step of dividing the three-dimensional part model into a plurality of three-dimensional areas each including at least one of the first feature points;
Each of the plurality of three-dimensional regions is position-adjusted such that the plurality of first feature points and the plurality of second feature points overlap, and a division region connection is made to connect the plurality of position-adjusted plurality of three-dimensional regions. Step and
Including
The medical support method which displays the projection image of said some three-dimensional area | region connected in the said division area connection step in the said display step as a projection image of the said three-dimensional site | part model . Receiving means for receiving data of a three-dimensional part model generated based on a group of tomographic images of the patient;
First feature point extraction means for extracting position information of a plurality of first feature points having predetermined features from the three-dimensional part model;
Storage means for correlating and storing ultrasound data acquired for the patient using an ultrasound probe and position and orientation data of the ultrasound probe for which the ultrasound data is acquired;
Ultrasonic model generation means for generating data of a three-dimensional ultrasonic model using the ultrasonic data and the position and orientation data;
Second feature point extraction means for extracting position information of the plurality of second feature points having the predetermined feature from the three-dimensional ultrasound model;
Deformation means for deforming the three-dimensional part model so that the plurality of first feature points and the plurality of second feature points overlap;
Transmission means for transmitting the projection image of the three-dimensional part model deformed by the deformation means to display means;
Equipped with
The deformation means is
Division area generation means for dividing the three-dimensional part model into a plurality of three-dimensional areas each including at least one of the first feature points;
Each of the plurality of three-dimensional regions is position-adjusted such that the plurality of first feature points and the plurality of second feature points overlap, and a division region connection is made to connect the plurality of position-adjusted plurality of three-dimensional regions. Means,
Have
The transmission unit, an image processing apparatus that sends a projection image of the divided region of the plurality of three-dimensional area coupling means coupled to said display means as a projection image of the three-dimensional site model. Receiving data of a three-dimensional part model generated based on a group of tomographic images of the patient;
A first feature point extraction step of extracting position information of a plurality of first feature points having predetermined features from the three-dimensional part model;
An accumulation step of correlating the ultrasound data acquired for the patient using the ultrasound probe with the position and orientation data of the ultrasound probe for which the ultrasound data is acquired, and accumulating the data in an accumulation unit;
An ultrasound model generation step of generating data of a three-dimensional ultrasound model using the ultrasound data and the position and orientation data;
A second feature point extraction step of extracting position information of the plurality of second feature points having the predetermined feature from the three-dimensional ultrasound model;
Deforming the three-dimensional part model so that the plurality of first feature points and the plurality of second feature points overlap;
A transmitting step of transmitting a projection image of the three-dimensional part model deformed in the deformation step to display means;
Only including,
The deformation step is
A division area generation step of dividing the three-dimensional part model into a plurality of three-dimensional areas each including at least one of the first feature points;
Each of the plurality of three-dimensional regions is position-adjusted such that the plurality of first feature points and the plurality of second feature points overlap, and a division region connection is made to connect the plurality of position-adjusted plurality of three-dimensional regions. Step and
Including
The control method of the image processing apparatus, in the transmitting step, transmitting the projection images of the plurality of three-dimensional regions connected in the division region connecting step to the display unit as a projection image of the three-dimensional part model . Receiving data of a three-dimensional part model generated based on a group of tomographic images of the patient;
A first feature point extraction step of extracting position information of a plurality of first feature points having predetermined features from the three-dimensional part model;
An accumulation step of correlating the ultrasound data acquired for the patient using the ultrasound probe with the position and orientation data of the ultrasound probe for which the ultrasound data is acquired, and accumulating the data in an accumulation unit;
An ultrasound model generation step of generating data of a three-dimensional ultrasound model using the ultrasound data and the position and orientation data;
A second feature point extraction step of extracting position information of the plurality of second feature points having the predetermined feature from the three-dimensional ultrasound model;
Deforming the three-dimensional part model so that the plurality of first feature points and the plurality of second feature points overlap;
A transmitting step of transmitting a projection image of the three-dimensional part model deformed in the deformation step to display means;
A control program of an image processing apparatus which causes a computer to execute
In the deformation step,
A division area generation step of dividing the three-dimensional part model into a plurality of three-dimensional areas each including at least one of the first feature points;
Each of the plurality of three-dimensional regions is position-adjusted such that the plurality of first feature points and the plurality of second feature points overlap, and a division region connection is made to connect the plurality of position-adjusted plurality of three-dimensional regions. Step and
On your computer,
In the transmission step, a control program of an image processing apparatus which causes a computer to execute projection images of the plurality of three-dimensional regions connected in the division region connecting step as the projection image of the three-dimensional part model .

Images