JP2016077574A - Identification marker and medical support system by augmented reality technology using the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an identification marker that can reduce a correction time of a display position of a virtual image, when aligning a display position of a photographed image of a patient's predetermined part with the virtual image superposed on the photographed image and displayed.SOLUTION: An identification marker is provided for superposing a photographed image photographed by photographing means and an associated virtual image, and displaying it by display means when photographed by the photographing means. The identification marker has a base body provided with a sewing hole, and sewn and fixed on a patient's predetermined part, and an identification image fixed on the base body and for displaying the associated virtual image by the display means when photographed by the photographing means.SELECTED DRAWING: Figure 1A

Description

  The present invention relates to an identification marker and a medical support system based on augmented reality technology using the identification marker.

Conventionally, when it is recognized that a predetermined identification image is included in a photographed image photographed by photographing means such as a digital video camera, a virtual image (three-dimensional computer graphics) corresponding to the identification image is captured in real time. AR (Augmented Reality) technology that displays images superimposed on each other is used. As one of the AR techniques, for example, there is a technique that uses an AR identification marker (two-dimensional barcode) as the identification image. In this technique, when the AR identification marker is recognized in the captured image, the virtual image associated with the AR identification marker is superimposed on the captured image at a position where the AR identification marker is recognized in the captured image. Display on a display means such as a liquid crystal display.
2. Description of the Related Art A medical support apparatus that provides information on surgery in real time to an operator using the AR technology is already known.
When the medical support device recognizes that the AR identification marker affixed in advance to a position away from the part, not the part to be operated on, is included in the captured image, the display and position of the AR identification marker Medical support that displays the virtual image of a three-dimensional part in the patient's body according to the posture and size superimposed on the captured image, and shows the operator the cut location, blood vessel position, etc. during surgery An apparatus has been proposed (see Patent Document 1).
However, although the medical support device can show the virtual image based on the display of the AR identification marker or the like to the surgeon, the AR identification marker is pasted at a position away from the site to be operated on. Therefore, if the virtual image indicating the position of the blood vessel of the part is superimposed on the display position of the part to be operated on in the captured image and displayed, the correction amount of the display position of the virtual image is large. Therefore, there is a problem that it takes time. Further, since the AR identification marker is affixed to the patient's skin or the like with a double-sided tape, the affixing position of the AR identification marker is shifted due to a patient's body fluid or the like, and the display position of the virtual image is likely to be unstable. It is necessary to correct the display position of the virtual image every time the AR identification marker application position is shifted. These are not effective because they can be related to the lives of patients in surgery that competes for 1 second.

JP 2014-131552 A

  An object of the present invention is to solve the above-described problems and achieve the following objects. That is, according to the present invention, it is possible to reduce the correction time of the display position of the virtual image when matching the display position of the captured image of a predetermined part of the patient with the virtual image to be displayed superimposed on the captured image. The object is to provide a label.

Means for solving the above-mentioned problems are as follows. That is,
<1> When photographed by the photographing means, an identification marker for superimposing a photographed image photographed by the photographing means and a corresponding virtual image on the display means,
A base body provided with a sewing hole and sewn and fixed to a predetermined part of a patient;
And an identification image for displaying the corresponding virtual image by the display means when photographed by the photographing means and fixed to the base body.
In the identification sign body according to <1>, the identification image is fixed to the substrate that is directly sewn to the part. Therefore, when the identification image is captured by the imaging unit, the captured image is captured. The corresponding virtual image is superimposed on the photographed image and displayed by the display means at the position of the identification image in FIG.

<2> It has an identification display component on which an identification image is displayed, the identification display component has at least one protrusion, and the base has a fitting hole or groove for fitting and fixing the protrusion. It is the identification label | marker as described in said <1>.
The identification marker body according to <2> has an identification display component on which an identification image is displayed, the identification display component has at least one protrusion, and a base is fitted with the protrusion. The base has the identification display component on which the identification image is displayed fixed.

<3> The identification marker according to any one of <1> to <2>, wherein the base body and the identification display component are formed of a biocompatible material and are sterilizable.
In the identification sign body described in <3>, since the base body and the identification display part are formed of a biocompatible material and can be sterilized, the identification sign body can be directly sewn to a predetermined part of a patient. It becomes.

<4> The identification marker according to any one of <1> to <3>, wherein the identification image is an AR identification marker.
In the identification marker described in <4>, since the identification image is an AR identification marker, the AR technology is used when it is recognized that the AR identification marker is included in the captured image captured by the imaging unit. It becomes possible.

<5> The identification marker according to any one of <1> to <4>, wherein the surface of the identification display component on which the identification image is displayed has at least one concave portion that diffuses and reflects light.
In the identification sign body according to <5>, the surface of the identification display component on which the identification image is displayed has at least one concave portion that diffuses and reflects light. Therefore, when the surface is photographed by the photographing unit, This prevents erroneous recognition of the identification image due to regular reflection of light with the gloss.

<6> Imaging means for imaging a predetermined part of the patient to which the identification marker body according to any one of <1> to <5> is sewn and fixed during surgery;
Display means for displaying a photographed image including an identification image in the identification marker, photographed by the photographing means;
Generating means for generating identification image data of an identification image associated with each virtual image data of a stereoscopic part in the body of the patient imaged in advance;
Data storage means for storing the virtual image data and the identification image data generated by the generation means;
Extracting means for recognizing the identification image of the identification marker and extracting the identification image data based on the captured image data of the captured image including the identification image;
The display means compares the identification image data extracted by the extraction means with the identification image data stored in the data storage means, and the identification image data extracted by the extraction means is the data storage means When it is determined that the identification image data matches one of the identification image data, the predetermined virtual image of the patient based on the virtual image data corresponding to the identification image data of the matching data storage means is used as the identification image of the identification marker. In a medical support system, the image is superimposed and displayed on the captured image at a position where is recognized.
The medical support system according to <6> includes an imaging unit, a display unit, a generation unit, a data storage unit, and an extraction unit in addition to the identification marker. In the pre-operative stage, the generation unit generates identification image data of the identification image associated with each virtual image data of a three-dimensional part in the body of the patient that has been imaged in advance. The data storage unit stores the virtual image data and the identification image data generated by the generation unit. At the time of surgery, the imaging means images a predetermined part of the patient on which the identification marker is sewn and fixed. The display means displays a photographed image including an identification image on the identification marker photographed by the photographing means. The extraction means recognizes the identification image based on the captured image data of the captured image including the identification image, and extracts the identification image data. Then, the medical support system compares the identification image data extracted by the extraction unit with the identification image data stored in the data storage unit, and the identification image data extracted by the extraction unit is stored in the data storage A virtual image of a predetermined part of the patient based on the virtual image data corresponding to the identification image data of the data storage means that matches the identification image data, The display unit superimposes the identification image on the body at the recognized position and displays the image. Therefore, the medical support system superimposes and displays the virtual image at a position close to the part in the captured image.

<7> Tomographic means for obtaining tomographic image data of a patient's part, and a virtual image for generating virtual image data of a three-dimensional part in the patient's body based on the tomographic image data obtained by the tomographic means The medical support system according to claim 6, further comprising a tomography apparatus comprising: a data generation unit; and a communication unit that transmits the virtual image data generated by the virtual image data generation unit to store the data in the data storage unit. It is.
In the medical support system according to <7>, a tomographic unit that acquires tomographic image data of a part of a patient, and a three-dimensional body inside the patient based on the tomographic image data acquired by the tomographic unit. A tomography apparatus having virtual image data generation means for generating virtual image data of a part and communication means for transmitting the virtual image data generated by the virtual image data generation means; Send to store in advance.

  According to the present invention, the conventional problems can be solved, and when the display position of a captured image of a predetermined part of a patient and a virtual image to be displayed superimposed on the captured image are matched, the virtual image It is possible to provide an identification marker that can reduce the display position correction time.

FIG. 1A is a schematic perspective view showing an example of an identification marker in the present invention. FIG. 1B is a schematic cross-sectional view of the identification marker shown in FIG. 1A. FIG. 2A is a schematic perspective view showing another example of the identification marker in the present invention. FIG. 2B is a schematic cross-sectional view of the identification marker shown in FIG. 2A. FIG. 3A is a schematic perspective view showing another example of the identification marker in the present invention. FIG. 3B is a schematic cross-sectional view of the identification marker shown in FIG. 3A. FIG. 4A is a schematic cross-sectional view showing a procedure for fixing the identification marker in the present invention to a predetermined part of a patient. FIG. 4B is an explanatory diagram when the identification marker shown in FIG. 4A is viewed from above. FIG. 5A is a schematic cross-sectional view showing a procedure for fixing the identification marker in the present invention to a predetermined part of a patient. FIG. 5B is an explanatory diagram when the identification marker shown in FIG. 5A is viewed from above. FIG. 6A is a schematic cross-sectional view showing a procedure for fixing the identification marker in the present invention to a predetermined part of a patient. FIG. 6B is an explanatory diagram when the identification marker shown in FIG. 6A is viewed from above. FIG. 7A is a schematic cross-sectional view showing a procedure for fixing the identification marker in the present invention to a predetermined part of a patient. FIG. 7B is an explanatory diagram when the identification marker shown in FIG. 7A is viewed from above. FIG. 8 is a schematic cross-sectional view showing a procedure for removing the identification marker in the present invention from a predetermined part of a patient. FIG. 9 is an explanatory diagram showing a medical support system based on augmented reality technology using the identification marker in the present invention. FIG. 10 is a block diagram showing a medical support system based on augmented reality technology using an identification marker in the present invention. FIG. 11 is a flowchart showing a flow of processing in which the tomography apparatus according to the present invention generates and transmits virtual image data. FIG. 12 is a flowchart showing a flow of processing in which the input / output terminal device according to the present invention generates identification image data and stores it together with virtual image data. FIG. 13 is a flowchart showing a flow of processing in which the input / output terminal device according to the present invention displays a virtual image superimposed on a captured image. FIG. 14A is an explanatory diagram showing a screen when only a virtual image is displayed in the present invention. FIG. 14B is an explanatory diagram showing a screen displaying a photographed image when photographing the identification marker and the liver in the present invention. FIG. 14C is an explanatory diagram showing a screen on which a virtual image of the liver is superimposed and displayed when the identification marker and the liver are photographed in the present invention.

(Identification marker)
The identification marker of the present invention includes an identification image, an identification display component, and a base, and may further include other members appropriately selected as necessary.

<Identification image>
The identification image is not particularly limited in shape and size as long as it is an image that can be captured by a later-described capturing unit such as a digital video camera, and can be appropriately selected according to the purpose.
The shape of the identification image is not particularly limited as long as it can be extracted as identification image data by an extraction unit (to be described later) that recognizes the display, position, orientation, and size from the captured image captured by the imaging unit. However, it can be selected appropriately according to the purpose. For example, a shape including a symbol, a character, a picture enclosed in a black square like a general AR (Augmented Reality) identification marker, a character, a picture, a QR code ( (Registered trademark) and the like. Moreover, in order to represent the above shapes, for example, the surface of the identification display component may be irradiated with a laser or the like to be changed to black.
The size of the identification image is not particularly limited as long as it does not interfere with the operation, and can be appropriately selected according to the purpose.
In addition, the identification image in an example described later is an AR identification marker, and is used when the AR technology is used. When the AR technology recognizes that the identification image is included in the captured image, a virtual image corresponding to the display, position, orientation, and size of the recognized identification image is superimposed on the captured image in real time. It is displayed on a display means such as a liquid crystal display monitor, which will be described later.
Note that the virtual image refers to three-dimensional computer graphics. Among computer graphics, computer graphics are generated by generating information on a three-dimensional space on a computer and projecting a virtual three-dimensional world. is there. The virtual image data is information for displaying the virtual image on the display means.
In the present invention, a tomography apparatus (to be described later) such as a CT (Computed Tomography) scanning apparatus generates virtual image data for each part of the patient's body based on the tomographic image data of the patient's body previously captured. Furthermore, the generation means described later generates the identification image associated with the virtual image data. For example, when the identification image associated with the virtual image data of the liver is generated and the identification image is fixed to the liver during surgery, the identification image fixed to the liver is captured by the imaging unit. The virtual image of the liver corresponding to the position, posture, and size of the identification image is superimposed and displayed at a position close to the liver in the captured image.
In addition, since the virtual image of the liver to be displayed can be translucent except for blood vessels by predetermined image processing, the liver in the captured image and the blood vessels other than the liver blood vessels to be displayed superimposed on the captured image are semi-transparent. By matching the display position with the virtual image that has been made transparent, it is possible to visually recognize the position of a blood vessel that does not require contact in the operation, and a more accurate operation is possible.

<Identification display parts>
The identification display component is not particularly limited in shape, structure, size, material, and the like, and can be appropriately selected according to the purpose.
The shape of the identification display component is not particularly limited as long as the entire identification image can be displayed, and can be appropriately selected according to the purpose. Examples thereof include a circular and square substantially flat plate shape. .
Further, the shape of the surface of the identification display component that displays the identification image is not particularly limited and can be appropriately selected according to the purpose. For example, erroneous recognition of the identification image due to regular reflection of light is prevented. Therefore, a shape having at least one concave portion that diffuses and reflects light by sandblasting or the like can be used.
The structure of the identification display component is not particularly limited as long as it can display the entire identification image and can be fixed to the base, and can be appropriately selected according to the purpose. In addition to the flat plate-like structure such as a quadrangle, or the flat plate-like structure, there may be mentioned a structure in which at least one protrusion is provided on the back surface for fitting and fixing in the fitting hole of the base. The identification display component may be structured to be detachably fixed to the base.
The size of the identification display component is not particularly limited as long as it is a size that does not interfere with surgery and can display the entire identification image, and can be appropriately selected according to the purpose.
The material of the identification display component is not particularly limited as long as it is a biocompatible material and can be sterilized because it contacts an organ or the like, and can be appropriately selected according to the purpose. For example, polyether ether ketone (PEEK), modified polyphenylene ether, and the like.

<Substrate>
There is no restriction | limiting in particular in the shape, a structure, a magnitude | size, a material, etc. as said base | substrate, According to the objective, it can select suitably.
There is no restriction | limiting in particular as a shape of the said base | substrate, According to the objective, it can select suitably, For example, circular, square flat plate shape etc. are mentioned.
The structure of the base body is not particularly limited as long as it has a structure capable of fixing the identification display component and has at least one sewing hole for sewing to a predetermined part of a patient. It can be selected as appropriate according to the conditions.
The predetermined site of the patient is an organ or the like that needs to be operated, and is a liver in this embodiment described later.
Further, when the structure of the identification display component is a flat plate shape, the base body may fix the entire identification display component with a frame.
In addition, for example, when a predetermined part of a patient is pierced with a suture needle that is locked with a suture threaded at one end, the distance between the sutures exposed from the surface of the part is likely to be the same. In order to make it easy to sew the base body to the portion, when at least one sewing hole is provided in the base body, at least any two of the plurality of sewing holes. The distance between the centers may be about 1.5 cm.
The size of the substrate is not particularly limited as long as it is a size that does not interfere with surgery and can display the entire identification image, and can be appropriately selected according to the purpose.
The material of the substrate is not particularly limited as long as it is a biocompatible material and can be sterilized for use in surgery, and can be appropriately selected according to the purpose. For example, polyether ether ketone (PEEK), Examples thereof include modified polyphenylene ether.
Although the identification display component for displaying the identification image and the base have been described as separate, there is no particular limitation, and the base can be appropriately selected according to the purpose. It is good also as the said identification label | marker body which has a hole and integrated the said base | substrate and the said identification display component.

(Medical support system)
The medical support system of the present invention includes the imaging means, the display means, the generation means, the data storage means, and the extraction means in addition to the identification marker, and further if necessary. For example, other members such as a terminal communication unit and a control unit may be appropriately selected.

<Photographing means>
The imaging means is not particularly limited in shape, structure, size, material, etc., as long as the entire identification image can be captured and the captured image is converted into an electrical signal. For example, a digital video camera using a CMOS (Complementary Metal-Oxide Semiconductor) image sensor, a CCD (Charge-Coupled Device) image sensor, or a tablet-type terminal having a digital video camera function. Etc.
There is no restriction | limiting in particular as a shape of the said imaging | photography means, According to the objective, it can select suitably.
There is no restriction | limiting in particular as a structure of the said imaging | photography means, According to the objective, it can select suitably, For example, the structure incorporated in the structure of a digital video camera itself, a tablet-type terminal, etc. are mentioned.
The size of the imaging means is not particularly limited as long as it is a size that does not interfere with surgery and can display the entire identification image, and can be appropriately selected according to the purpose. It is desirable to be small so as not to interfere with the operation.
There is no restriction | limiting in particular as a material of the said imaging | photography means, According to the objective, it can select suitably.

<Display means>
The display means is not particularly limited in shape, structure, size, material, and the like, and can be appropriately selected according to the purpose. For example, for a liquid crystal display, a digital video camera, a tablet terminal, or the like. Examples include on-board liquid crystal displays and projectors.
There is no restriction | limiting in particular as a shape of the said display means, According to the objective, it can select suitably.
The structure of the display means is not particularly limited and can be appropriately selected according to the purpose. For example, the structure of the liquid crystal display monitor itself, a liquid crystal display monitor incorporated in a digital video camera, a tablet terminal, or the like. For example.
The size of the display means is not particularly limited as long as it is a size that does not interfere with the operation and is visible to the operator, and can be appropriately selected according to the purpose.
There is no restriction | limiting in particular as a material of the said display means, According to the objective, it can select suitably.

<Generation means>
As the generation means, identification image data of an associated identification image is generated for each virtual image data of a three-dimensional part in the patient's body generated based on tomographic image data captured in advance by a tomography apparatus. If it does, there is no restriction | limiting in particular, According to the objective, it can select suitably.

<Data storage means>
The data storage means is not particularly limited as long as it can store the virtual image data and the identification image data generated by the generation means, and can be appropriately selected according to the purpose. Examples include media.

<Extraction means>
The extraction means may be capable of extracting the identification image display, position, orientation, and size information as the identification image data based on the captured image data of the captured image including the identification image. If there is no restriction | limiting in particular, according to the objective, it can select suitably.

<< Other means >>
There is no restriction | limiting in particular as said other means, According to the objective, it can select suitably, For example, a terminal communication means, a control means, etc. are mentioned.

<< Terminal communication means >>
The terminal communication means is not particularly limited as long as it can communicate with a connected device or the like, and can be appropriately selected according to the purpose.

<< Control means >>
The control means is not particularly limited as long as it controls each means based on software stored in a storage medium or the like, and can be appropriately selected according to the purpose.

(Tomography equipment)
The tomography apparatus of the present invention includes tomography means, virtual image data generation means, and communication means, and may further include other members appropriately selected as necessary. In addition, examples of the tomographic apparatus of the present invention include a CT scanning apparatus and an MRI (Magnetic Resonance Imaging) apparatus.

<Tomographic means>
There is no restriction | limiting in particular as said tomography means, According to the objective, it can select suitably, For example, X-ray, MRI etc. are mentioned.

<Virtual image data generation means>
The virtual image data generation means is not particularly limited as long as it can generate virtual image data of a three-dimensional part in the patient's body based on the tomographic image data acquired by the tomography means, depending on the purpose. It can be selected appropriately.

<Communication means>
The communication means is not particularly limited as long as it can communicate with a connected device or the like, and can be appropriately selected according to the purpose.

Hereinafter, the identification marker of the present invention and a medical support system using augmented reality technology using the identification marker will be described.
The identification marker includes the identification image, the identification display component, and the base. The base body is provided with the sewing holes for sewing to a site where a patient is operated. After the base body is sewn to the part by the sewing hole, the identification display component on which the identification image is displayed is fixed to the base body. That is, since the identification image is fixed to the substrate that is sewn directly to the part, when the identification image is captured by the imaging unit, the position where the identification image is recognized, that is, in the captured image. The virtual image of the three-dimensional part in the patient's body is superimposed and displayed by the display means at the display position of the part.

In addition to the identification marker, the medical support system includes the imaging unit, the display unit, the generation unit, the data storage unit, and the extraction unit.
First, the generation unit generates identification image data of the identification image associated with each virtual image data of a three-dimensional part in the body of the photographed patient. Next, the data storage unit stores the virtual image data and the identification image data generated by the generation unit in advance. At the time of surgery, the imaging means images a predetermined part of the patient to which the identification marker is sewn and fixed, and the display means identifies the identification marker that was imaged by the imaging means. Display captured images including images. Then, the extraction means recognizes the identification image based on the captured image data of the captured image including the identification image, and extracts the identification image data. The medical support system compares the identification image data extracted by the extraction unit with the identification image data stored in the data storage unit, and the identification image data extracted by the extraction unit is stored in the data storage unit. When it is determined that it matches any one of the identification image data, a virtual image of a three-dimensional part in the patient's body based on the virtual image data corresponding to the identification image data of the matching data storage unit is identified. At the position where the identification image of the marker is recognized, it is superimposed on the captured image and displayed by the display means.

(Example)
Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the embodiment. Note that the reference numerals such as “1” in the drawings mean the same thing.

FIG. 1A is a schematic perspective view showing an example of an identification marker in the present invention. FIG. 1B is a schematic cross-sectional view of the identification marker shown in FIG. 1A.
1A and 1B, the identification marker 1 includes an identification image 11, an identification display component 12, and a base body 20.

The identification marker 1 is sewn and fixed to a predetermined part of the patient during surgery, and fixes the identification image 11.
When the identification marker 1 is imaged by the imaging unit 111 described later, a virtual image of a three-dimensional part in the body of the patient previously captured by the tomography apparatus 120 described later is added to the captured image captured by the imaging unit 111. Is used when using the AR technology in which the identification image 11 in the captured image is recognized and superimposed on the captured image.

  The identification image 11 is an AR identification marker for displaying the corresponding virtual image on the display unit 112 when the identification image 11 is captured by the imaging unit 111.

  The identification display component 12 has the identification image 11 displayed on the surface thereof, and is fixed by being fitted into the frame 22 of the base body 20 as a whole. Further, the surface of the identification display component 12 has at least one concave portion that diffuses and reflects light so that the identification image 11 is not erroneously recognized by regular reflection of light. For example, the surface of the identification display component 12 is fine by sandblasting. A recess is provided.

The base body 20 is sewn and fixed to a predetermined part of the patient during the operation. The base body 20 is provided with four sewing holes 21 through which a suture thread 101, which will be described later, passes when being sewn and fixed. The base body 20 fixed with a suture 101 by a sewing hole 21 to a predetermined part of a patient is fitted with the entire identification display component 12 by a frame 22 provided on the surface, and the identification displayed on the identification display component 12 is displayed. The image 11 is fixed to a predetermined part of the patient. In this embodiment, the predetermined part of the patient is the liver.
Moreover, although the description has been made that the base frame fixes the entire identification display component, there is no particular limitation, and it can be appropriately selected according to the purpose. For example, as shown in FIGS. 2A and 2B it can.

FIG. 2A is a schematic perspective view showing another example of the identification marker in the present invention. FIG. 2B is a schematic cross-sectional view of the identification marker shown in FIG. 2A.
2A and 2B, the identification marker 2 includes an identification image 11, an identification display component 12, and a base body 20.
The identification image 11 is the same as the identification image 11 shown in FIG.

  Similar to the identification display component 12 shown in FIG. 1A, the identification display component 12 has the identification image 11 displayed on the front surface, and at least one protrusion 13 is provided on the back surface as shown in FIG. 2B. The portion 13 is fixed by being fitted into a fitting hole 23 provided in the base body 20.

The base body 20 is provided with four sewing holes 21, which are sewn and fixed to a predetermined part of the patient through the sewing holes 21. The base body 20 is provided with one fitting hole 23. The protrusion 13 is fitted into the fitting hole 23 to fix the identification display component 12.
In addition, although the description has been given that the base is fitted and fixed to the whole identification display component or the protrusion, there is no particular limitation, and it can be appropriately selected according to the purpose, for example, it can be detachably fixed. . Further, the description has been made that the protrusion is fitted into the fitting hole and the identification display component is fixed. However, there is no particular limitation, and it can be appropriately selected according to the purpose. For example, it is not a through-hole. It can be a groove. Furthermore, in FIGS. 1A and 1B and FIGS. 2A and 2B, the entire or part of the identification display component on which the identification image is displayed is fitted and fixed, but there is no particular limitation. For example, as shown in FIGS. 3A and 3B, the base and the identification display component can be integrated.
Further, the identification display component 12 and the base body 20 are made of a biocompatible material and can be further sterilized. There is no restriction | limiting in particular as a biocompatible material, According to the objective, it can select suitably, For example, polyether ether ketone (PEEK), modified polyphenylene ether, etc. are mentioned.

FIG. 3A is a schematic perspective view showing another example of the identification marker in the present invention. FIG. 3B is a schematic cross-sectional view of the identification marker shown in FIG. 3A.
3A and 3B, the identification marker 3 has an identification image 11 and a sewing hole 21.
The identification image 11 and the sewing hole 21 are the same as the identification image 11 and the sewing hole 21 shown in FIG.
The identification mark body 3 has an identification image 11 displayed on the surface thereof, further provided with a sewing hole 21, and is sewn and fixed to a predetermined part of the patient through the sewing hole 21.

Next, an example of a procedure for fixing the identification marker 1 shown in FIGS. 1A and 1B to the liver as a predetermined part of the patient will be described with reference to FIGS. 4A to 7B.
4A, FIG. 5A, FIG. 6A, and FIG. 7A are schematic cross-sectional views showing a procedure for fixing the identification marker in the present invention to a predetermined part of a patient. 4B, FIG. 5B, FIG. 6B, and FIG. 7B are explanatory views when the identification markers shown in FIG. 4A, FIG. 5A, FIG. 6A, and FIG.
4A and 4B, the two sutures 101 are passed through the liver 301 using two suture needles 102 each having the suture 101 locked to one end.
5A and 5B, the base body 20 is placed on the upper surface of the liver 301 in a state where both ends of the two sutures 101 are passed through the four sewing holes 21 of the base body 20.
6A and 6B, both ends of the two sutures 101 are tied, and the base body 20 is fixed to the liver 301.
7A and 7B, the identification marker 1 is fixed to the liver 301 by fitting and fixing the identification display component 12 to the base body 20 fixed to the liver 301.
In addition, when removing the identification marker 1 from the liver 301, it is as shown in FIG.

FIG. 8 is a schematic cross-sectional view showing a procedure for removing the identification marker in the present invention from a predetermined part of a patient.
In FIG. 8, the identification marker 1 can be removed from the liver 301 by lifting and floating one end of the identification marker 1 and cutting the suture 101 with a scalpel or the like from the direction indicated by arrow A in the figure.

Next, a medical support system based on augmented reality technology using an identification marker will be described.
FIG. 9 is an explanatory diagram showing a medical support system based on augmented reality technology using the identification marker in the present invention.
In FIG. 9, the medical support system 100 includes an identification marker 1, an input / output terminal device 110, a tomography apparatus 120, and a network 130.
Since the identification marker 1 is the same as that shown in FIGS. 1A and 1B, description thereof is omitted.

  The input / output terminal device 110 is, for example, a tablet-type terminal, which has a digital video camera function and a display function, and is a terminal capable of displaying a photographed image of a predetermined part of a patient on a liquid crystal display screen. is there.

  The tomography apparatus 120 is, for example, a CT scan apparatus, an MRI apparatus, or the like, acquires tomographic image data of a predetermined part of the patient, and based on the acquired tomographic image data, a virtual image of a three-dimensional part in the patient's body. Data is generated.

  The network 130 is not particularly limited as long as the input / output terminal device 110 and the tomography apparatus 120 are communicably connected to each other, and can be appropriately selected according to the purpose. Examples thereof include the Internet.

FIG. 10 is a block diagram showing a medical support system based on augmented reality technology using an identification marker in the present invention.
In FIG. 10, the medical support system 100 includes an identification marker 1, an input / output terminal device 110, a tomography apparatus 120, and a network 130.

  The input / output terminal device 110 includes an imaging unit 111, a display unit 112, a generation unit 113, a data storage unit 114, an extraction unit 115, a terminal communication unit 116, and a control unit 117.

  The photographing means 111 is, for example, a digital video camera using a CMOS image sensor, a CCD image sensor, or a tablet type terminal having a digital video camera function, and the identification marker 1 is sewn and fixed during surgery. A predetermined part of the patient is photographed.

  The display unit 112 is, for example, a liquid crystal display, a liquid crystal display mounted on a digital video camera, a tablet-type terminal, or the like, a projector, and the like, and displays a photographed image photographed by the photographing unit 111.

  The generation unit 113 generates identification image data of an identification image associated with each virtual image data of a three-dimensional part in the body of a patient imaged in advance.

  The data storage unit 114 is, for example, a storage medium such as a RAM (Random Access Memory), and stores virtual image data transmitted by the tomography apparatus 120 and identification image data generated by the generation unit 113.

  The extraction unit 115 recognizes the identification image of the identification marker based on the captured image data of the captured image, and extracts identification image data including information on the display, position, orientation, and size of the recognized identification image. Thereby, the medical support system 100 can display a virtual image of a three-dimensional part in the patient's body of the type, position, posture, and size according to the identification image data of the extraction unit 115 on the display unit 112.

  The terminal communication unit 116 is communicably connected to a later-described communication unit 123 of the tomography apparatus 120 via the network 130, and receives virtual image data by the tomography apparatus 120.

  The control means 117 is, for example, a CPU (Central Processing Unit), and is not particularly limited as long as it controls each means based on software stored in a storage medium or the like, and is appropriately selected according to the purpose. can do.

The input / output terminal device 110 configured as described above compares the identification image data extracted by the extraction unit 115 with the identification image data stored in the data storage unit 114, and the identification image data extracted by the extraction unit 115 is If it is determined that it matches any one of the identification image data in the data storage unit 114, a virtual image of a predetermined part of the patient based on the virtual image data corresponding to the identification image data in the data storage unit 114 that matches the identification image 1 The identification image 11 is superimposed on the photographed image at the recognized position.
Although the input / output terminal device has explained that the virtual image of the predetermined part of the patient is displayed on the captured image at the position where the identification image of the identification marker is recognized, there is no particular limitation. Depending on the purpose, it can be selected as appropriate. For example, it may be performed by a control means.

  The tomography apparatus 120 is, for example, a CT scan apparatus, and includes a tomography means 121, a virtual image data generation means 122, and a communication means 123.

  The tomography unit 121 is, for example, X-ray or MRI, and acquires tomographic image data of the patient's body.

  The virtual image data generation unit 122 generates virtual image data of a three-dimensional part in the patient's body based on the tomographic image data acquired by the tomography unit 121.

  The communication unit 123 is communicably connected to the terminal communication unit 116 of the input / output terminal device 110 via the network 130, and stores the virtual image data generated by the virtual image data generation unit 122 in the data storage unit 114. To send.

Next, the processing flow of the tomography apparatus and the input / output terminal apparatus will be described.
The process of generating and transmitting virtual image data performed by the tomographic apparatus 120 is represented by S in the flowchart of the flowchart showing the flow of the process of generating and transmitting virtual image data by the tomographic apparatus of the present invention in FIG. And will be described with reference to FIG.
In FIG. 11, the tomography means 121 acquires tomographic image data of the patient's body (S11). The virtual image data generation means 122 generates virtual image data of a three-dimensional part in the patient's body based on the tomographic image data (S12). The communication unit 123 transmits the virtual image data generated by the virtual image data generation unit 122 to the input / output terminal device 110 (S13), and the process ends.

Next, the process of generating identification image data based on the virtual image data performed by the input / output terminal device 110 and storing it together with the virtual image data is performed by the input / output terminal device according to the present invention shown in FIG. Description will be made with reference to FIG. 10 and FIG. 14A according to the step represented by S in the flowchart showing the flow of processing to be stored together with the virtual image data. Moreover, FIG. 14A is explanatory drawing which shows the screen at the time of displaying only the virtual image in this invention.
In FIG. 12, the terminal communication means 116 receives the virtual image data of the three-dimensional part in the patient's body from the tomography apparatus 120 (S21). The generation unit 113 generates identification image data associated with each virtual image data (S22). The data storage means 114 stores the virtual image data and the identification image data (S23), and this process is terminated.
Note that, when the liver virtual image 401 is displayed on the display unit 112 based on the virtual image data stored in the data storage unit 114 in S23, not only the virtual image of the liver itself but also the outer shape of the liver itself is displayed. A virtual image of the blood vessels of the liver can also be displayed.

Next, a process of generating identification image data based on virtual image data performed by the input / output terminal device 110 and storing it together with the virtual image data is performed by the input / output terminal device of the present invention shown in FIG. The process will be described with reference to FIGS. 10, 14B, and 14C in accordance with the step represented by S in the flowchart showing the flow of processing to be displayed.
Moreover, FIG. 14B is explanatory drawing which shows the screen which displayed the picked-up image at the time of image | photographing the identification marker body and liver in this invention. FIG. 14C is an explanatory diagram showing a screen on which a virtual image of the liver is superimposed and displayed when the identification marker and the liver are photographed in the present invention.
In FIG. 13, the imaging means 111 images the patient's liver 301 (S31). At this time, if the extraction unit 115 determines that the identification image 11 is not included in the photographed image obtained by photographing the liver 301, the process proceeds to S31. As shown in FIG. 14B, when the extraction unit 115 recognizes and determines that the identification image 11 is included in the captured image obtained by capturing the liver 301, the process proceeds to S33 (S32). If it is determined that the identification image 11 is included in the captured image, the extraction unit 115 extracts the identification image 11 (S33). It is determined whether or not the identification image 11 extracted by the input / output terminal device 110 matches the stored identification image. If it is determined to match, the process proceeds to S35, and if it does not match, the process proceeds to S31 ( S34). If it is determined that the extraction means 115 match, the input / output terminal device 110 displays the liver virtual image superimposed on the captured image of the liver 301 as shown in FIG. 14C (S35), and the process ends.

  The identification marker body of the present invention and the medical support system using the identification marker body are suitable by fixing the identification marker body of the present invention to a predetermined part of a patient in a medical action such as laparotomy for a human body or the like. Can be used.

DESCRIPTION OF SYMBOLS 1 Identification marker body 11 Identification image 12 Identification display component 13 Protrusion part 20 Base body 21 Sewing hole 22 Framework 23 Fitting hole 100 Medical support system 110 Input / output terminal device 111 Imaging means 112 Display means 113 Generation means 114 Data storage means 115 Extraction Means 116 Terminal communication means 117 Control means 120 Tomography apparatus 121 Tomography means 122 Virtual image data generation means 123 Communication means 130 Network 201 Suture 202 Suture needle 301 Liver 401 Virtual image of liver

Claims (7)

When photographed by the photographing means, an identification sign for superimposing the photographed image photographed by the photographing means and the corresponding virtual image to be displayed by the display means,
A base body provided with a sewing hole and sewn and fixed to a predetermined part of a patient;
And an identification image for displaying the corresponding virtual image by the display means when captured by the imaging means. Having an identification display part on which an identification image is displayed;
The identification display component has at least one protrusion,
The identification marker body according to claim 1, wherein the base body has a fitting hole or groove for fitting and fixing the protruding portion.   3. The identification marker according to claim 1, wherein the substrate and the identification display component are made of a biocompatible material and can be sterilized.   The identification marker according to any one of claims 1 to 3, wherein the identification image is an AR identification marker.   The identification marker body according to any one of claims 1 to 4, wherein the surface of the identification display component on which the identification image is displayed has at least one concave portion that diffuses and reflects light. Imaging means for photographing a predetermined part of a patient to which the identification marker body according to any one of claims 1 to 5 is sewn and fixed during surgery;
Display means for displaying a photographed image including an identification image in the identification marker, photographed by the photographing means;
Generating means for generating identification image data of an identification image associated with each virtual image data of a stereoscopic part in the body of the patient imaged in advance;
Data storage means for storing the virtual image data and the identification image data generated by the generation means;
Extracting means for recognizing the identification image of the identification marker and extracting the identification image data based on the captured image data of the captured image including the identification image;
The display means compares the identification image data extracted by the extraction means with the identification image data stored in the data storage means, and the identification image data extracted by the extraction means is the data storage means If it is determined that it matches any one of the identification image data, a virtual image of the part of the patient based on the virtual image data corresponding to the identification image data of the matching data storage means is used as the identification image of the identification marker. A medical support system, wherein the image is superimposed and displayed on the captured image at a position where the image is recognized. Tomography means for obtaining tomographic image data of a patient's part;
Virtual image data generation means for generating virtual image data of a three-dimensional part in the patient's body based on the tomographic image data acquired by the tomography means;
The medical support system according to claim 6, further comprising a tomography apparatus including: a communication unit that transmits the virtual image data generated by the virtual image data generation unit to store the virtual image data in the data storage unit.