JP6095032B1 - Composite image presentation system - Google Patents

Abstract

The present invention is an optical marker-based method that complements data based on the current camera image captured by a single camera, and enables easy and accurate three-dimensional position information in the entire coordinate system of the optical marker and HMD in real space. To detect. An optical tracking means for detecting local position information in a local coordinate system of an optical marker arranged at a predetermined site in real space using a camera attached to an HMD. In addition, the optical marker 24 and the HMD 11 have coordinate conversion means 32 for obtaining global position information in the entire coordinate system. The coordinate conversion unit 32 uses at least one of a camera image simultaneously captured by the cameras 18 of the plurality of doctors D and a camera image captured in the past, and localizes the plurality of optical markers 24 captured in each camera image. After specifying the relative positional relationship between all the optical markers 24 from the position information, global position information for each optical marker 24 is obtained. [Selection] Figure 1

Description

  The present invention relates to a composite image presentation system for generating a composite image obtained by superimposing a superposed image acquired elsewhere on a part of a real space in the outside world and allowing a user to visually recognize the composite image.

  In recent years, applications of various technologies such as virtual reality (VR) and augmented reality (AR) have been studied. As an application of AR technology to the medical field, a video see-through type head-mounted display that allows a user to visually superimpose a predetermined superimposed image on a camera image obtained by capturing a real space in the outside world is used to provide medical education support and minimal invasiveness. Systems that support surgery are being studied. As the system, for example, a doctor wearing a head-mounted display is used when performing puncture or the like, and a camera image of a doctor's field of view captured by a video camera attached to the head-mounted display is used for an ultrasonic diagnostic apparatus. There is a system in which a doctor visually recognizes an ultrasound image of an affected area of a patient to which a probe is applied (see, for example, Non-Patent Document 1).

  In the system of Non-Patent Document 1, an ultrasonic image is superimposed on a camera image according to the position and orientation of the probe. In order to acquire the three-dimensional position information of the probe required at this time, an optical marker is attached to the probe, and the optical marker is imaged by the video camera, so that the video camera and the optical are detected from the state of the optical marker in the camera image. A method for obtaining the relative positional relationship between markers is proposed in Non-Patent Document 1.

  When the optical marker is attached only to the probe as in the method of Non-Patent Document 1, only the relative positional relationship between one optical marker and the video camera is known, and the entire coordinates with the fixed point in real space as the origin The three-dimensional position information of the optical marker in the system cannot be grasped. Therefore, the beam is scanned by moving the probe along a predetermined range of the affected area of the patient, the three-dimensional position information of the probe at the time of the beam scanning is measured every predetermined time, and ultrasonic image data at each time and the probe at that time When there is a need to acquire volume data corresponding to the three-dimensional position information, the method of Non-Patent Document 1 performs beam scanning while fixing the position and posture of the video camera mounted on the head of the doctor. Therefore, it is difficult to obtain volume data with a certain accuracy.

  Therefore, in Non-Patent Document 2, a magnetic marker is attached to each of the head mounted display and the probe, a change in magnetic field from a magnetic base station that generates a magnetic field of various strengths is detected by the magnetic marker, and the magnetic base station is used as a reference. A method for detecting the three-dimensional position information of the head mounted display and the probe using the coordinate system as a whole coordinate system has been proposed.

  Further, in Non-Patent Document 3, a plurality of optical markers are further fixedly arranged around the patient, and the fixedly arranged optical markers and the optical markers provided on the probe are compared with the technique of Non-Patent Document 1. A method for detecting the three-dimensional position information of each optical marker in the global coordinate system has been proposed.

Shun'ichi Tano, Keisuke Suzuki, Kenji Miki, Natsuko Watanabe, Mitsuru Iwata, Tomohiro Hashiyama, Junko Ichino, Ken Nakayama, Simple Augmented Reality System for 3D Ultrasonic Image by See-through HMD and Single Camera and Marker Combination, Proceedings of the IEEE -EMBS, International Conference on Biomedical and Health Informatics (BHI 2012), 2012.1, pp.464-467 Naoshi Kosugi, Shunichi Tano, Tomonori Hashiyama, Kenji Miki, Mitsuru Iwata, Classification of Medical AR Systems in Ultrasound Diagnosis, Fuzzy System Symposium 2015, 2015.9, pp.588-593 Natsuko Watanabe, Shunichi Tano, Tomonori Hashiyama, Junko Ichino, Kenji Miki, Mitsuru Iwata, Proposal of an ultrasonic device interface that can be flexibly adapted to the operating room environment and the operator, Fuzzy System Symposium 2012, 2012.9, pp.649- 654

  However, in the system using magnetism as described in Non-Patent Document 2, it is very weak to surrounding metal, and when used in an environment containing many metal parts, detection error becomes large and frequent calibration is required. In addition, the error increases as the distance between the magnetic base station and the magnetic marker increases. Further, according to the method of Non-Patent Document 3, at least one of the optical markers fixedly arranged in addition to the optical marker provided on the probe moving in the real space with the doctor's own camera wearing the head mounted display. If one is not always imaged, the three-dimensional position of the probe in the global coordinate system cannot be specified, and in this case, the superimposed image cannot be superimposed on the camera image. Therefore, in order to avoid such a situation, no matter where the probe is moved, a large number of optical markers are captured so that at least one of the optical markers fixedly arranged is captured in the camera image where the probe is captured. It is necessary to place the marker in the real space. In addition, in order to identify the relative positional relationship between the optical markers that are fixedly arranged, it is necessary to perform a prior work for imaging all the optical markers that are fixedly arranged with one camera worn by a doctor. It is.

  The present invention has been devised by paying attention to such a problem. The object of the present invention is to supplement data based on the current camera image captured by one camera, mainly using an optical marker, and To provide a composite image presentation system capable of reliably generating a composite image in a desired state by easily and accurately detecting three-dimensional position information in an entire coordinate system of an optical marker and a head mounted display existing in space. is there.

  In order to achieve the above object, the present invention mainly provides a superimposed image acquired separately on a head mounted display attached to a user and attached with a camera that captures an image of real space existing in the user's field of view. In a composite image presentation system that presents a composite image to be visually recognized by a user in a state of being superimposed on a partial area of real space, a position information detection device that detects three-dimensional position information of a predetermined part in the real space, and the position information A processing device that generates the composite image based on the three-dimensional position information detected by the detection device and presents the composite image on the head-mounted display. The position information detection device uses a camera image captured by the camera. And optical tracking means for detecting three-dimensional position information at the predetermined part, and the optical tracking means includes a plurality of optical tracking means. By recognizing the image state of each optical marker imaged in the camera image and each optical marker installed at each of the predetermined portions, local coordinates with the camera fixed point as the origin for each of these optical markers An optical marker detection unit that detects local position information that is three-dimensional position information in the system, and the processing device determines a fixed point in the real space for the predetermined part and the head mounted display from the local position information. Coordinate conversion means for obtaining global position information which is three-dimensional position information in the global coordinate system as the origin, and composite image generation means for generating the composite image based on the global position information, in the coordinate conversion means, Using a plurality of the camera images having different imaging positions and / or imaging timings with the camera, After specifying the relative positional relationship between all the optical markers from the local position information of the plurality of optical markers captured in each camera image, at least one optical marker and the origin of the global coordinate system The global position information for each optical marker is obtained based on the relative positional relationship.

  In the present specification and claims, the “three-dimensional position information” is information obtained by adding a tilt, which is a rotation angle around the three orthogonal axes, to the three-dimensional position in the three orthogonal axes in a predetermined coordinate system. Means. The “coordinate component of the three-dimensional position information” means six components of a three-dimensional position (x, y, z) and a rotation angle (α, β, γ).

  According to the present invention, data based on a camera image captured by one camera is complemented using past camera images and camera images captured by other users, and optical markers and head mounts that exist in real space. The global position information of the display can be detected easily and accurately. Accordingly, the superimposed image can be accurately superimposed on a predetermined position of the camera image in accordance with the position and orientation of the optical marker in the camera image captured by the user. Also, the global position information can be detected mainly by the optical marker, and as a preliminary preparation, the calibration of the origin position in the sensing system using magnetism is not necessary, and the user can set the optical marker fixedly arranged in advance. The system can be used simply by imaging with a camera. Furthermore, the recognition result of the optical marker captured in the camera image of another user can be shared although it is not captured in the camera image of one user, and all the optical markers are necessarily captured by the camera of one user. It is possible to shorten the pre-work and reduce the number of fixed optical markers as much as possible, while reliably detecting global position information of optical markers and head mounted displays in real space. be able to.

  Further, when a composite marker in which optical markers are fixedly arranged on a plurality of surfaces of a polyhedron is used, the optical marker can be imaged with a camera from a wider range of angles, and the recognition accuracy of the optical marker can be improved.

  Furthermore, according to the configuration in which the marker position calculation unit calculates the global position information of the optical marker in consideration of the error characteristic related to the position measurement error in the position information detection device, the position measurement by the optical marker is performed by the non-optical marker. Instead of simply substituting for position measurement, it is appropriate to use these markers comprehensively and make a superiority or inferiority judgment using the probability distribution for each coordinate component of 3D position information. The user data of the position can be used effectively.

It is a conceptual diagram showing the main structures of AR display system containing the composite image presentation system of this invention. It is a block diagram showing the whole structure of the AR display system. It is a figure for demonstrating the error information memorize | stored in the error database. It is a flowchart for demonstrating the process sequence in a marker position calculation part. It is a conceptual diagram in the 1st example for demonstrating the process in a marker position calculation part. It is a conceptual diagram in the 2nd example for demonstrating the process in a marker position calculation part. It is a conceptual diagram in the 3rd example for demonstrating the process in a marker position calculation part. It is a graph showing the relationship between the standard deviation in each path | route and the last estimated value. It is a schematic perspective view of a stereo optical marker.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a conceptual diagram showing the main configuration of an AR display system including the composite image presentation system of the present invention, and FIG. 2 is a block diagram showing the overall configuration of the AR display system. In these drawings, the AR display system 10 is applied in a medical field, and a plurality of (n) doctors D (D1, D2,..., Dn) who are users are head mounted displays 11 (hereinafter referred to as “HMD11”). And a composite image obtained by superimposing a corresponding diagnostic image such as an ultrasound image as a superimposed image on an image (mainly affected area) of the patient S existing in the field of view of each doctor D. This is a display system.

  This AR display system 10 detects the HMD 11 worn on the head of each doctor D, the image diagnostic device 13 for acquiring a diagnostic image of the affected part of the patient P, and the three-dimensional position information of a predetermined part in the real space. And a processing device 16 that generates the composite image based on the detection result of the position information detection device 14 and presents the composite image to the HMD 11. The position information detection device 14 and the processing device 16 constitute a composite image presentation system that presents the composite image to the HMD 11.

  The HMD 11 is a video see-through type equipped with a video camera 18 that captures a real space camera image as a background of the composite image, and the composite image obtained by superimposing the superimposed image on the camera image is a wearer. It is displayed in front of. In the present embodiment, as the HMD 11, a compound eye type that presents the composite image to the left and right eyes of the wearer is used.

  The video camera 18 is arranged so that each doctor D wearing the HMD 11 can acquire almost the same camera image as if he / she actually viewed the real space in front of himself / herself. Therefore, each doctor D can visually recognize the patient S existing in front of him or the state of the affected part through the camera image displayed on the display unit of the HMD 11. In the present invention, as long as the camera image can be acquired, other various cameras can be employed.

  As the diagnostic imaging apparatus 13, a two-dimensional ultrasonic diagnostic apparatus is applied. In this two-dimensional ultrasonic diagnostic apparatus, an ultrasonic image such as a two-dimensional image which is a tomographic image on the same cross section as the beam scanning surface is generated by beam scanning with the probe 19 (see FIG. 1). Further, a beam scan is performed in a predetermined range of the affected part, and a measurement result to be described later of the three-dimensional position information in the same coordinate system of the probe 19 at the time of the beam scan is used. A function that enables generation of a stereoscopic image is provided. The medical information image such as an ultrasonic image and a three-dimensional image can be visually recognized by the doctor D through the HMD 11 in a state of being three-dimensionally superimposed on a predetermined position of the camera image in the real space as a superimposed image. Note that the diagnostic imaging apparatus 13 is not limited to the ultrasonic diagnostic apparatus of the present embodiment, and other diagnostic imaging apparatuses such as CT and MRI can be applied.

  As shown in FIG. 2, the position information detection device 14 uses a video camera 18 to detect three-dimensional position information in a predetermined part of real space, and does not use the video camera 18. And non-optical tracking means 22 for detecting three-dimensional position information at a predetermined site in the real space.

  The optical tracking means 21 recognizes the optical marker 24 installed at each of a plurality of predetermined sites in the real space and the image state of each optical marker 24 captured in the camera image, so that the optical marker at each site is installed. 24 includes an optical tracking system including an optical marker detection unit 25 that detects three-dimensional position information in a three-axis orthogonal coordinate system with the center of the video camera 18 as the origin. That is, the optical tracking means 21 can detect the three-dimensional position information of the installation site through the optical marker 24.

  The optical markers 24 are fixedly arranged at a plurality of locations on the body surface of the patient S lying on his / her back. Although not particularly limited, in the present embodiment, as shown in FIG. 1, they are fixedly arranged at four locations around the affected area of the patient S. In addition, a predetermined pattern is attached to the surface of the optical marker 24, and a plurality of patterns of optical markers 24 having different patterns are prepared, and the optical markers 24 having different patterns are used for each arrangement site.

  When the optical marker 24 is captured in the camera image captured by the video camera 18, the optical marker detection unit 25 recognizes the image of the optical marker 24 and performs the following process using a known method. That is, here, since the pattern, shape, and size of the optical marker 24 are known, an ID that indicates where the optical marker 24 is located is specified from the pattern of the optical marker 24 in the camera image. The position and orientation of the optical marker 24 are determined from the size and shape of the optical marker 24 in the camera image. Here, the position and orientation of the optical marker 24 are acquired as three-dimensional position information by the HMD coordinate system H with the origin of the center of the video camera 18 that images the optical marker 24 as the local coordinate system. The unit of the three-dimensional position information obtained using the optical marker 24 is a position (mm) and an angle (degree).

  Therefore, the optical marker detection unit 25 uses the HMD coordinate system H set for each HMD 11 of each doctor D for the optical marker 24 captured by the video camera 18 of the HMD 11 attached to each doctor D. Will be obtained sequentially. In this HMD coordinate system H, when there are n doctors D, there are n (H1, H2,..., Hn) which are the same as the number of HMD11, and the movement of the head of the doctor D to which the HMD11 is attached It is treated as a local coordinate system that moves with it.

  The non-optical tracking means 22 includes a magnetic base station 27 that generates magnetic fields of various strengths in various directions in real space, and a magnetic marker that functions as a sensor that detects the magnitude of magnetism from the magnetic base station 27. 28 (non-optical marker) and a magnetic tracking system 29 configured to detect the three-dimensional position information of the installation site from the magnetic detection state of the magnetic marker 28. That is, the magnetic tracking means 22 can detect the three-dimensional position information of the installation site through the magnetic marker 28. The unit of the three-dimensional position information obtained using the magnetic marker 28 is a position (mm) and an angle (degree).

  Although not particularly limited, in the present embodiment, the magnetic marker 28 is fixed to a predetermined portion of the probe 19 that moves sequentially, and is relatively moved to two of the optical markers 24 that are fixedly arranged. It is mounted so as not to be capable of relative rotation. The integrated marker of the optical marker 24 and the magnetic marker 28 functions as a composite marker 30 (see FIG. 1) composed of a plurality of markers, and the three-dimensional position information is obtained by both the optical marker detection unit 25 and the magnetic marker detection unit 29. Can be acquired.

  The magnetic marker detection unit 29 measures the direction and strength of the magnetic field generated from the magnetic base station 27, so that the magnetic marker 28 attached to the probe 19 or the optical marker 24 for each ID of the magnetic marker 28. The three-dimensional position information in the orthogonal three-axis magnetic coordinate system M with the predetermined part of the magnetic base station 27 as the origin is sequentially acquired. This magnetic coordinate system M exists at one place, and since the magnetic base station 27 is fixed, it is handled as a local coordinate system that is fixed so that it cannot move and rotate.

  As the non-optical tracking means 22, by tracking the non-optical marker without using the video camera 18, the three-dimensional position information in the local coordinate system with the reference point set in advance as the origin for the non-optical marker. As long as it is configured by a three-dimensional position sensor that identifies

  Although not shown in the drawings, as the other non-optical tracking means 22, for example, an ultrasonic tracking system using ultrasonic waves is adopted, and the magnetic tracking system can be used instead of or in combination with the magnetic tracking system. In this ultrasonic tracking system, a transmitter that generates ultrasonic waves of various frequencies and strengths is fixed as an ultrasonic marker (non-optical marker) at a predetermined site in real space, and a plurality of ultrasonic tracking systems installed at an ultrasonic receiving station are used. By measuring the frequency and intensity of ultrasonic waves with a microphone, using the principle of triangulation, etc., the ultrasonic coordinate of the three orthogonal axes with the origin at the predetermined point of the ultrasonic receiving station for the ultrasonic marker installation site The three-dimensional position information in the system is acquired sequentially. Similar to the magnetic coordinate system M, this ultrasonic coordinate system is also handled as one local coordinate system fixed by a fixed arrangement of ultrasonic base stations. Even when this ultrasonic tracking system is employed, the processing described later is performed based on the acquired three-dimensional position information in the same manner as the three-dimensional position information acquired by the magnetic tracking system.

  In the following description, the three-dimensional position information in the HMD coordinate system H and the magnetic coordinate system M, which are local coordinate systems, is appropriately referred to as “local position information”. In addition, the three-dimensional position information in the patient coordinate system P, which is an entire coordinate system of three orthogonal axes with a fixed point fixed in a preset real space as the origin, is appropriately referred to as “global position information”.

  The processing device 16 is configured by a computer including an arithmetic processing device such as a CPU and a storage device such as a memory and a hard disk, and a program for causing the computer to function as the following means and units is installed.

  As shown in FIG. 2, the processing device 16 includes an image capturing means 31 for capturing various image data obtained by the image diagnostic device 13 and the video camera 18, and an optical marker detected by the position information detecting device 14. 24 and the coordinate conversion means 32 which converts the local position information of the magnetic marker 28 into the global position information, and the composite image generation means 33 which generates the composite image presented to the HMD 11 of each doctor D based on the global position information. ing.

  The coordinate conversion unit 32 stores in advance information used when performing various processes, and a data recording unit 35 that records local position information of each optical marker 24 and magnetic marker 28 detected by the position information detection device 14. From the database 36, the marker position calculation unit 37 for calculating the global position information of each optical marker 24 and magnetic marker 28, and the global position information of each optical marker 24 and magnetic marker 28 calculated by the marker position calculation unit 37, And an HMD position calculation unit 38 for obtaining global position information of each HMD 11.

  The data recording unit 35, for the optical marker 24 imaged by the video camera 18 of each HMD 11, the local position information of the optical marker 24, that is, the position (x, y, z) in the HMD coordinate system H and their coordinate axes. The rotation angles (α, β, γ) are recorded for each HMD 11. For example, when a certain optical marker 24 is captured by the video cameras 18 of n doctors D1 to Dn, local position information in different n HMD coordinate systems H to H1 to Hn is recorded for the optical marker 24. Will be. Further, as the local position information of the magnetic marker 28, the position (x, y, z) in the magnetic coordinate system M and the rotation angles (α, β, γ) around these coordinate axes are recorded. The local position information of the optical marker 24 and the magnetic marker 28 is pooled at a constant time interval (for example, about several tens of mm / sec), and recorded as a local position data set for one cycle which is a set of data in a lump. . The local position data set is stored for each cycle, and the data recording unit 35 stores the current local position data set and the past local position data set that is the previous cycle. Further, in the data recording unit 35, information regarding the presence / absence of movement of each optical marker 24 and magnetic marker 28 is registered in advance and recorded as a set with the information. In the example of FIG. 1, only the magnetic marker 28 attached to the probe 19 is registered as “movable”, and the other optical markers 24 and the magnetic marker 28 are registered as “movable”. The registration contents can be changed sequentially according to the situation.

  The database 36 includes an error database 42 in which information on position measurement errors of the optical tracking unit 21 and the non-optical tracking unit 22 is stored, and a composite marker 30 in which the optical marker 24 and the magnetic marker 28 are integrated. , 28 and a mutual positional relationship database 43 in which the relative positional relationships are stored.

  As shown in FIG. 3, the error database 42 includes three-dimensional position information (position (x, y, z) in each coordinate system for each of the position detection method using the optical tracking system and the position detection method using the magnetic tracking system. ), Orientation (α, β, γ)) for each of a large number of patterns in which six components are arbitrarily combined, the standard deviation (± Δx, ± Δy, ± Δz, ± Δα, ± Δβ) of errors expected for each component , ± Δγ) is stored.

  In the present embodiment, the mutual positional relationship database 43 is attached with the magnetic marker 28 for the optical marker 24 represented by “A” in FIG. 1 and the optical marker 24 represented by “D” in FIG. Therefore, for each of these two composite markers 30, the IDs of the optical marker 24 and the magnetic marker 28 constituting the two and the relative positions (x, y, z) between the markers 24, 28 are provided. ) And relative rotation angles (α, β, γ) are stored.

  The marker position calculation unit 37 performs processing for converting each local position information of each optical marker 24 and each magnetic marker 28 into global position information. The processing purpose here is as follows.

  The three-dimensional position information of the markers 24 and 28 detected by the position information detection device 14 is not a unified coordinate system, but the relative positional relationship between the HMD 11 and the optical marker 24, the magnetic base station 27, and the magnetic field. It only represents the relative positional relationship with the marker 28. Finally, the synthesized image generation means 33 displays the medical information image corresponding to the position and orientation of the probe 19 on the site of the probe 19 in the camera image that is the background image displayed on the HMD 11 of each doctor D. A process of generating a composite image by superimposing it as a superimposed image is performed. In order to perform this image generation processing, the three-dimensional position information of all the optical markers 24 and magnetic markers 28 is represented by a unified patient coordinate system P, and the result is used to move with the operation of each doctor D. It is necessary to grasp the three-dimensional position information of each HMD 11 with the unified patient coordinate system P. Therefore, the marker position calculation unit 37 does not necessarily capture all the optical markers 24 with the video camera 18 of one doctor D, and does not necessarily capture the imaging results or data recording with the video camera 18 of the other doctor D. The past local position data sets recorded by the unit 35 are used complementarily, and global position information of the optical marker 24 and the magnetic marker 28 is obtained.

  A specific processing procedure in the marker position calculation unit 37 will be described below using first to third cases shown in FIGS. 5 to 7 along the flowchart of FIG.

  First, the current local position data set is acquired from the data recording unit 35 (step S101). Next, a camera image in which a plurality of optical markers 24 are captured in one image is searched from the acquired local position data set, and each optical marker 24 captured in the same camera image has a relative position. All combinations with known relationships are extracted (step S102).

  Here, as a first example, the optical marker 24 and the magnetic marker 28 are arranged as shown in FIG. 1, and the camera images acquired by the video camera 18 of the first doctor D1 include “A” and “B” in FIG. The optical marker 24 represented by (2) is simultaneously imaged, and in the camera image acquired by the video camera 18 of the second doctor D2, the optical markers 24 represented by “C” and “D” in the same figure are simultaneously imaged. And In the first case, the optical markers 24 of “A” and “B” in the HMD coordinate system H1 in the HMD 11 of the first doctor D1 are extracted as a combination whose relative positional relationship is known, and the second person The optical markers 24 of “C” and “D” in the HMD coordinate system H2 in the HMD 11 of the doctor D2 are extracted as a combination whose relative positional relationship is known. In FIG. 5, the optical markers 24 constituting the same combination in the first example are shown surrounded by the same broken line frame. In addition, if the camera image separately acquired by the video camera 18 of the doctor D2 also includes the optical markers “A” and “B”, “A” and “B” in the HMD coordinate system H2 are used. "Is further extracted as a combination whose relative positional relationship is known. That is, even when the optical markers 24 of the same combination are captured by the video cameras 18 of different HMDs 11, they are extracted as different types of combinations.

  Next, it is determined whether or not all the optical markers 24 are connected only by each combination of the extracted optical markers 24 (step S103). In the first case, there are a combination of optical markers 24 of “A” and “B” and a combination of optical markers 24 of “C” and “D”, and there is an optical marker 24 common to these sets. Therefore, it is determined here that all the optical markers 24 are not connected between the combinations. On the other hand, in the case of the second case of FIG. 6, the “B” and “C” optical markers 24 are simultaneously captured in the camera image acquired by the video camera 18 of the third doctor D3 in addition to the first case. , The optical marker 24 of “B” is imaged by the video camera 18 of the first doctor D1, and the optical marker 24 of “C” is imaged by the video camera 18 of the second doctor D2, It is determined that all the optical markers 24 are connected.

  When it is determined that all the optical markers 24 are not connected as in the first case, the connection state between the combinations of the optical markers 24 is ensured using the detection result of the non-optical tracking means 22. It is confirmed whether or not it can be performed (step S104). That is, here, when the composite marker 30 in which the magnetic marker 28 is provided integrally with the optical marker 24 is further used from the current local position data set, whether or not the connection state of the optical marker 24 can be secured. Is confirmed. In the case of the first case of FIG. 5, since the optical markers 24 of “A” and “D” are composite markers 30, the optical marker 24 of “A” and the optical marker 24 attached thereto are added according to the information in the mutual positional relationship database 43. The relative positional relationship between the magnetic marker 28 and the relative positional relationship between the “D” optical marker 24 and the magnetic marker 28 attached thereto are known. Therefore, from the detection result of the three-dimensional position information of each magnetic marker 28 in the same magnetic coordinate system M, the relative positional relationship between the optical markers 24 of “A” and “D” is found through the relative positional relationship of each magnetic marker 28. To do. As a result, it is determined that the optical markers 24 of “A” and “D” are connected through the non-optical tracking means 22 and the connected state of each optical marker 24 is secured, as virtually indicated by a one-dot chain line in FIG. (Step S105).

  If the connected state of each optical marker 24 cannot be secured even after the above processing, the past local position data set one cycle before is extracted from the data recording unit 35 and added to the current local position data set (step S107), the above-described processing is repeated. If the connection state of the combination of the optical markers 24 is still not secured, the local position data set in the previous cycle is added to the previous local position data set (step S107), and the above processing is repeated. Is called. If the connected state of each optical marker 24 is not ensured even when referring to a past local position data set having a preset number of cycles (step S106), it is determined that the coordinate conversion process is impossible (step S108). The composite image generation means 33 performs processing described later.

  On the other hand, when the connection state is secured for all the optical markers 24, another connection state of the optical marker 24 is added using the detection result of the non-optical tracking means 22 (step S109). For example, in the third case of FIG. 7 in which the optical marker 24 of “B” is further combined with the composite marker 30 with respect to the first case, the optical markers 24 of “A” and “B” are optical tracking means 21. In addition to the detection result, the connection state can be secured by the detection result of the non-optical tracking means 22 by the magnetic markers 28 integrated with each other, and the connection state is the connection state based on the detection result of the optical tracking means 21. Added to others.

  Then, global position information, which is three-dimensional position information of all optical markers 24 in the patient coordinate system P, is calculated for all paths with error information (step S110). That is, when the connection state of all the optical markers 24 is secured, the relative positional relationship between at least one optical marker 24 and the origin of the patient coordinate system P is set in advance, so that all the optical markers 24 Global location information can be obtained. At this time, coordinate calculation in a plurality of paths is possible depending on how the optical markers 24 are linked. Here, for all the optical markers 24, global position information is obtained for each of all paths, and error characteristic information from the error database 42 is used to detect the position of each path by the optical tracking system used in the path calculation. Error data by the position detection method by the method and magnetic tracking system is integrated.

  This processing will be described in detail below using the third case of FIG. In the third case, it is assumed that the origin O of the patient coordinate system P is set substantially at the center of the optical marker 24 of “A”. The origin of the patient coordinate system P may be any fixed point in real space where the relative positional relationship with the optical marker 24 or the magnetic marker 28 that does not move is unchanged. For example, the origin of the patient coordinate system P can be set at a point where the distances from the optical markers 28 that do not move are equal.

  For example, when obtaining the global position information of the optical marker 24 of “C”, the optical marker 24 of “A” that is the origin of the patient coordinate system P, and the optical marker 24 of “B” and “D” are respectively Since it is the composite marker 30, the following first to third paths exist.

  As a first path, the relative positional relationship between the optical markers 24 of “A” and “D” based on the detection result of the local position information of the magnetic marker 28 in the non-optical tracking means 22, and the local of the optical marker 24 in the optical tracking means 21. The global position information of the optical marker 24 of “C” is obtained from the relative positional relationship between the optical markers 24 of “C” and “D” based on the detection result of the position information.

  As the second path, the relative positional relationship between the optical markers 24 of “A” and “B” based on the detection result of the local position information of the optical marker 24, and “B” and “ The global position of the optical marker 24 of “C” is determined by the relative positional relationship of the optical marker 24 of “D” and the relative positional relationship of the optical marker 24 of “C” and “D” based on the detection result of the local position information of the optical marker 24. Location information is required.

  As the third path, the detection result of the magnetic marker 28 instead of the detection result of the optical marker 24 is used for the relative positional relationship of the optical markers 24 of “A” and “B” with respect to the second path. The global position information of the optical marker 24 is obtained.

  At this time, error information regarding a position measurement error due to the characteristics of the optical tracking system or the magnetic tracking system is added to each path. In other words, the optical tracking system has a characteristic that the detection accuracy of the position in the line of sight depth direction and the detection accuracy of the rotation angle in that direction is worse than the detection accuracy of the position and rotation angle in the other two axes. There are characteristics such as weakness and unstable angle. Therefore, the standard deviation, which is information on the position measurement error for each tracking system due to these characteristics, is extracted from the error database 42 that is databased in units of measurement distance and rotation angle, and the erroneous standard deviation is integrated for each path. The

  That is, here, for each optical marker 24, each coordinate component of the global position information, that is, the position (x, y, z) and the rotation angle (α, β, γ) for each path when the global position information is obtained. For each, standard deviation information of the error accumulated in each procedure in the path is added.

For example, for the optical marker “C”, the global position information (x ₁ ± Δx ₁ , y ₁ ± Δy ₁ , z ₁ ± Δz ₁ , α ₁ ±) to which the error information is added for the first path described above. Δα ₁ , β ₁ ± Δβ ₁ , γ ₁ ± Δγ ₁ ) are obtained. Similarly, for the second route, global position information (x ₂ ± Δx ₂ , y ₂ ± Δy ₂ , z ₂ ± Δz ₂ , α ₂ ± Δα ₂ , β ₂ ± Δβ ₂ , to which error information is added, is added. γ ₂ ± Δγ ₂ ) is obtained. For the third route, global position information (x ₃ ± Δx ₃ , y ₃ ± Δy ₃ , z ₃ ± Δz ₃ , α ₃ ± Δα ₃ , β ₄ ± Δβ ₃ , γ with error information added is added. ₃ ± Δγ ₃ )

Next, for each optical marker 24, for each coordinate component of the global position information, a normal distribution with the calculated value of the global position information as an average is calculated for each path from the standard deviation accumulated for each path. When the normal distributions for each route are superimposed, the value having the highest appearance probability in consideration of all the routes is determined as the final estimated value of the global position information that seems to have the smallest error (step S111). For example, in the above-described example, with respect to the x coordinate, as shown in FIG. 8, x _d that is the maximum value of the sum of the probability distributions of the first to third routes is set as the final estimated value. This process is performed for each optical marker 24, and for each optical marker 24, global position information after adjustment in consideration of a position measurement error by the optical tracking system or the magnetic tracking system is determined.

  From the global position information of each optical marker 24 thus obtained, the global position information of the single magnetic marker 28 attached to the probe 19 is obtained through the composite marker 30 whose relative positional relation can be specified by the mutual positional relation database 43. (Step S112).

  In the HMD position detection unit 38, the relative position of each HMD 11 and each optical marker 24 obtained by the optical tracking means 21 from the global position information of each optical marker 24 and magnetic marker 28 finally determined by the marker position calculation unit 37. Based on the relationship, the global position information of each HMD 11 is specified.

  In the composite image generating unit 33, the superimposed image captured by the image capturing unit 31 is three-dimensionally superimposed on the doctor D in the vicinity of the tip position of the probe 19 in the camera image captured by the image capturing unit 31. A visible composite image is generated. The composite image is generated for each camera image of each HMD 11 based on the global position information of each HMD 11 and probe 19 specified by the coordinate conversion means 32. For this reason, each doctor D becomes visible in a state where a three-dimensional medical information image or the like is superimposed on the distal end portion of the probe 19 existing in his / her field of view in a direction corresponding to the posture of the probe 19.

  When the coordinate conversion unit 32 determines that the global position information of each optical marker 24 and each magnetic marker 28 cannot be acquired, the composite image generation unit 33 causes the tip of the probe 19 captured by each HMD 11 to be acquired. A composite image is generated so that some information to be displayed specified in advance is displayed at the position.

  As shown in FIG. 9, a three-dimensional optical marker 45 in which a plurality of optical markers 24 are attached to each surface excluding the installation surface of the polyhedron can be employed as a composite marker. With this stereoscopic optical marker 45, even if the video camera 18 captures images from various directions, any one of the optical markers 24 can be imaged in a plane state, and the horizontal optical marker 24 cannot recognize the horizontal direction. Recognition from the direction is also possible. For example, if camera shooting at an angle within 45 degrees is necessary, the stereoscopic optical marker 45 may be a hexahedron or more. Thus, as the number of surfaces to which the optical marker 24 is attached increases, it becomes easier to image any one of the optical markers 24 in a form close to the front, regardless of the position of the stereoscopic optical marker 45. The detection accuracy of the used optical tracking means 21 can be increased. Also in this stereoscopic optical marker 45, the relative positional relationship between the optical markers 24 constituting the stereoscopic optical marker 45 is stored in the mutual positional relationship database 43 and used for the above-described processing. Further, a composite marker in which a non-optical marker such as the magnetic marker 28 is further attached to the stereoscopic optical marker 45 can be adopted.

  Moreover, about the coordinate conversion means 32 of the said embodiment, the global position information of the optical marker 24 is adjusted using a long-distance parallax image from the camera image imaged with the video camera 18 of each doctor D's HMD11. An adjustment part can be further provided. That is, in the system of the present invention, a plurality of doctors D wears the HMD 11, and the ID of the optical marker 24 imaged on each HMD 11 is found from the camera image captured by the video camera 18 of each HMD 11, The position on the camera image is also obtained. For this reason, when a certain one optical marker 24 appears in common in the camera images of the plurality of HMDs 11, the position of the optical marker 24 can be specified from the parallax image. Therefore, the adjustment unit selects a set of camera images in which the position of the HMD 11 is farthest from among a plurality of camera images captured by the same optical marker 24, and uses the parallax images in these camera images. And the process which improves the precision of the global position information of each optical marker 24 calculated by the marker position calculation part 37 is performed. In this process, the database 36 is further provided with a parallax database in which error data of binocular parallax due to the camera image is stored, and the distance from the information from the parallax database to the optical marker 24 specified by the parallax image. The global position information of each optical marker 24 obtained by the marker position calculation unit 37 is adjusted so as to match. Then, after the adjustment, as in the processing in the embodiment, global position information of the magnetic marker 28 and each HMD 11 is obtained, and the composite image is generated. According to this, a parallax image between the HMDs 11 at a distance can be used, and the global position information of each optical marker 24 can be detected with high accuracy.

  Furthermore, in the present embodiment, the application example by a plurality of users has been illustrated and described, but the present invention is not limited to this and can be applied to a single user. When applied to a single user in this way, a past local position data set is obtained based on a plurality of lines of sight of the single user, thereby enabling complementary data processing. Further, even when applied to a single user, a non-optical marker can be used in combination as in the above embodiment, and the error database 36 can be used. Furthermore, when using the above-mentioned distant parallax images, the same processing can be performed by using camera images obtained at different times.

  In addition, the position measurement mainly of the optical marker in the present invention is compatible with the video see-through type HMD 11, and by this combination, the display error of the superimposed image of the HMD 11 can be made almost zero. However, the present invention is not limited to this, and if a camera is provided, an optical see-through type or a single-application type can be applied as the HMD.

  Moreover, although the case where the AR display system 10 is used in the medical field has been illustrated and described in the above embodiment, the present invention is not limited to this and can be applied to other uses with the same configuration.

  In addition, the configuration of each part of the apparatus in the present invention is not limited to the illustrated configuration example, and various modifications are possible as long as substantially the same operation is achieved.

DESCRIPTION OF SYMBOLS 10 AR display system 11 Head mounted display 14 Position information detection apparatus 16 Processing apparatus 18 Video camera 21 Optical tracking means 22 Non-optical tracking means 24 Optical marker 25 Optical marker detection part 30 Composite marker 32 Image conversion means 33 Composite image generation means 45 Three-dimensional Optical marker (composite marker)
D Doctor (user)
H HMD coordinate system (local coordinate system)
M Magnetic coordinate system (local coordinate system)
P Patient coordinate system (overall coordinate system)

Claims (9)

Is attached to the user of the medical field, to a head mounted display camera for capturing an image of the real space is attached to exist in the field of view of the user, in a part region of real space diagnostic images obtained separately as superimposed image In a medical composite image presentation system that presents a composite image to be visually recognized by a user in a superimposed state,
A position information detecting device for detecting three-dimensional position information of a predetermined part in the real space, and generating the composite image based on the three-dimensional position information detected by the position information detecting device and presenting it on the head mounted display And a processing device for
The position information detecting device includes an optical tracking unit that detects three-dimensional position information in the predetermined part using a camera image captured by the camera,
The optical tracking means recognizes the optical markers respectively installed at the plurality of the predetermined sites and the image states of the optical markers captured in the camera image, so that each of the optical markers An optical marker detection unit that detects local position information that is three-dimensional position information in a local coordinate system with a fixed point as an origin;
The processing device includes coordinate conversion means for obtaining global position information, which is three-dimensional position information in a global coordinate system with a fixed point in the real space as an origin, for the predetermined part and the head mounted display from the local position information. And a composite image generating means for generating the composite image based on the global position information,
In the coordinate conversion means, a plurality of the camera images having different imaging positions and / or imaging timings with the camera are used, and from the local position information of the plurality of optical markers captured in each camera image, After specifying the relative positional relationship between the optical markers, based on the relative positional relationship between at least one of the optical markers and the origin of the global coordinate system, the global position information in each of the optical markers is obtained , and the camera is When only one camera is used, the current camera image captured by the camera and the camera image captured in the past are used. When a plurality of cameras are used, the current camera image captured simultaneously by each camera is used. A composite image presentation system using at least one of the camera image and the camera image captured in the past . The position information detection device further includes non-optical tracking means for detecting three-dimensional position information at a predetermined site in the real space without using the camera,
The non-optical tracking unit tracks a non-optical marker fixed to at least one of the predetermined parts, thereby obtaining a three-dimensional position in the local coordinate system with a preset reference point as an origin for the non-optical marker. It is composed of a three-dimensional position sensor that identifies local position information that is information,
The coordinates conversion means, the non also utilize local location of the optical markers, the synthetic image presentation system according to claim 1 Symbol placement and the characteristic of obtaining the global position information of each optical marker. A composite marker in which the optical marker and the non-optical marker are integrally provided so as not to be relatively movable and relatively unrotatable is installed at the predetermined portion, and the composite marker is also derived from the global position information by the coordinate conversion means. The composite image presenting system according to claim 2 , wherein the composite image presenting system is used. A composite marker in which the optical marker is fixedly arranged on a plurality of faces of a polyhedron is installed at the predetermined portion, and the composite marker is also used for deriving the global position information by the coordinate conversion means. claim 1 Symbol placement synthetic image presentation system. In the coordinate conversion means, whether or not there is a possibility of movement is stored in advance for each of the optical markers, and among the camera images captured in the past, only the camera image in which the optical marker that cannot be moved is captured is used. The composite image presentation system according to claim 1, wherein: The composite image generation unit generates a composite image including preset display information when the global position information cannot be acquired by the coordinate conversion unit and the superimposed image cannot be overlaid. claim 1 Symbol placement synthetic image presentation system. The coordinate conversion unit calculates the global position information of the optical marker and the non-optical marker in consideration of an error database in which information on position measurement errors in the position information detection apparatus is stored and the error characteristics. A marker position calculator,
In the marker position calculation unit, when the global position information is obtained from the local position information for each optical marker, the position measurement that can occur for each of a plurality of paths derived based on the relative positional relationship of the optical markers The composite image presentation system according to claim 2 , wherein an error is taken into account. In the error database, for each of the position detection method by the optical tracking means and the position detection method by the non-optical tracking means, a standard deviation of error is stored for each of various values of the three-dimensional position information,
In the marker position calculation unit, for each of the paths, for each coordinate component of the three-dimensional position information, a numerical range in consideration of the standard deviation is specified, and the value having the highest probability is obtained by combining the paths. The composite image presentation system according to claim 7 , wherein the composite image presentation system is a final estimated value of global position information. The coordinate conversion unit selects a set of the camera images in which the position of the head mounted display is farthest from a plurality of the camera images captured by the same optical marker, and each of the camera images based on said binocular parallax of the optical markers, claim 1 Symbol placement synthetic image presentation system and adjusting the global positional information in the optical markers that are imaged.

Images