JP7519354B2 - Using Optical Code in Augmented Reality Displays - Google Patents

Description

Mixed reality or augmented reality is an area of computing technology that may combine images from the physical and virtual computing worlds into a mixed reality world. In mixed reality, people, places, and objects from the physical and virtual worlds become a composite environment. Mixed reality experiences may be delivered through existing commercial or custom software along with the use of VR (virtual reality) or AR (augmented reality) headsets.

Augmented reality (AR) is an example of mixed reality in which a live direct or indirect view of a physical, real-world environment is augmented or supplemented by computer-generated sensory input, such as audio, video, graphics, or other data. Augmentation is performed in which real-world locations are viewed and associated with environmental elements. With the aid of advanced AR techniques (e.g., the addition of computer vision and object recognition), information about the real world around the user becomes interactive and may be digitally modified.

A problem faced by AR systems or AR headsets is identifying the position and orientation of objects with high accuracy. Similarly, aligning the position of virtual elements with a live view of the real-world environment can be difficult. The alignment resolution of an AR headset may align virtual objects to the physical objects being viewed, but the alignment resolution may only be to within a few centimeters. While providing alignment to within a few centimeters may be useful for entertainment and less demanding applications, increasing the positioning and alignment resolution of AR systems may be desirable in the fields of science, engineering, and medicine. As a result, the positioning and alignment process may be performed manually, which can be time-consuming, cumbersome, and inaccurate.

FIG. 1 illustrates an exemplary augmented reality (AR) environment in which image data of the medical implement and the patient can be referenced and aligned to an actual view of the patient using one or more optical codes attached to the patient and medical implement.

FIG. 1 shows an example of an optical code having image visible markers affixed to a patient.

1 shows an example of visual data that may be captured by a camera of a real view of a patient's body and a medical tool, each labeled with an optical code.

FIG. 1 illustrates an example of a view in an augmented reality (AR) display with annotations to guide positioning and orientation of a medical tool during a medical procedure.

1A-1C are diagrams illustrating examples of views that may be displayed on an augmented reality display device to display the use of medical tools or guidance information during a medical procedure.

An example is shown using an augmented reality (AR) display that allows positioning and orientation of perspective images, as well as image projection from an image dataset onto the human body using an optical code.

1 shows an example of a fluoroscopy device that is movable relative to a person's body, where the fluoroscopy image and image projection are also moved and/or modified.

A fluoroscopy device is shown that is movable relative to a person's body and generates fluoroscopic images and image projections within a coronal field of view for the fluoroscopy device to enable combined viewing through an augmented reality (AR) headset or AR display.

1 illustrates an ultrasound transducer used in combination with an optical code to enable combined display of an image dataset and an ultrasound image via an augmented reality (AR) headset or display.

1 is a flow chart of an exemplary method of using an augmented reality headset to reference a medical tool with respect to an image dataset and a patient's body.

We show a method of using an augmented reality (AR) display to align a person's body using a fluoroscopic image, an image projection from an image dataset, and an optical code.

FIG. 1 illustrates an exemplary system that may be used in using an optical code on a medical implement to enable the medical implement to be referenced to an image dataset and a patient's body.

FIG. 1 is a block diagram illustrating an example of a computing system for processing the present techniques.

Techniques are provided that allow an augmented reality (AR) headset to be used to identify one or more optical codes on a medical implement that is within the field of view of the AR headset's camera during a medical procedure. The medical implement can reference an image dataset that is aligned with the person's body using one or more optical codes and image visible markers on the person's body. The image dataset may be a previously acquired image of a portion of the person's body using a non-optical imaging modality (e.g., using MRI (magnetic resonance imaging), CT (computed tomography) scan, X-ray, etc.). The image dataset can be aligned with the person's body using image visible markers that are a fixed distance from at least one optical code located on the person's body. For example, both the image visible markers and the optical code (e.g., AprilTag or 2D optical barcode) may be attached to a piece of material (e.g., co-located with each other or fixed and close to each other) to facilitate alignment of the image dataset with the person's body. Image visible markers are markers that can be seen in a non-visible imaging modality, such as a captured radiographic image or image dataset, that may not be optically visible to the AR headset. The image dataset may be captured with a representation of the image visible markers using images captured by a machine that captures the structure of the human body in a non-optical imaging modality. The representation of the image visible markers in the image dataset can be aligned with the patient's body using a known fixed position of the image visible markers relative to one or more optical codes affixed to the person's body (as described in more detail below). For example, the image visible markers may be radiopaque markers, MRI beads, echogenic structures for ultrasound, etc.

Referencing the medical implement to the image dataset may also include identifying the position and orientation of the medical implement relative to the person's body and the image dataset. Thus, the medical professional views the virtual interior of the patient using the image dataset, while viewing the actual patient through the AR headset, and references the medical implement using an optical code on the medical implement. Visual image data including the medical implement may be captured using a visible light camera in the AR headset. One or more optical codes visibly displayed on the medical implement may also be scanned. For example, the position and orientation of the medical implement may be determined by scanning one or more optical codes (e.g., an APRIL code or a 2D (two-dimensional) bar code). The medical implement may be a medical instrument, a trocar, a catheter, an orthopedic hardware, a surgical implant, a clamp, an electrocautery blade or system, an operating room object, an equipment object, a treatment object, a medical procedure object, a treatment object, an insertable implement, an implantable object, a medical device, etc.

Using the AR headset, visual indicators, annotations, or virtual tools may be integrated into the image dataset to guide the positioning and orientation of the medical tool relative to the patient's body and the image dataset. The medical tool or virtual tool may also have a graphic indicator (e.g., a computer-generated graphic or symbol) that is displayed in proximity to or as an overlay on the medical tool or virtual tool using the AR headset to highlight the medical tool or virtual tool, or the graphic indicator may indicate whether the medical tool is associated with a medical procedure. Medical information, which is instructional information describing the use of the medical tool in a medical procedure, may also be obtained. Furthermore, the contour of the medical tool may be detected using one or more optical codes on the medical tool as a starting point. The use of automatic detection and alignment may avoid or reduce time-consuming and tedious manual alignment of the medical tool and image dataset with the actual view of the patient.

In another configuration of the present technology, an augmented reality (AR) headset or AR display can align and display fluoroscopic images and image projections from an image dataset with respect to the person's body. The position and orientation of the fluoroscopic images as image projections may be defined by the position and orientation of a movable fluoroscopic device with respect to the person's body. In this context, the description of an imaging device that is movable with respect to the person's body includes mobility of the imaging device that changes the orientation of the imaging device or moves the emitter, detector, transducer, and/or imaging components of the imaging device with respect to the person's body. One or more optical codes on the fluoroscopic device can be used to determine the position and orientation of the fluoroscopic device with respect to the person's or patient's body. The fluoroscopic images and image projections may be displayed with a fluoroscopic image of a portion of the person's body. This alignment and display can use one or more optical codes and image visible markers on the person's body and one or more optical codes on the fluoroscopic device. The optical codes on the person's body and the optical codes on the fluoroscopic device can be identified in visual image data captured by the camera of the AR headset.

At least one of the optical codes on the person's body may have a fixed position relative to the image visible markers. This allows the image data set (e.g., radiographic image) to be registered to the person's body when viewed through an AR display (e.g., an AR headset) using a fixed distance between the image visible markers and one or more optical codes on the person's body. The image projection may be created from the image data set based on the position and orientation of the fluoroscopy device. The fluoroscopic image from the fluoroscopy device may be registered with the person's body and the image projection based on the image visible markers (e.g., radiopaque objects) and/or one or more optical codes that define the position and orientation of the fluoroscopy device. Furthermore, the fluoroscopic image may be virtually displayed in the AR headset at a position relative to the person's body where the x-ray beam is passing through the person's body, and the fluoroscopic image may be registered to overlay a portion of the person's body being imaged with the x-ray beam. The image projection may be oriented parallel to the fluoroscopic image and may be displayed in the AR headset as being virtually in at least a portion of the path of the x-ray beam. The registered image may be displayed using an AR headset along with the real-world or actual view of the patient, or the registered image may be displayed on a separate AR display (e.g., a separate display screen). This process combines, aligns, and orients the live fluoroscopic image, the image dataset (e.g., augmented reality image or image projection), and the actual view of the person, thereby allowing useful aspects of the fluoroscopic image (e.g., navigation of radiopaque objects within the person's body) and the image dataset (e.g., better tissue contrast, etc.) to be combined during a medical procedure.

In another configuration using fluoroscopic images, one or more optical codes of the fluoroscopic device can be used to detect changes in the position and orientation of the fluoroscopic device relative to the person's body. The image projection and fluoroscopic image position and orientation may then be modified as defined by the changes in the position and orientation of the fluoroscopic device. For example, the movement of the projection of the image data set may be co-localized or synchronized to match the fluoroscopic image based on the changes in the orientation and position of the fluoroscopic device. The zoom of the fluoroscopic device may also be detected using a radiopaque object on the person's body, and the size of the radiopaque object may be used to adjust the size of the image projection when viewed on the AR display. Graphic indicators, virtual tools, or virtual target systems may also be included in the image data set and co-localized to the fluoroscopic image to guide the positioning and orientation of a fluoroscopic object (e.g., a trocar or needle) relative to the person's body and the image data set using the AR display.

In another configuration, an optical code detected or captured by a camera or AR headset can be used to verify that the correct medical procedure is being performed on the correct patient. Additionally, information related to the person or patient in the medical procedure may also be obtained using the optical code and displayed to the medical professional using the AR system. The optical code can also help verify the patient's identity. Verification may also be performed using one or more optical codes on the body and medical implement to determine that the correct part of the body and the correct medical implement are in the medical procedure.

1 illustrates an exemplary augmented reality (AR) environment 100 in which an image dataset of a patient 106 or person may be aligned with an actual view of the patient 106 using an optical code 200 affixed to the patient 106. The environment 100 may include a physical space 102 (e.g., an operating room, a laboratory, etc.), a user 104, a patient 106, a number of optical codes 200 on the patient, a medical tool 118 having an optical code 198, and an AR headset 108 in communication with a server 112 via a computer network 110. A virtual user interface 114 and a virtual cursor 122 are also shown in dashed lines to indicate that these virtual elements are generated by the AR headset 108 and are visible to the user 104 through the AR headset 108.

The AR headset 108 may be an AR computing system capable of augmenting a real view of the patient 106 with an image dataset. For example, the AR headset 108 may be used by the user 104 to augment a real view of the patient 106 with one or more 3D image dataset views or radiological images of the patient 106, including but not limited to bones 106b (shown in FIG. 1), muscles, organs, or fluids. The AR headset 108 may allow the image dataset (or a projection of the image dataset) to be dynamically reconstructed. Thus, as the user 104 moves around the patient 106, sensors in the AR headset 108 determine the position of the user 104 relative to the patient 106, and the patient's internal anatomy, displayed using the image dataset, may be dynamically reconstructed as the user selects different orientations relative to the patient. For example, the user 104 may walk around the patient 106. The AR headset 108 may then augment the actual view of the patient 106 with one or more acquired radiological images or image datasets (MRI, CT scan, etc.) of the patient 106, so that both the patient 106 and the image dataset of the patient 106 can be viewed by the user 104 from any angle (e.g., projected images or slices from the image dataset can also be displayed). The AR headset 108 may be a modified version of Microsoft HOLOLENS, Meta Company META 2, Epson MOVERIO, Garmin VARIA VISION, or other AR headset.

The optical code 200 may be affixed to the patient 106 prior to generation of image data (e.g., acquisition of an MRI, CT scan, X-ray, etc.) of the patient 106, and then remain affixed to the patient 106 while the patient 106 is viewed by the user 104 via the AR headset 108. The optical code 200 and image visible markers may then be used by the AR headset 108 to automatically align an image dataset of the patient 106 with an actual view of the patient 106. Furthermore, by using the same optical code 200 used during image data capture to automatically capture image data, it may be ensured that the image data captured by the AR headset 108 is consistent with the actual patient 106 being viewed through the AR headset 108.

The AR headset 108 has sensor technology that can map or detect the surface of the patient, and can also map the surface of the medical tool 118, and this detected surface mapping data can be co-registered to the image dataset. The medical tool 118 can be moved frequently within the environment 100, and the real-time position of the medical tool 118 can be tracked in the 3D space 102 using an optical code, and the medical tool 118 can be referenced to the image dataset 116 or the body of the patient 106. When the user 104 inserts a portion of the medical tool 118 into the body of the patient 106, the AR headset 108 can display a virtual insertion portion of the medical tool 118 projected onto the image dataset 116 to depict the medical tool 118 in the inner anatomy of the patient 106. In this way, even if the actual insertion portion of the medical tool 118 is hidden from the actual view of the user 104, the virtual insertion portion of the medical tool 118 can be projected into the actual view of the user 104 and referenced to the image dataset. The medical implement 118 can be tracked using one or more optical codes attached to the medical implement 118, which can then be detected by the AR headset to establish a continuously updated position of the medical implement 118 with reference to the image dataset 116 and the body of the person or patient 106. In some embodiments, the medical implement 118 can be anything that the user 104 wants to insert into the patient 106. For example, the medical implement 118 can include, but is not limited to, a needle, a trocar, a scalpel (as shown in FIG. 1), a scope, a drill, a probe, a clamp, an implant, or another medical instrument.

The virtual user interface 114 may be generated by the AR headset 108 and may include options for modifying the display of the projected internal anatomy of the patient 106 from the image data set 116 of the patient 106. The virtual user interface 114 may include other information that may be useful to the user 104. For example, the virtual user interface 114 may include information about the patient or medical implement 118 (e.g., medical instruments, implants, etc.) that are identified with an optical code. In another example, the virtual user interface 114 may include a medical chart or other medical data of the patient 106. In some configurations, the image data 116 or captured radiological data of the person may be displayed by the AR headset 108 using a volume of the image data 116 to display the radiologically captured anatomy of the patient 106 (e.g., bone 106b, tissue, vessels, fluid, etc.) from the image data. The image data may include axial, coronal, sagittal, or oblique slices of the image data. A slice may be a two-dimensional (2D) slice having a depth as well as a height and width (e.g., one or more layers of voxels), a three-dimensional (3D) slice, and/or a four-dimensional (4D) slice (a 3D image with a time series of images). The user 104 may control the virtual user interface 114 using hand gestures, voice commands, eye movements, a remote control (e.g., a finger clicker), a 3D mouse, a VR wand, finger sensors, haptic technology, or other control methods.

In one configuration, multiple users may be present at the same time, each wearing an AR headset 108, to view the patient 106 augmented with image data of the patient 106. For example, there may be multiple AR headsets 108 used during a medical procedure. One AR headset 108 may be used by a first medical professional to adjust and manipulate the radiological image displayed on both AR headsets, and a second headset may be used by a second medical professional to assist in performing the medical procedure on the patient. Additionally, one medical professional may be able to turn the radiological image on or off at the request of the other medical professional.

2 illustrates the optical code 200 of FIG. 1 secured to the patient 106 of FIG. 1. With reference to both FIG. 1 and FIG. 2, the optical code 200 may be perceptible to an optical sensor, such as an optical sensor built into the AR headset 108. In some embodiments, the optical code 200 may be an AprilTag, a linear barcode, a matrix two-dimensional (2D) barcode, a quick response (QR) code, or some combination thereof. AprilTag is a type of two-dimensional barcode that may be a visual reference system useful for augmented reality and camera calibration. AprilTags may be used to calculate the 3D position, orientation, and identity of the tag relative to a camera, sensor, or AR headset.

The optical code 200 may be linked to medical data of the patient 106 such that the medical data of the patient 106 may be accessed by the optical code 200. For example, the optical code 200 may be used to automatically retrieve image data sets used in a medical procedure of the patient using an AR system.

The optical code 200 may further be associated with a marker 206 perceptible to a non-optical imaging modality or an image visible marker. Examples of non-optical imaging modalities include, but are not limited to, an MRI modality, a computed tomography (CT) scanning modality, an X-ray modality, a positron emission tomography (PET) modality, an ultrasound modality, a fluorescence modality, an infrared thermography (IRT) modality, a 3D mammography, or a single photon emission computed tomography (SPECT) scanning modality. In another example, the non-optical image or image data set may be an image or image data set that includes a combination of two or more forms of non-optical imaging modalities listed above (e.g., two or more images combined with each other, combined segments of two or more non-optical images, a CT image fused with an MRI image, etc.). Each image data set in a separate modality can have an image visible code in the individual image data set, allowing PET images, CT images, MRI images, fluoroscopic images, etc. to be registered and referenced with the optical code on the person's body in the AR system view. By forming the markers 206 from a material perceptible to the non-optical imaging modality, the markers 206 or image visible markers can be displayed in the image data set of the patient 106 captured using the non-optical imaging modality. Examples of markers 206 include, but are not limited to, metal spheres, liquid spheres, radiopaque plastic, metal impregnated rubber, metal strips, paramagnetic materials, and pieces of metallic ink.

The markers 206 or image visible markers may be arranged in a pattern or may have a fixed position relative to the position of the optical code 200. For example, in the embodiment disclosed in FIG. 2, the optical code 200 may be printed on the material 202 (e.g., bandage, paper, plastic, metal foil, etc.) and the markers 206 may be affixed to the material 202 (e.g., embedded in the material 202 and not visible on any surface of the dressing). In this embodiment, the markers 206 may be arranged in a pattern having a fixed position relative to the position of the optical code 200 by being arranged in a fixed pattern within the dressing 202. Alternatively, the markers 206 may be embedded within the optical code 200 itself, for example, the markers 206 are embedded in the ink in which the optical code 200 is at least partially printed on the material 202, the ink comprising a material that is radiopaque and perceptible to non-optical imaging modalities, such as ink particles that are not transparent to x-rays. In these embodiments, the optical code 200 itself can function as both the optical code and the pattern of markers. Additionally, the markers 206 can be placed by (at least temporarily) attaching or printing the optical code 200 directly to the skin 106a of the patient 106. By placing the markers 206 in a pattern that has a fixed position relative to the position of the optical code 200, the fixed position can be used to calculate the position of the markers 206 or the pattern of image visible markers relative to the visible position of the optical code 200, or to image the visible markers, even if the markers 206 themselves are invisible or imperceptible to the sensors of the AR headset 108.

Once the optical code 200 and markers 206 are attached to the patient 106 in a fixed pattern, a non-optical imaging modality (in which the markers 206 are perceptible) can be used to capture image data of the patient 106 and the markers 206. In particular, the image data can include the internal anatomical tissue of the patient 106 (e.g., bones 106b, muscles, organs, or fluids, etc.) and the pattern of the markers 206 in a fixed position relative to the position of the internal anatomical tissue of the patient 106. In other words, not only does the internal anatomical tissue of the patient 106 appear in the image data of the patient 106, but the markers 206 also appear in the image data set of the patient 106 in a fixed pattern, and the position of this fixed pattern of the markers 206 appears in the image data set in a fixed position relative to the position of the internal anatomical tissue of the patient 106. In one example, when the non-optical imaging modality is a CT scan modality, the CT scan image can display the bones 106b, organs, and soft tissues of the patient 106, as well as the markers 206 positioned in fixed positions relative to the positions of the bones 106b, organs, and soft tissues of the patient 106.

Furthermore, the patient 106 may be moved, for example, from a medical imaging room at a hospital to an operating room at the hospital. The user 104 (e.g., a medical professional) can then use the AR headset 108 to determine the location of the optical code 200 on the person's or patient's body. The AR headset 108 can then automatically retrieve image data of the patient 106 based on the optical code.

After detecting the optical code 200 in the 3D space 102, the AR headset 108 can automatically calculate the position of the pattern of markers 206 in the 3D space 102 and relative to each other. This automatic calculation can be based on the sensed position of the optical code 200 in the 3D space 102, and can also be based on known fixed positions of the pattern of markers 206 relative to the position of the optical code 200. Even if the markers 206 are not perceptible to the AR headset 108 (e.g., due to the markers 206 being embedded in or under a material), the AR headset 108 can automatically calculate the position of the pattern of markers 206 based on the position of the optical code 200 and the fixed positions of the pattern of markers 206 relative to the position of the optical code 200. In this example, these fixed positions can allow the AR headset 108 to automatically calculate the position of the pattern of markers 206 in the 3D space 102 relative to each other, even if the AR headset 108 does not directly sense the positions of the markers 206.

After calculating the position of the markers 206 or the pattern of image visible markers in the 3D space 102, the AR headset 108 can register the position of the internal anatomy of the patient 106 in the 3D space 102 by aligning the calculated position of the pattern of markers 206 in the 3D space 102 with the position of the pattern of markers 206 in the image dataset. The registration can be performed based on the calculated position of the pattern of markers 206 in the 3D space 102 and the fixed position of the image dataset set for the markers 206 relative to the position of the internal anatomy of the patient 106. This registration can then enable the AR headset 108 to display the internal anatomy of the patient 106 in real time from the image data projected onto the actual view of the patient 106.

Thus, the optical code 200 and associated pattern of markers 206 may be used by the AR headset 108 to automatically align image data of the patient 106 with an actual view of the patient 106. Additionally, one or more optical codes 200 (e.g., AprilTag and 2D barcodes, or another combination of optical codes) may be used to automatically read out image data acquired during image data capture, ensuring that the image data read out by the AR headset 108 matches the actual patient 106 as viewed through the AR headset 108.

In a further example, multiple optical codes 200 can be attached to the patient 106 simultaneously to further ensure accurate alignment of the image data of the patient 106 with the actual view of the patient 106 in the 3D space 102. Also, the pattern of five markers 206 disclosed in FIG. 2 may be replaced with another pattern, such as a pattern of three markers or a pattern of seven markers, and each optical code may have a different pattern. Furthermore, because the markers 206 are fixed to the outer layer of the patient 106, the markers 206 do not all lie in one plane, but instead can conform to any curvature of the outer layer of the patient 106. In these embodiments, the fixed position of the pattern of markers 206 relative to the position of the optical code 200 may be established after fixing the optical code 200 and markers 206 to the patient 106 to account for any curvature on the outer layer of the patient 106.

FIG. 3 shows a visual image that may be captured by a camera of an AR headset or AR system. The visual image may include a patient 300's body and a medical implement 314, each with an optical code affixed to it. The optical code may be used to identify the location and orientation of the medical implement 312 in the visual image, and the optical code may be used as a starting point to identify the contour or shape of the medical implement 312. Additionally, the optical code 310 on the person 300's body may be used to access additional information about the patient (e.g., a patient record in a patient database) or the virtual object (e.g., in an object database) that may be used as an overlay or additional virtual object for display.

In one configuration, multiple optical codes 314 may be used on the medical implement 312 to determine the position and orientation of the medical implement. For example, optical codes 314 may be affixed to multiple separate faces or surfaces of the medical implement. Each of these multiple optical codes may be coded to identify a particular face, aspect, or orientation of the medical implement. The multiple optical codes may also be used in identifying a desired 3D (three dimensional) virtual image to associate with the medical implement. For example, a highlight graphic or virtual object (e.g., a virtual medical instrument) may be selected as an overlay for the medical implement.

When the optical codes 310, 314 attached to the patient 300 and medical implement 314 are used to identify the position and orientation of the patient's body and medical implement 314 in the visual image, this position and orientation information can be tracked for each optical code and used when determining the position and orientation of the medical implement and the patient in the procedure. The optical codes 310 can be captured during the process of capturing visual images of the patient during the procedure. As a result, the optical codes 310 can be used to reference the medical implement 314 against a previously captured radiological image of the patient in an augmented reality display (e.g., in an AR headset, etc.).

Additionally, the optical code 314 of the medical implement can be used to identify the particular type of medical implement 312 (e.g., a medical instrument or an orthopedic implant). Once the position and orientation of the medical implement is identified 314, the position of the medical implement can be determined relative to the image data set and the patient's body as described above. To better describe the position of the medical implement 314, the AR system can access information describing the size, shape, and contours of the medical implement, as well as any other relevant information.

In one example, additional information related to the medical tool 312 may also be obtained. For example, information regarding the use of the medical tool may be obtained and displayed in the AR headset. This may include information on how to best align the medical tool with the patient's body, tips for inserting implants into the bone, settings for electronic medical sensors, or other guidance information for the medical tool.

Additionally, the AR system may have records identifying which medical implements 312 are associated with a particular medical procedure. Using this information in combination with the optical code, the augmented reality system may determine whether a particular medical implement 312 is correctly associated with or used in a current medical procedure. For example, in a medical procedure in which a medical implement 312 is implanted in a patient, a data store associated with the augmented reality system may be accessed to ensure that the correct implant is implanted in the patient and the correct tool is used. This combination of optical codes, procedure data, and patient data may be used to check whether the correct patient is being operated on, whether the correct body part is being operated on, whether an implant for the correct side of the body is being used, or whether the correct implant size is being used. Using optical codes on the medical implements 312 prior to a procedure may increase confidence that the correct medical implements 312 are in the operating room. In another example, the AR system may display an optical indicator that confirms that a given medical implement 312 is authorized for the procedure or that alerts the user that an incorrect medical implement 312 or instrument is present.

4 illustrates identifying additional information related to the medical implement 312 during a medical procedure. As discussed above, the AR system can use the optical code 314 to access information related to the patient 300, the medical implement 312, and/or the medical procedure being performed. The AR system can display information related to the use of the medical implement via the AR headset to assist the user during the procedure. In one example, this information can describe the proper use of the medical implement. For example, if the AR system captures the optical code 314 on the medical implement 312, the value in the optical code 314 can be used as a lookup value to retrieve information for that device from a database. Such information can further include planned medical procedure steps for using the medical implement 312, which can be displayed in the AR headset.

The medical professional can pre-plan certain aspects of the medical procedure. For example, the location and orientation of the incision or the path to cut the tissue can be pre-planned using annotations. These plans can be entered into a database associated with the augmented reality system. If desired, the user can instruct the augmented reality system to display pre-planned medical procedure information relevant to the user in the AR headset. In this case, the user or system can have pre-defined annotations 400, 402 to describe the positioning of the medical instrument 612. A virtual guidance system can be displayed by the AR headset to indicate the pre-planned path or alignment points, lines or planes for the medical instrument for the medical procedure. As a result, the augmented reality display can display guidance annotations 404 with instructions to move the medical instrument 312 to the circular interpretations 400, 402. Once the medical professional moves the position and orientation of the medical instrument to the proper position and orientation (e.g., in three dimensions) visually depicted using the graphics in the AR headset, a graphic indicator, virtual tool, or virtual target system can be displayed to indicate that the proper position and orientation has been met. The positioning and orientation of the medical tool 312 may be guided in three or two dimensions. In one example, a red indicator may be displayed when the medical tool is not aligned and a green indicator may be displayed via the AR headset when the medical tool is aligned. Thus, the annotations may be modified, animated, and/or change color as the medical tool moves near a defined location, target, entry point, or target object.

Similarly, a virtual instrument or tool may be displayed to allow alignment of the visible instrument with the virtual instrument. For example, a graphic or virtual instrument may be displayed or projected using an AR headset. Once the visible instrument or real-world instrument is aligned with the virtual instrument or tool, the AR system may display a message or graphic indicating that the visible instrument is aligned with the virtual instrument. Furthermore, this alignment of the virtual instrument may also allow alignment with visible anatomical structures in a previously acquired image dataset or an image dataset acquired during a procedure (e.g., a CT scan or MRI images obtained during a procedure).

5 shows that the incision point 504 is displayed on the AR display as an overlay or virtual object on a particular part of the patient's body. The user can use this virtual incision point to guide the initial placement of the medical instrument relative to the body of the person 300 and/or the image dataset at the beginning of the procedure. Similarly, the ability to detect the position and orientation of the medical instrument can also include the ability to assist the medical professional by guiding the depth or angle of the incision, the proper placement of clamps, etc. Furthermore, the system can display instructions for the correct angle or orientation to hold a given medical instrument. For example, the AR display can present a graphic indicator, virtual tool, or virtual target system for the correct movement of the instrument in the current medical procedure based on the position and orientation of a portion of the patient's body. For example, the system can display a visual representation of a line along which an incision customized to the person's individual body is made. Similarly, the system can estimate the current depth of the instrument and indicate that the medical instrument is to be moved relative to the patient's body and/or the image dataset. During the procedure, the augmented reality system can use one or more optical codes (e.g., optical codes of either the patient 310 or the medical tool 312) to access, align, and display planning information within the AR display to guide the medical professional.

In the medical preparation configuration, the use of optical codes, as described in this disclosure, may enable a medical professional preparing a patient for a medical procedure to be graphically instructed to identify portions of the patient's body or anatomical tissue where skin should be prepped or other medical preparation should be performed prior to the medical procedure. For example, after the AR headset identifies one or more optical codes on the patient's body, graphic lines, shapes, or virtual tools may be displayed in the AR headset to assist the medical professional in performing the medical preparation. This may guide the medical professional to position, orient, and prepare the patient in the correct anatomical location, prepare the correct incision points, and/or prepare the correct portion of the patient's anatomical tissue. For example, it may be difficult to identify the correct vertebrae or hip location of a patient, and this technology provides such guidance. This guidance may improve the overall throughput of surgical technicians in a surgical suite.

6 illustrates a technique 602 for displaying fluoroscopic images 654 and image projections 656 from an image dataset aligned with respect to the body of a person 606a using an augmented reality (AR) system. A fluoroscopic device 660 can transmit an X-ray beam (e.g., continuous X-rays) through the patient to obtain a series of fluoroscopic images or live video of the fluoroscopic imaging of the patient that can be viewed by a medical professional. The fluoroscopic device 660 may also be movable with respect to the person's body. The fluoroscopic images 654 and image projections 656a can have positions and/or orientations defined by the fluoroscopic device 660 and/or image visible markers. A camera or optical sensor (e.g., visible light sensor or IR sensor) linked to the AR system or AR headset 608 can capture visual image data of a portion of the person's body on the operating table 603, and the fluoroscopic device 660 that is movable with respect to the person's body. The one or more optical codes on the body of the person 606a and the one or more optical codes on the vision devices 652, 658 can be identified and scanned from the visual image data using optical code recognition and scanning techniques or other optical code recognition techniques.

The image projection 656 may be selected to display the portion of the image data set associated with the fluoroscopic image 654 (e.g., parallel, oblique, or in another fixed orientation) that is captured. The image projection may also display a particular anatomical tissue type of the person's body, such as a vein, nerve, or bone. The fluoroscopic image 654 may be a monolayer projection (e.g., a two-dimensional (2D) projection). Registration, positioning, and orientation may be performed using the optical code 620 and image visible markers on the person's body, a representation of the image visible markers on the image projection, the image visible markers captured in the fluoroscopic image 654, and the optical codes 652, 658 on the fluoroscopic device (e.g., a C-arm apparatus, a catheterization lab, an angiography lab, or a fluoroscopic device with a movable emitter and detector).

At least one of the optical codes 620 on the person's body can have a fixed position relative to the image visible markers (as described in detail above). This allows the image data set (e.g. a captured radiological image) to be aligned with the body of the person 606 using the fixed positions of the image visible markers with reference to one or more optical codes on the person's body. The person's body may be covered with a cloth 607, but the body's internal anatomical structure 650 can be virtually viewed using the image data set.

Image projections 656 may be created from the image dataset based on the position and orientation of the fluoroscopy device 660. The image projections 656 may be coronal projections, oblique projections, or another projection orientation consistent with the orientation of the fluoroscopy device 660. The image projections are projected from the image dataset to define a single planar view or "slab" that is a multiplanar reconstruction (MPR) of the image dataset (e.g., multi-layer projections). The image projections may be of any selected thickness and may be MIPs (maximum intensity projections) of tissue in the appropriate views.

The position and orientation of the fluoroscopy device 660 relative to the body of the person 606a can be determined using one or more optical codes 652, 658 on the fluoroscopy device 660. The fluoroscopy image 654 from the fluoroscopy device 660 can be registered with the person's body and an image data set established based on the image visible markers and/or optical codes on the body of the person 606a. Additionally, the fluoroscopy image 654 can be positioned and oriented using the position and orientation of the fluoroscopy device 660 relative to the body of the person 606.

The radiopaque markers can be used as image visible markers to align the fluoroscopic images 654 with the body of the patient 606a. In some circumstances, the radiopaque markers can be the same image visible markers used to register the image datasets, or the radiopaque markers can be completely separate radiopaque markers (e.g., lead rectangles) that may have separate optical codes. For example, the radiopaque markers can be a first type of imaging modality marker used to register the fluoroscopic images 654 with the person's body, and a second imaging modality marker (e.g., an MRI-type marker or an ultrasound marker) can be used to register the image datasets with the person's body. The registered dataset images, image projections, and fluoroscopic images can also be displayed using the AR headset 608 or on a separate AR display 662 along with a real-world view of the patient. Thus, augmented reality images can be combined with a live fluoroscopic intervention or diagnostic procedure performed on the body of the person 606a.

As previously mentioned, multiple useful views can be provided to the medical professional using the fluoroscopy device and AR system. One view can include the ability to take a partially transparent fluoroscopic image and overlay the fluoroscopic image such that the fluoroscopic image is anatomically aligned on the actual view of the patient using an optical code. Another view can allow the image dataset to be composited with the fluoroscopic image and aligned with the patient's body using an optical code. Furthermore, the projections from the image dataset can be moved or reconstructed to match the medical professional's changing actual view of the body through the AR system or AR headset. Yet another view can be provided with an AR system that displays a combined view of the fluoroscopic image and a projection of the image dataset (e.g., a 2D rectangular slice) parallel to the fluoroscopic image overlaid in alignment with the patient (e.g., what the medical professional would see if he or she were in the same perspective as the x-ray beam itself). In this configuration, the projections can be moved or reconstructed as the fluoroscopy device moves.

The medical tool 618 having an optical code may also be referenced with respect to the image data set and image projection 656 as previously described. This allows a medical professional to view the medical tool 618 with simultaneous reference to the image data set or image projection 656 and the fluoroscopic image 654. The fluoroscopic image 654 may be set to any level of transparency desired by the medical professional or user.

7 illustrates that the position of the fluoroscopy device 654 may change relative to the person's or patient's body 606a to allow the medical professional 604 to acquire fluoroscopy images 660 from different perspectives. Changes in the position and orientation of the fluoroscopy device relative to the person's body may be detected and quantified using one or more optical codes 652, 658 captured by a camera associated with the AR headset 608 or AR system. Changes in the position and orientation of the fluoroscopy device 660 may alter the position and orientation of the image projection 656 and the fluoroscopy image 654. For example, the position and/or orientation of the image projection 656 may be moved based on a detected change in the position and orientation of the fluoroscopy device 660 relative to the patient's 606a's body as seen by the AR headset using one or more optical codes 652, 658. The image projection 656 from the image dataset may be reconstructed using the revised position and orientation of the fluoroscopy device 660, or a new projection may be created.

For example, the fluoroscopy device 660 may be rotated 45 degrees in one axis. As a result, the image projection 656 may be recreated in its new orientation, and the fluoroscopy image 654 may be displayed in a rotated orientation when viewed on the AR display to allow the orientation of the fluoroscopy image 654 and image projection 656 to be aligned in the proper orientation relative to the body of the person 606a when viewed through the AR headset. The fluoroscopy image 654 may have a modified orientation in 3D space relative to the person's body as defined by an optical code, image visible markers on the body, and/or as defined by a modified position and/or orientation of the fluoroscopy device 660. Thus, as the position and orientation of the x-ray beam changes, the position and orientation of the image projection 656 and the fluoroscopy image 654 change.

Determining the position and orientation of the fluoroscopy device 660 relative to the patient also allows the AR system to reconstruct the image projection 656 so that it is parallel to the fluoroscopic image 654 obtained from the fluoroscopy detector. Additionally, the fluoroscopic image 654 can be positioned relative to the body of the person 606a based on the location from the body where the fluoroscopic image 654 is being captured (e.g., using an x-ray beam). Thus, the fluoroscopic image 654, image projection 656, and the patient's body 606a may be aligned such that a medical professional can use the image projection 656 as an overlay on the fluoroscopic image 654 to view the person's or patient's anatomy. The positioning and orientation of the image projection 656 and the fluoroscopic image 654 can represent an AR (augmented reality) view based on the portion of the person's body through which the x-ray beam is passing (as opposed to the medical professional's point of view).

The AR system can reconstruct the 3D image dataset to provide projections from any angle consistent with the position and orientation of the fluoroscopy device 654. For example, if the fluoroscopy device 660 is positioned to capture a lateral view, a lateral projection of the image dataset can be provided along with a lateral fluoroscopy image view. By combining and registering the real-world view of the person's body, the image projections from the image dataset, and the fluoroscopy image, medical professionals can better see and navigate the anatomy within the patient's body. The 3D (three-dimensional) image dataset provides better soft tissue contrast (e.g., organs and blood vessels can be seen in the internal anatomy) and a 3D reference environment that fluoroscopy images cannot provide.

In one configuration, the fluoroscopy device 660 may be zoomed relative to the body of the person 606a while the fluoroscopy image 654 is being captured. As a result, adjustments to the image projection 656 may be made based on changes in size of the image visible markers (i.e., radiopaque markers 830) captured in the fluoroscopy image 654 to allow the image projection 656 to match the zoom of the fluoroscopy image 654. For example, if an image visible marker of known size is captured when the fluoroscopy image 654 is zoomed in (e.g., a lead marker of a predetermined size or length), the amount that the image projection is zoomed in may be determined by a visual increase (or decrease in the case of a zoom out) in the size of the image visible marker in the fluoroscopy image. Alternatively, the zoom value may be electronically reported by the fluoroscopy device 660, or the zoom value may be provided in or with the fluoroscopy image 654, and the image projection 654 may be modified to match the fluoroscopy zoom, as electronically reported by the fluoroscopy device.

Furthermore, because magnification may be applied to a fluoroscopy device, there may be variations in the magnification effect that may be difficult to measure directly. For example, magnification variations may occur as the distance between the x-ray detector and the object changes. Thus, radiopaque markers may be used for zoom adjustments. Thus, if a radiopaque marker (e.g., L-shaped) is identified in the magnified display and the radiopaque marker is twice the known physical size, the image dataset may be scaled to match twice the size of the radiopaque marker. When the image dataset is zoomed out or in, it is unlikely that the image dataset will align well with the actual view of the person's body. As a result, the zoomed image dataset and the fluoroscopy image may be displayed in registration on an AR display 662 separate from the AR headset, or in a portion of the AR headset field of view that is not directly overlaid on the person's body (e.g., from the side of the center of the AR headset field of view).

To reduce magnification effects due to the distance of the patient's anatomy from the x-ray source or x-ray detector, a round radiopaque sphere can be placed at the isocenter of the patient. If magnification is used in the fluoroscopy device, the magnification can vary depending on the distance of the body part from the x-ray beam source. For example, the magnification of body parts closer to the x-ray beam source may be greater. To correct for these magnification differences, the radiopaque marker used can be a metal sphere of known size (e.g., 1 cm). The metal sphere can be placed near the body part of interest (e.g., the isocenter of the image) and then the zoom of the image data set can be set to match the metal sphere. Thus, if the metal sphere appears smaller or larger in the fluoroscopic image, the image data set can be zoomed based on its size. The metal sphere also appears the same from all angles. This consistency of appearance allows the metal sphere to be an effective marker for detecting fluoroscopic image zoom applied to the image data set.

The use of the AR display 662 may also provide the ability to change the view of the fluoroscopic images and image projections to be displayed aligned on the AR display in order to display the virtual patient to better match the position and orientation of the actual view of the patient as viewed directly by the medical professional. Regardless of the orientation of the fluoroscopic images captured by the fluoroscopic device, the display of the aligned fluoroscopic images and image projections may be oriented in the AR display to assist the physician and/or match the actual view of the patient's position and/or orientation. For example, the fluoroscopic images may be flipped horizontally, vertically, or anisotropically in some other way compared to the patient's body as viewed in reality. The difficult viewing direction may be due to the capture direction of the fluoroscopic device 660. Thus, the images with the aligned fluoroscopic images 654 and image projections 656 may be flipped or reoriented (e.g., flipped horizontally, flipped by a certain angle, moved to reverse the misaligned orientation) for viewing by the medical professional to make the medical procedure easier to perform or more closely match what the medical professional sees in the actual view. This ability to change the image orientation allows for more intuitive interaction when performing a procedure on the patient. For example, performing a medical procedure with fluoroscopic image guidance can be very difficult when everything is posterior.

Graphic indicators, virtual tools, or virtual target systems may also be used in the image data set or image projection to guide the location and orientation of a fluoroscopically visible object (e.g., a needle or catheter) relative to the person's body and image data set when viewed using the AR display. Similarly, graphic indicators may be placed on the fluoroscopic image 654 to help guide a fluoroscopic object during a medical procedure. Alternatively, the graphic indicators may be used to guide any medical tools 618 used in the fluoroscopic procedure.

8A shows a side view of a cross-sectional view of a body of a person 806, a combination of image projections 804 from a registered image data set 820, a registered fluoroscopic image 802, and a fluoroscopy device 814, which may enable a medical professional to fluoroscopically guide a visible item 810 within the body of the person 806. A see-through object 810 (e.g., a needle) may be observed and guided by a medical professional within the fluoroscopic image 802 relative to the image data set 820 and/or the image projection 804 registered with the body of the person. The image projection 804 may appear superimposed on the fluoroscopic image 802 in an AR headset or AR display. Alternatively, the image projection 804 may appear to have the fluoroscopic image 802 overlaid on top of the image projection 804, or the image projection 804 may appear to be within the registered image data set 820. As discussed above, graphic indicators, virtual tools, or virtual target systems may be provided on the image dataset or image projection to guide the position and orientation of the see-through object 810 relative to the body of the person 806 and the image dataset 820.

The transparency of the fluoroscopic image 802 registered with the image dataset can be changed depending on how much of the image projection 804 or real-world body of the person 806 the medical professional wishes to see. The fluoroscopically visible object 810 or medical tool can also have one or more optical codes on the fluoroscopically visible object 810 that are used to reference the fluoroscopically visible object 810 to the image projection 804. Furthermore, the position of the medical tool relative to the person's body and the image projection 804 or image dataset 820 can be determined using the optical code on the medical tool and one or more optical codes on the body of the person 808, allowing the medical tool to be referenced to the image data and the fluoroscopic image when viewed through the AR display.

The image projections 804 from the image dataset 820 may have been captured using computed tomography (CT) images or magnetic resonance imaging (MRI). The image projections 804 can then be overlaid on the live fluoroscopic image 802. Although the fluoroscopic image 802 is a live image, the fluoroscopic image does not have the 3D quality or soft tissue contrast resolution of an MRI (magnetic resonance imaging) or CT (computed tomography) image. If the fluoroscopic image 802 references the patient's body with one or more optical codes (e.g., AprilTag) and the projections 804 of the previously acquired 3D image dataset reference image visible tags on the patient's body, the medical professional can see the virtual end of the needle as the tip moves within the patient's 806 body using an AR headset or AR display. This combines valuable aspects of virtual images, fluoroscopic images, and the actual view of the patient. Although the needle, catheter tip, or similar radiopaque object can be seen under fluoroscopy, the medical professional cannot see certain soft tissues in the fluoroscopic image. Thus, the combination of the actual view of the patient, the image dataset, the fluoroscopic image, the optical tag on the fluoroscopic device, the image visible tag, the medical tool, and the optical tag on the medical tool can enable a medical professional to view the medical tool with reference to the image dataset and/or the fluoroscopic image.

In another configuration, as shown in FIG. 8B, an ultrasound probe or transducer 862 may be used to obtain an ultrasound image 860 of a portion of a patient's 850 body while a medical procedure is being performed by a medical professional. The ultrasound transducer 862 may be movable or moveable relative to the person's body. The ultrasound image 860 or sonogram may be a 2D, 3D, or 4D (e.g., including a time series) ultrasound image produced by sound waves bouncing or echoing off body tissue. The echoes are processed by a computer to create the ultrasound image or sonogram.

These ultrasound images 860 may be relatively fast and inexpensive to obtain during a medical procedure, but the resolution, accuracy, and localization of the ultrasound images may not be as advanced as other types of imaging, such as CT scans, MRIs, and other imaging modalities. This technology provides the ability to combine ultrasound images 860 with other imaging modalities to assist medical professionals in performing medical procedures.

Thus, one or more optical codes 864 may be disposed or attached to the ultrasound probe or ultrasound transducer 862 (e.g., on the outer housing). The optical code on the ultrasound transducer 862 may be detected by a sensor in the AR headset. As a result, the position and orientation of the ultrasound transducer 862 may be detected, and the position and orientation of the ultrasound beam and the ultrasound image may be determined based on the position and orientation of the ultrasound transducer 862. Furthermore, one or more optical codes 852 on the patient or person's body may be detected to determine the position and orientation of the patient 850. Knowing the position and orientation of the patient 850 and the ultrasound image 860 allows the ultrasound image 860 to be aligned and projected to the correct position on the patient using the AR headset. The ultrasound image 860 projected on the patient in the AR headset may be partially transparent (e.g., using a transparency value set by a medical professional) or the ultrasound image may be opaque.

The ultrasound image 860 or sonogram can also be combined with an image dataset 866 (e.g., MRI), which can be more accurate, clearer, higher resolution, larger, and have better tissue contrast information compared to the ultrasound image 860. As previously described, the optical code 864 may be attached to an image visible marker 856. This image visible marker 856 can allow for the image dataset 866 previously acquired from the patient to be aligned with the patient's body (as previously described in detail). Thus, the medical professional can view the ultrasound image 860 combined with the high-resolution image dataset 866 as aligned and projected to the correct location on the patient's body via an AR headset or AR system. For example, if the medical professional is performing a medical procedure on the patient's liver using an ultrasound device, a limited portion of the patient's liver can be viewed at any one time using the ultrasound image, but the entire liver can be viewed using the image dataset acquired together with the ultrasound image. The ultrasound image and the image dataset can be co-localized in the 3D space viewed by the AR headset.

Typically, the ultrasound image 860 is not fixed to a reference point relative to the patient. Using the optical code as described above provides a reference point in the 3D space seen by the AR headset. Additionally, the reference point (i.e., the optical code) can be used to align the ultrasound image 860 with one or more image data sets 866.

Ultrasound transducers can use electronically guided fan beams or linear beams. If the ultrasound beam is moved or guided mechanically or electronically, this guidance can be taken into account when determining the position of the ultrasound image.

When a medical professional is attempting to perform a medical procedure such as a breast biopsy, the medical professional may use ultrasound equipment, but it may be difficult to see the actual lesion in the ultrasound image. However, the lesion may be visible in a CT, MRI, PET, nuclear, or other image dataset that is displayed in combination or co-localization with the ultrasound image. Thus, ultrasound images may be used to provide images captured during the procedure (e.g., in real time), while previously captured detailed anatomy may be simultaneously referenced using an image dataset of higher spatial or contrast resolution.

The transducer may pass through the surface of the body or may be inserted into an orifice in the body, so that fusion or synthetic views of the ultrasound images and image data sets can provide many different synthetic views of the patient's body.

As described in FIG. 8B, the registration of the real-time image with the patient's body and image dataset may be applied to any type of real-time medical imaging where the position and orientation of the real-time image may be obtained from a transducer, emitter or detector of an imaging device. An additional example of such real-time imaging that may be used instead of an ultrasound image is a CT fluoroscopic image.

FIG. 9a is a flow chart of an exemplary method of using an augmented reality headset to co-localize a medical tool with respect to an image dataset and a patient's body during a medical procedure. In these and other embodiments, the method 900 may be executed by one or more processors based on one or more computer-readable instructions stored on one or more non-transitory computer-readable media.

The method may capture visual image data of a portion of a person's or patient's body and a medical implement using an optical sensor in an AR headset, as in block 902. For example, a visible patient having an optical code may be registered or detected by a camera in an AR headset used by a medical professional. One or more optical codes associated with the person's body and the medical implement may be identified in the visual image data, as in block 904. As previously discussed, the optical code on the person's body may be located in a fixed position relative to the image visible markers.

As per block 906, the image dataset may be aligned with the person's body using one or more optical codes on the person's body and fixed positions of the image visible markers relative to the optical codes when referenced to a representation of the image visible markers in the image dataset. Further, as per block 908, the position and/or orientation of the medical implement relative to the person's body may be determined using the medical implement and one or more optical codes on the person's body to enable the medical implement to be referenced to the person's body and the image dataset. The alignment and composition of these multiple image aspects may be viewed through an AR headset. In one example, the image dataset of the radiology image may be presented with different levels of opacity depending on the needs of the user or medical professional. This opacity may also be adjusted from time to time.

In one configuration, the method can include providing visual indicators, virtual guidance systems, virtual tools, virtual highlights, or annotations on the image dataset to guide the positioning and orientation of the object relative to the patient's body using an AR headset. For example, the virtual tools or visual indicators can include one or more of: graphic highlighting (e.g., to highlight organs or masses within the body), 3D color schemes, 3D surgical paths or virtual treatment tracks, graphic highlighting of the target anatomical tissue, graphic highlighting of lesions, graphic highlighting of important structures to avoid, or 3D visual codes and graphic structures (e.g., lines, planes, cylinders, volumes, boundaries, or 3D shapes) in or around the anatomical structures. The method can further include mapping the contour or 3D contour of the medical tool using the optical code as a starting reference point. Additionally, because the position and orientation of the medical tool is known, a medical tool (e.g., an instrument or implant) that is partially inserted into the patient's body can still be monitored by the system to ensure correct placement and positioning relative to the image dataset and ultimately the person's body.

The method may include querying a database to determine whether the utilized or detected medical implement is assigned for use in the medical procedure. In one example, the AR system may store medical procedure records indicating which patients are undergoing a particular medical procedure. These records may include a list of specific medical implements (e.g., medical instruments, implants, etc.) associated with each procedure. If the medical implement is uniquely identified using an optical code, the optical code may be transmitted to the system and matched against a predefined list of medical implements (each of which may be associated with a separate optical code). Thus, if it is determined that the identified object is associated with the current medical procedure, a graphical indicator may be displayed that represents the medical implement being associated with the medical procedure. For example, a visible green indicator (e.g., a check mark or a green medical implement highlight) may indicate a match of the procedure. In another example, a medical implement identified as associated with the procedure may always be highlighted (e.g., surrounded by a green indicator in the AR display). On the other hand, if it is determined that the medical implement is not associated with the current procedure, a negative visual indication may also be displayed. For example, a flashing red "X" or a red highlight outline may be a negative indicator.

The method can include displaying, on the augmented reality display, medical information related to the medical implement. The information related to the medical implement can include instructional information describing use of the medical implement in a medical procedure.

One problem faced by physicians and other medical professionals when performing procedures on patients is ensuring that the correct medical tool is used on the correct patient anatomy. If the wrong person, wrong appendage, wrong location, wrong implant is operated on, or the wrong implant, wrong tool size, or wrong medical tool is utilized, poor medical outcomes can result. The present technology can provide improved medical outcomes by utilizing an optical code attached to the medical tool.

In a further configuration, the technique can be used to simulate medical procedures. A patient's body or patient anatomy can be simulated using a simulated structure. The simulated structure can be plastic or cadaveric bone covered with soft materials (plastics and rubber representing tissue, arteries, etc.) or other simulated materials. The simulated anatomy can include optical codes and image visibility codes. An image dataset of a patient on which a future procedure will be performed can then be aligned with the simulated anatomy.

The actual medical tools used in the procedure may also be included in the simulation and may be within the field of view of the AR headset. Similarly, a limited portion of the medical tool may be used (e.g., only the handle of a trocar) and the virtual tool may be viewed within the simulated patient. Additional overlays using fluoroscopy, ultrasound, or other medical images captured in real time may also be used. Thus, the medical professional may perform the same functions as described above as a simulation to better understand challenges, issues, or other problems that may arise in the actual procedure being planned. This simulation may also be used for training purposes.

Similarly, a medical trainer or sonographer can combine previously captured image data sets (e.g., MRI or CT images) with actual ultrasound images being captured in real time for training purposes (as described above). Due to the poor spatial and contrast resolution of ultrasound images, it can be difficult to clearly see what is displayed in the ultrasound image. This allows medical technicians to be trained on what organs, bones, blood vessels, and other tissues appear under ultrasound by using images with better spatial and contrast resolution (e.g., MRI, CT, etc.).

FIG. 9b illustrates a method for verifying a medical procedure using an optical code. The method may include, as in block 920, detecting visual image data of a portion of a patient's body and a medical tool using an optical sensor in an AR headset.

As per block 922, one or more optical codes visibly displayed on the patient's body and the medical implement may be identified. The one or more optical codes on the patient's body are in fixed positions relative to the image visible markers (as described above). As per block 924, the image data set may be aligned with the patient's body using the known fixed positions of the image visible markers with reference to the one or more optical codes on the patient's body.

As in block 926, the optical code and image visible markers may be used to verify that the correct patient is in the medical procedure based on the correct registration of the surface of the patient's body that aligns with the surface of the image data set. The surface of the patient's body and the surface of the image data set may be created using a polygonal mesh, spline, or another mathematical model of the surface. If the surfaces (e.g., mesh) are similar or match, the correct patient is recognized. This can be analogized to a contour type of "fingerprint" of the body, since every individual body is unique. Additionally, the identity of the person or patient in the medical procedure may also be verified using one or more optical codes. An additional function or aspect of this verification is to verify that the correct part of the body and the correct medical tool are being used in the medical procedure using one or more optical codes, as in block 928.

In a further configuration, the same optical code used during registration of the image data can be used to automatically acquire the image dataset, ensuring that the image data acquired by the AR headset matches the patient viewing through the AR headset without time-consuming, tedious, and inaccurate manual searching of the image data.

FIG. 10 illustrates an exemplary system that can be used to reference a medical implement to an image dataset and a patient's body using an optical code when viewed through an AR headset. The system 1000 can include a camera device 1002, an augmented reality system 1020, a display device 1030, and multiple databases. The system 1000 can also include one or more processors, memories, file systems, communication units, operating systems, and user interfaces that can be communicatively coupled together. In some embodiments, the system 1000 can be, for example, a desktop computer, a server computer, a laptop computer, a smartphone, a tablet computer, an embedded computer, an AR headset, a VR headset, etc.

The camera device 1002 can be configured to capture visual data. In one example, the camera device 1002 can be used to capture visual data during a medical procedure. The visual data captured by the camera device 1002 can include images of a person's body (or a portion of the body) and one or more medical implements (e.g., medical instruments, implants, etc.). The camera device 1002 can transmit the captured optical data to the augmented reality system 1020. The system can also include surface sensors, optical sensors, infrared sensors, Lidar sensors, or other sensors to detect and assist in the mapping of the actual view or actual view detected by the AR system. Any object or surface can be detected for an operating room, a patient, a room, a physical shape, a medical implement, or any other physical surroundings or objects.

The augmented reality system 1020 may include an image processing engine 1022, a fiducial alignment module 1024, an image generation module 1026, and an augmented display buffer 1028. For example, the image processing engine 1022 may receive the captured visible image data from the camera device 1002 and analyze the visible image data to identify one or more optical codes, objects, or people within the visible image data. A number of different techniques may be used to identify medical tools within the visible image data, including, but not limited to, feature extraction, segmentation, and/or object detection.

The image processing engine 1022 also identifies optical codes that may be affixed to the body of both the patient in the image and the medical implement in the visible image data. Once the image processing engine 1022 identifies an optical code (e.g., an AprilTag, a barcode, a QR code, etc.), the image processing unit 1022 accesses the optical code database 1046 to retrieve information related to the optical code. In some examples, the optical code is associated with a particular patient, a particular procedure, or a particular object. The optical code can be used to more precisely identify the location and orientation of the medical implement, body, or fluoroscopy device.

In some embodiments, the fiducials and registration module 1024 engages the image processing engine 1022 to reference the image data set relative to any identified medical implements, the person's body, and each other. Additionally, the fiducials and registration module 1024 can use optical code information in the medical implement database 1044 to properly identify the size and shape of the medical implement. Once the position and orientation of the medical implement and the patient's body are determined relative to each other, the fiducial registration controller 1026 can register any associated radiological images in the radiological image data 1042 with both of the patient's bodies. In some examples, the radiological images are received from the radiological image database 1042 based on the patient records in the patient record database 1040.

The image generation module 1026 can generate graphical data, virtual tools, 3D surgical paths, 3D coloring or shading of masses or organs, or highlighting of masses, organs or targets for display on a display device 1030 layered on top of the patient's body or medical tool. In some examples, this information can be loaded into an expanded display buffer 1028. This information may then be transmitted to the display device 1030 for display to the user.

In one example, the patient database 1040 includes multiple patient records. Each patient record can include one or more medical procedures performed on the patient. The patient records can also include notes, instructions, or plans for the medical procedures. The patient records can also be associated with one or more radiological images in the radiological image database 1042. In some examples, the radiological images include representations of image visible markers that allow the fiducials and registration module 1026 to properly register the image dataset with the patient's body using a fixed position of the optical code relative to the image visible markers. In some examples, the medical implement data 1044 includes information describing medical implements, including medical instruments, implants, and other objects.

In some embodiments, the augmented reality system may be located on a server, which may be any computer system capable of functioning in conjunction with the AR headset or the display device 1030. In some embodiments, the server may be configured to communicate with the AR headset over a computer network to communicate image data to the AR headset or to receive data from the AR headset.

FIG. 11 illustrates a computing device 1110 on which modules of the present technology may be executed. A computing device 1110 on which a high level example of the present technology may be executed is shown. The computing device 1110 may include one or more processors 1112 in communication with a memory device 1112. The computing device may include a local communication interface 1118 for components within the computing device. For example, the local communication interface may be a local data bus and/or any associated address or control bus as desired.

The memory device 1112 may include modules 1124 executable by the processor(s) 1112 and data for the modules 1124. The modules 1124 may perform the functions described above. A data store 1122 may also be located within the memory device 1112 for storing data related to the modules 1124 and other applications along with an operating system executable by the processor(s) 1112.

Other applications may also be stored in memory device 1112 and executable by processor(s) 1112. The components or modules discussed in this description may be implemented in software using a high-level programming language that is compiled, interpreted, or executed using a hybrid approach.

A computing device may also have access to I/O (input/output) devices 1114 that are usable by the computing device. One example of an I/O device is a display screen that can display output from the computing device. Other known I/O devices may be used with the computing device as desired. Networking devices 1116 and similar communication devices may also be included with the computing device. Networking devices 1116 may be wired or wireless networking devices that connect to the Internet, a LAN, a WAN, or other computing networks.

The components or modules shown as being stored in the memory device 1112 may be executed by the processor(s) 1112. The term "executable" may refer to a program file in a format that can be executed by the processor 1112. For example, a program in a high-level language may be compiled into machine code in a format that can be loaded into a random access portion of the memory device 1112 and executed by the processor 1112, or source code may be loaded by another executable program and interpreted to generate instructions in a random access portion of the memory to be executed by the processor. The executable program may be stored in any portion or component of the memory device 1112. For example, the memory device 1112 may be a random access memory (RAM), a read-only memory (ROM), a flash memory, a solid state drive, a memory card, a hard drive, an optical disk, a floppy disk, a magnetic tape, or any other memory component.

Processor 1112 may represent multiple processors, and memory 1112 may represent multiple memory units operating in parallel with the processing circuitry. This may provide parallel processing channels for processing and data within the system. Local interface 1118 may be used as a network to facilitate communication between any of the multiple processors and the multiple memories. Local interface 1118 may use additional systems designed to coordinate communications, such as load balancing, bulk data transfer, and similar systems.

Some of the functional units described herein have been labeled as modules to more particularly emphasize their implementation independence. For example, a module may be implemented as a hardware circuit comprising custom VLSI circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in a programmable hardware device such as a field programmable gate array, programmable array logic, programmable logic device, or the like.

Modules may also be implemented in software for execution by various types of processors. An identified module of executable code may include one or more blocks of computer instructions that may be organized as, for example, an object, a procedure, or a function. Nevertheless, the executable files of an identified module need not be physically located together, but may include heterogeneous instructions stored in different locations that constitute a module and that, when logically combined together, accomplish the purpose stated for the module.

In fact, a module of executable code may be a single instruction, or many instructions, and may even be distributed across several different code segments, among different programs, and across several memory devices. Similarly, operational data may be identified and illustrated herein in modules, and may be embodied in any suitable form and organized within any suitable type of data structure. Operational data may be collected as a single data set or may be distributed across different locations, including across different storage devices. Modules may be passive or active, including agents operable to perform a desired function.

The techniques described herein may be stored on computer-readable storage media, including volatile and non-volatile, removable and non-removable media implemented in any technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other computer storage medium that may be used to store the desired information and the techniques described.

The devices described herein may also include communications connections or networking devices and networking connections that enable the devices to communicate with other devices. A communications connection is an example of a communication medium. Communication media typically embodies computer-readable instructions, data structures, program modules, and other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. A "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media include wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, radio frequency, infrared, and other wireless media. As used herein, the term computer-readable media includes communication media.

With reference to the examples illustrated in the drawings, specific language has been used herein to describe the same. It will nevertheless be understood that no limitation of the scope of the technology is intended thereby. Changes and further modifications of the features shown herein, as well as further applications of the examples shown herein that may occur to those skilled in the art having this disclosure in hand, should be considered within the scope of the description.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the preceding description, numerous specific details, such as embodiments of various configurations, have been provided to provide a thorough understanding of embodiments of the described technology. However, one of ordinary skill in the art will recognize that the technology can be practiced without one or more of the specific details, or with other methods, components, devices, etc. In other cases, well-known structures or operations have not been shown or described in detail to avoid obscuring aspects of the technology.

Although the subject matter has been described in language specific to structural features and/or operations, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features and operations described above. Rather, the specific features and operations described above are disclosed as exemplary forms of implementing the claims. Numerous modifications and alternative configurations may be devised without departing from the spirit and scope of the described technology.

Claims (29)

1. A method for using an augmented reality (AR) headset to co-localize an image dataset and a medical tool having one or more optical codes, the method comprising:
detecting visual image data of a portion of a human body and the medical tool using an optical sensor of the AR headset;
identifying one or more optical codes associated with the body of the person and the medical implement, the optical codes on the body of the person being located in a fixed position relative to an image visible marker provided on the body of the person ;
aligning the image dataset with the body of the person using one or more optical codes on the body of the person when viewed through the AR headset and using the fixed positions of the image visible markers relative to the optical codes when referencing a representation of the image visible markers in the image dataset;
determining a position of the medical tool relative to the body of the person using one or more optical codes on the medical tool and the body of the person to enable the medical tool to be referenced to the image dataset and the body of the person when viewed through the AR headset;
A method comprising: The method of claim 1, further comprising providing visual indicators, virtual tools, virtual guidance systems, virtual highlighting, virtual treatment tracks, graphic highlighting of targeted anatomical tissues, graphic highlighting of lesions, graphic highlighting of critical structures to avoid, or annotations on the image dataset to guide positioning and orientation of the medical tool relative to the patient's body using the AR headset. acquiring patient information using the optical code associated with the portion of the body of the person;
displaying the patient's information via the AR headset to verify that the correct patient and the correct body part are receiving the medical procedure;
The method of claim 1 further comprising: querying a database to determine if the medical device is allocated for use in a medical procedure;
displaying a graphical indicator to a medical professional representing whether the medical implement is associated with the medical procedure; and
The method of claim 1 further comprising: The method of claim 1, further comprising displaying a graphical virtual instrument to enable alignment of the visible instrument with the virtual instrument. The method of claim 1, further comprising obtaining medical information associated with the medical device, the medical device including instructional information describing use of the medical device in a medical procedure. The method of claim 1, wherein the medical device is a medical instrument, a trocar, a catheter, a needle, orthopedic hardware, or a surgical implant. The method of claim 1, further comprising mapping the contour of the medical device using one or more optical codes as a starting reference point. The method of claim 1, further comprising automatically acquiring the image data set for the body of the person using one or more optical codes on the body of the person. 1. A method for registering a perspective image to a human body and an image projection from an image dataset using an augmented reality (AR) display, comprising:
detecting visual image data of a portion of the body of the person using an optical sensor of the AR display and a see-through device movable relative to the body of the person;
identifying one or more optical codes on the body of the person and on the imaging device, the one or more optical codes on the body of the person having fixed positions relative to image visible markers provided on the body of the person ;
registering an image dataset of the body of the person using the fixed positions of the image visible markers relative to the one or more optical codes on the body of the person when viewed through the AR display;
determining a position and orientation of the fluoroscopy device relative to the body of the person using the one or more optical codes on the fluoroscopy device;
generating an image projection of the image data set based in part on the position and orientation of the fluoroscopy device;
Displaying the image projection through the AR display; and
using the AR display to display a perspective image from the perspective device aligned with the body of the person based on the position and orientation of the image visible markers or the perspective device, and the image projection;
A method comprising: detecting changes in the position and orientation of a fluoroscopy device relative to the body of the person;
modifying the image projection and fluoroscopic image position and orientation as defined by changes in position and orientation of the fluoroscopy device;
The method of claim 10 further comprising: 11. The method of claim 10, wherein adjusting the position of the image projection as viewed by the AR display is based on changes in the position and orientation of the fluoroscopy device compared to the position and orientation of the patient's body as detected using the one or more optical codes on the fluoroscopy device. receiving a zoom value from the fluoroscopy device zoomed in on the body of the person;
adjusting the image data set as defined by a zoom value of the fluoroscopy device;
The method of claim 10 further comprising: The method of claim 13, wherein adjusting the image dataset is performed based on using a size of the image visible markers captured in the fluoroscopic image to enable the image dataset to match a zoom of the fluoroscopic image. The method of claim 10, wherein a see-through object in the fluoroscopic image can be viewed and navigated by a medical professional with respect to the image data set aligned with the body of the person. 11. The method of claim 10, further comprising providing a graphical indicator, virtual tool, or virtual target system on the image dataset to guide the position and orientation of a see-through object relative to the body of the person and the image dataset when viewed using the AR display. The method of claim 10, wherein determining the orientation of the fluoroscopy device further comprises determining a position and orientation of the fluoroscopy device relative to the body of the person. The method of claim 10, further comprising reconstructing the image projections of the image dataset and moving the perspective images when viewed on the AR display in response to changes in orientation and position of the perspective device. The method of claim 10, wherein the transparency of the fluoroscopic image registered with the image dataset may be corrected. identifying one or more additional optical codes visibly displayed on the medical device;
11. The method of claim 10, further comprising: determining a position of the medical tool relative to the body of the person and the image projection using one or more additional optical codes on the medical tool and one or more optical codes on the body of the person to enable the medical tool to see the image data when viewed through the AR display. 1. A method for using an augmented reality (AR) display to align a fluoroscopic image with a human body having one or more optical codes by using a position and orientation of a fluoroscopic device having one or more optical codes, the method comprising:
identifying the one or more optical codes on the body of the person and on the vision device movable relative to the body of the person using an optical sensor of the AR display;
determining the position and orientation of the fluoroscopy device relative to the body using the one or more optical codes on the fluoroscopy device;
and using the AR display to display a fluoroscopic image from the fluoroscopic device aligned with the body of the person by referencing the optical code on the body of the person and the position and orientation of the fluoroscopic device. identifying one or more optical codes on the body of the person having fixed positions relative to image visible markers provided on the body of the person ;
registering an image dataset of the body of the person using the fixed positions of the image visible markers relative to the one or more optical codes on the body of the person when viewed through the AR display;
displaying the image dataset via the AR display in registration with the body of the person and with the perspective image;
22. The method of claim 21 further comprising: generating an image projection of the image data set based on the position and orientation of the fluoroscopy device;
displaying the image projection via the AR display in registration with the body of the person along with the perspective image;
23. The method of claim 22, further comprising: 23. The method of claim 22, wherein the image projections can be reconstructed based on movement of the fluoroscopy device to correspond with the changed position of the fluoroscopy device. 22. The method of claim 21, further comprising generating an image projection of the image dataset based on the position and orientation of an observer using an AR headset. 1. A method of aligning an ultrasound image with respect to a human body using an augmented reality (AR) display, comprising:
detecting visual image data of a portion of the body of a person and an ultrasound transducer using an optical sensor of the AR display;
identifying one or more optical codes on the body of the person and on the ultrasound transducer;
determining a position and orientation of the ultrasound transducer relative to the body of the person using the one or more optical codes on the ultrasound transducer;
and using the AR display to display an ultrasound image from the ultrasound transducer aligned with the body of the person by referencing the optical code on the body of the person and the position and orientation of the ultrasound transducer. 1. A method for verifying a medical procedure using an optical code, comprising:
Detecting visual image data of a patient's body part and a medical tool using an optical sensor of the AR headset;
identifying one or more optical codes visibly displayed on the patient's body and on the medical implement, the one or more optical codes on the patient's body being in a fixed position relative to an image visible marker provided on the patient's body ;
registering an image dataset with the patient's body using known fixed positions of the image visible markers with reference to the one or more optical codes on the patient's body;
verifying that the correct patient is undergoing the medical procedure based on a correct registration of a surface of the body of the patient with a surface of the image data set;
using the one or more optical codes to verify that the correct part of the body and the correct medical implement are in the medical procedure;
A method comprising: 28. The method of claim 27, further comprising acquiring patient data for display in the AR headset using the one or more optical codes. 28. The method of claim 27, further comprising using the AR headset to provide annotations to the image dataset to guide the position and orientation of the medical tool relative to the patient's body and the image dataset.

Images