US20180256258A1 - Augmented reality diagnosis guidance - Google Patents
Augmented reality diagnosis guidance Download PDFInfo
- Publication number
- US20180256258A1 US20180256258A1 US15/919,139 US201815919139A US2018256258A1 US 20180256258 A1 US20180256258 A1 US 20180256258A1 US 201815919139 A US201815919139 A US 201815919139A US 2018256258 A1 US2018256258 A1 US 2018256258A1
- Authority
- US
- United States
- Prior art keywords
- augmented reality
- virtual
- patient
- display
- impingement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/25—User interfaces for surgical systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/361—Image-producing devices, e.g. surgical cameras
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/012—Head tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/04815—Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/04817—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance using icons
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
- G16H40/63—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for local operation
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
- A61B2034/101—Computer-aided simulation of surgical operations
- A61B2034/102—Modelling of surgical devices, implants or prosthesis
- A61B2034/104—Modelling the effect of the tool, e.g. the effect of an implanted prosthesis or for predicting the effect of ablation or burring
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
- A61B2034/101—Computer-aided simulation of surgical operations
- A61B2034/105—Modelling of the patient, e.g. for ligaments or bones
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
- A61B2034/108—Computer aided selection or customisation of medical implants or cutting guides
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B2090/364—Correlation of different images or relation of image positions in respect to the body
- A61B2090/365—Correlation of different images or relation of image positions in respect to the body augmented reality, i.e. correlating a live optical image with another image
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
- A61B2090/372—Details of monitor hardware
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/048—Indexing scheme relating to G06F3/048
- G06F2203/04806—Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
Definitions
- Diagnostics are used to evaluate a patient to determine whether the patient needs a surgical procedure, such as a total hip arthroplasty, ligament repair, knee replacement, or the like. These procedures are performed hundreds of thousands of times a year in the United States. Surgical advancements have allowed surgeons to use preoperative planning, display devices, and imaging, to improve diagnoses and surgical outcomes. While these advances have allowed for quicker and more successful diagnoses, these diagnostic techniques are highly technical and difficult for patients to understand.
- FIG. 1 illustrates a clinician and a patient using augmented reality in accordance with some embodiments.
- FIGS. 2A-2F illustrate diagnostic augmented reality displays in accordance with some embodiments.
- FIG. 3 illustrates a clinician augmented reality display and a patient augmented reality display in accordance with some embodiments.
- FIG. 4 illustrates a display including virtual range of motion components in accordance with some embodiments.
- FIG. 5 illustrates a three dimensional rendering of a virtual component from two dimensional images in an augmented reality display in accordance with some embodiments.
- FIG. 6 illustrates a system for diagnosis using augmented reality in accordance with some embodiments.
- FIG. 7 illustrates a flow chart showing a technique for diagnosis using augmented reality in accordance with some embodiments.
- FIG. 8 illustrates generally an example of a block diagram of a machine upon which any one or more of the techniques discussed herein may perform in accordance with some embodiments.
- An augmented reality (AR) device allows a user to view displayed virtual objects that appear to be projected into the real environment, which is also visible.
- AR devices typically include two display lenses or screens, including one for each eye of a user. Light is permitted to pass through the two display lenses such that aspects of the real environment are visible while also projecting light to make virtual elements visible to the user of the AR device.
- An AR device includes an AR display which may be used to display a virtual component while allowing real objects to be viewed.
- the virtual component may include measurements, angles, an identification of an impingement (e.g., a femoroacetabular impingement) or a potential impingement, a representation of an implant, a representation of patient anatomy, a range of motion plane or axis, or the like.
- an impingement e.g., a femoroacetabular impingement
- a potential impingement e.g., a representation of an implant, a representation of patient anatomy, a range of motion plane or axis, or the like.
- alignment angles may be shown virtually on an image of a patient or on a virtually presented skeleton, for example overlaid on the patient, using the AR display.
- Patient imaging may be performed (e.g., using a CT scan, an Mill, an X-ray, or the like) to create at least one patient image.
- two images may be used (e.g., two-dimensional images) to create a three-dimensional representation of an aspect of a patient's anatomy.
- the three-dimensional representation may be displayed virtually using an AR display.
- the three-dimensional representation may be manipulated or observed in the AR display (e.g., rotated, moved, zoomed in or out on, virtually cut, etc.).
- Two or more AR devices may be used in a coordinated manner, for example with a first AR device controlling one or more additional AR devices, or in a system with defined roles.
- a user may select a role (e.g., clinician or patient) and the AR device may display information relevant to that role.
- the AR display for a clinician may include more information or more complex or technical information than the AR display for a patient, which may be simplified for ease of understanding.
- the clinician AR device may control what is viewed on the patient AR device.
- FIG. 1 illustrates a clinician 102 and a patient 108 using augmented reality in accordance with some embodiments.
- the clinician is wearing an augmented reality (AR) device 104 which may be used to display a virtual object to the clinician 102 .
- AR augmented reality
- the virtual object may not be visible to others if they are not wearing an AR device.
- an AR device e.g., the AR device 110 , which may have a camera 112
- the person may not be able to see the virtual object.
- the patient 108 may be able to see the virtual object in a shared augmented reality with the clinician 102 , or may be able to see a modified version of the virtual object (e.g., according to customizations unique to the clinician 102 or the patient 108 ).
- Augmented reality is explained in more detail below.
- Augmented reality is a technology for displaying virtual or “augmented” objects or visual effects overlaid on a real environment.
- the real environment may include a room or specific area, or may be more general to include the world at large.
- the virtual aspects overlaid on the real environment may be represented as anchored or in a set position relative to one or more aspects of the real environment.
- a virtual object may be configured to appear to be resting on a table.
- An AR system may present virtual aspects that are fixed to a real object without regard to a perspective of a viewer or viewers of the AR system (e.g., the clinician 102 ).
- the virtual object may exist in a room, visible to a viewer of the AR system within the room and not visible to a viewer of the AR system outside the room.
- the virtual object in the room may be displayed to the viewer outside the room when the viewer enters the room.
- the room may act as a real object that the virtual object is fixed to in the AR system.
- the AR device 104 may include one or more screens, such as a single screen or two screens (e.g., one per eye of a user).
- the screens may allow light to pass through the screens such that aspects of the real environment are visible while displaying the virtual object.
- the virtual object may be made visible to the clinician 102 by projecting light.
- the virtual object may appear to have a degree of transparency or may be opaque (i.e., blocking aspects of the real environment).
- An AR system may be viewable to one or more viewers, and may include differences among views available for the one or more viewers while retaining some aspects as universal among the views.
- a heads-up display may change between two views while virtual objects may be fixed to a real object or area in both views. Aspects such as a color of an object, lighting, or other changes may be made among the views without changing a fixed position of at least one virtual object.
- a user may see the virtual object presented in an AR system as opaque or as including some level of transparency.
- the user may interact with the virtual object, such as by moving the virtual object from a first position to a second position.
- the user may move an object with his or her hand. This may be done in the AR system virtually by determining that the hand has moved into a position coincident or adjacent to the object (e.g., using one or more cameras, which may be mounted on an AR device, such as AR device camera 106 or separate, and which may be static or may be controlled to move), and causing the object to move in response.
- Virtual aspects may include virtual representations of real world objects or may include visual effects, such as lighting effects, etc.
- the AR system may include rules to govern the behavior of virtual objects, such as subjecting a virtual object to gravity or friction, or may include other predefined rules that defy real world physical constraints (e.g., floating objects, perpetual motion, etc.).
- An AR device 104 may include a camera 106 on the AR device 104 .
- the AR device camera 106 may include an infrared camera, an infrared filter, a visible light filter, a plurality of cameras, a depth camera, etc.
- the AR device 104 may project virtual items over a representation of a real environment, which may be viewed by a user.
- the clinician 102 may control a virtual object using the AR device 104 , a remote controller for the AR device 104 , or by interacting with the virtual object (e.g., using a hand to “interact” with the virtual object or a gesture recognized by the camera 106 of the AR device 104 ).
- the AR device 104 and the AR device 110 may be part of a shared AR system.
- the shared AR system may allow the AR devices to view the same or similar content, to be in communication with each other, for one to control the other, or the like.
- the clinician 102 may control the AR device 110 of the patient 108 using the clinician's own AR device 104 (e.g., using a remote, a gesture, a button on the AR device 104 , or the like).
- the clinician 102 and the patient 108 may select their respective roles when initiating the AR devices 104 and 110 respectively.
- the AR devices may automatically provide appropriate virtual objects to be displayed.
- the clinician 102 may select the clinician role for the AR device 104 .
- the clinician 102 may be presented with a plurality of choices for display, such as diagnostic tools, viewing imaging of the patient 108 , selecting a patient file, education information, or the like.
- the AR device 104 may display virtual objects or information corresponding to the choice.
- the AR device 110 may automatically display virtual objects or information (which may correspond to the choice, to the clinician-displayed virtual objects or information, or may be derived from the choice or the clinician-displayed virtual objects or information) for view by the patient 108 .
- the clinician 102 may choose to view an image of a bone of the patient 108 .
- the image may be previously obtained.
- the AR device 104 may display, in response to the choice, a three-dimensional view of the bone, and for example, additional clinical information (e.g., technical data).
- the AR device 110 may automatically display, in response to the choice a three-dimensional view of the bone without the additional clinical information or with basic educational information (e.g., labels for names of parts of the bone).
- the clinician 102 may manipulate the virtual representation of the bone (e.g., using a button or gesture to rotate or zoom the bone) and the changes to the bone may be automatically reflected in the AR device 110 of the patient 108 .
- the AR device 104 of the clinician 102 may automatically identify a bony impingement (e.g., using the camera 106 or by analyzing an image).
- a virtual representation of the impingement may be displayed (e.g., as a red virtually-highlighted area) using the AR device 104 or the AR device 110 , for example on an image, presented on the patient 108 (e.g., with a three-dimensional skeletal overlay on the patient 108 ), or the like.
- the virtual representation of the impingement may move as the patient 108 moves, and may disappear or appear to fade away if the impingement is (temporarily) removed (e.g., if it is temporarily not present when the patient 108 sits or stands).
- FIGS. 2A-2F illustrate diagnostic augmented reality displays in accordance with some embodiments.
- FIG. 2A illustrates a first view 202 A of a patient without augmented or virtual components and a second view 202 B of the patient with a virtual skeleton overlaid on the patient.
- the virtual skeleton may be based on imaging of the patient.
- the skeleton may be created by using two images (e.g., X-rays) of the patient to create a three-dimensional model of the patient's skeleton, which may be displayed virtually using an AR display as shown in the second view 202 B.
- the skeleton may be determined in real time from an image of the patient using a camera of an AR device that is then used to display the skeleton, based on a generic or template skeleton model scaled or deformed to fit the image of the patient taken by the AR device.
- one or more landmarks on the virtual skeleton may be associated with bony landmarks of the patient (such as at the knee joint).
- the landmarks may be identified and tracked by the AR device and used to align the virtual skeleton with the real-life moving patient anatomy.
- the one or more landmarks may be used to align and scale the virtual skeleton to the patient (e.g., a foot landmark, a head landmark, a hip landmark, a hand landmark, etc.).
- the landmarks may be identified using a registration device, such as a pointer or finger within an AR view. For example, a clinician may point to or touch each identified landmark to register the landmark to the patient. After registration, the virtual skeleton may be applied to the patient in the AR view.
- a generic or template skeleton may be overlaid on the patient.
- a skeleton based on the two images e.g., X-rays
- a clinician may toggle between the views using an AR device.
- FIG. 2B illustrates an AR display 203 of a patient.
- the AR display 203 may include technical information displayed virtually, such as height, weight, BMI, co-morbidities (e.g., hypertension, cardiovascular disease, peripheral vascular disease, congestive heart failure, renal function impairment, diabetes, respiratory disease, etc.) for a patient being viewed through the AR display 203 .
- the AR display 203 may include patient information (e.g., name, age, allergies, potential patient-specific complications, etc.), readiness information, time or counters, or the like.
- the AR display 203 may include virtually displayed diagnosis type information, such as information from a patient file, reported pain, any previously identified impingement, referral notes (e.g., from a general practitioner), family medical history information (e.g., parental impingement procedures or co-morbidities), or the like.
- the clinician may interact with the virtually displayed diagnosis type information, for example by zooming in or out, selecting the virtually displayed diagnosis type information (e.g., if additional information is available, such as a diagnosis recommendation), or the like.
- the displayed information may be accessed or changed in real time, including updating the information for display in the AR display 203 .
- a clinician using the AR display 203 may change the information displayed, such as by using a gesture or input device, or may save the information or a snapshot of the AR display 203 , such as to a patient file.
- FIG. 2C illustrates an AR display 205 including a virtual skeleton overlaid on a patient.
- the AR display 205 may include aspects described above in relation to AR display 203 .
- the virtual skeleton overlaid may be generated from an image of the patient or using a camera of an AR device including the AR display 205 .
- the virtual skeleton may be virtually coupled to the patient, for example if the patient moves, then the skeleton may move accordingly.
- FIG. 2D illustrates an AR display presenting a first view 204 A of a patient image and a second view 204 B of the patient image.
- the first view 204 A includes a virtually highlighted area 212 of an image or of a virtual representation of anatomy of a patient.
- the image may be obtained using x-ray, MRI, CT scan, or the like.
- the virtual representation may be a representation of an obtained image or may be a three-dimensional representation that may be created from, for example, two or more images used to create a three-dimensional model that may then be displayed virtually.
- the virtually highlighted area 212 may correspond with an impingement.
- the impingement may be identified from an image of anatomy of the patient or form virtually projected anatomy of the patient. For example, the impingement may be automatically identified and the virtually highlighted area 212 may be projected based on the automatic identification.
- the patient may report pain in an area, which may be evaluated for impingement.
- a clinician may virtually view the anatomy (e.g., at least part of a skeleton) and identify an impingement or a potential impingement.
- the AR display may highlight the impingement or the potential impingement (e.g., in red) virtually (e.g., display the virtually highlighted area 212 ).
- the virtually displayed anatomy may be manipulated by the clinician (e.g., rotated, zoomed, moved, etc.). The manipulation may be performed using a gesture, a button (e.g., on a remote or on the AR device), or the like. The manipulation may allow the clinician to better identify the impingement.
- the clinician may manipulate a portion of the virtually displayed anatomy while leaving other portions of the virtually displayed anatomy static (or move other portions in a different direction relative to the first portion). For example, the clinician may cause a virtually displayed knee to bend or hip to swivel or shoulder to rotate.
- impingement may be detected by the clinician or automatically by the AR device (e.g., by identifying overlap in the virtually displayed anatomy). The impingement may then be displayed as the virtually highlighted area 212 .
- the second view 204 B may include one or more virtual angles 206 or virtual lines used to indicate, for example, potential acetabular component position for a surgical intervention, an implant alignment, anatomy (e.g., bone, muscle, ligament, other soft tissue, or the like) or position, orientation, or location of anatomy, or may include Spinopelvic angles, described in more detail below with respect to FIG. 2E .
- the virtual angles 206 may be automatically detected and projected onto the image or virtual representation, for example using the highlighted portion as an origin location.
- FIG. 2E illustrates an AR display 207 including one or more virtual angles 208 .
- the virtual angles 208 referred to herein as Spinopelvic angles, are displayed virtually in the AR display 207 .
- the Spinopelvic angles may include a pelvic incidence (PI), a sacral tilt (ST), a pelvic femoral angle (PFA), an anteinclination (AI) of the pre-operative acetabulum, and a sacral acetabular angle (SAA).
- the PI and SAA may be static numbers that are the same standing and sitting.
- FIG. 2F illustrates a first AR display 210 A and a second AR display 210 B showing virtual components displayed in a real environment.
- the first AR display 210 A includes a virtual component (e.g., an impingement or potential impingement) highlighted on a virtual skeleton of a patient walking on a treadmill.
- the second AR display 210 B includes a virtual component (e.g., an impingement or potential impingement) highlighted on a virtual skeleton of a patient sitting on a chair.
- the virtual skeleton may be virtually coupled to the patient, for example if the patient moves, then the skeleton may move accordingly.
- the first AR display 210 A may allow a clinician to observe skeletal (or other anatomical aspects which may be displayed virtually) action when the patient walks, for example to examine range of motion, potential impingement, or other clinical analysis.
- the second AR display 210 B may allow a clinician to make observations or an examination of the patient in a seated position.
- the impingement or potential impingement (the highlighted virtual component) of AR displays 210 A or 210 B may be detected automatically (e.g., using an AR device or a camera) or may be identified by a clinician.
- the highlighting may be automatically applied, and when identified by the clinician, the highlighting may be applied in response to receiving input from the clinician indicating the identification.
- FIG. 3 illustrates a clinician augmented reality display 300 and a patient augmented reality display 301 in accordance with some embodiments.
- the clinician AR display 300 includes virtually displayed information, and example virtually displayed components or augmented images.
- bones 302 and 303 may be on a printed or screen-displayed image or may be virtually displayed.
- An implant or trial 304 may be displayed virtually on the bone 302 .
- a gap 308 may be displayed for the clinician on the clinician AR display 300 .
- the clinician AR display 300 may include a virtual representation 310 of the patient AR display 301 .
- the patient AR display 301 includes a simplified version of the view available in the clinician AR display 300 .
- the bones 302 and 303 may be shown on the patient AR display 301 without additional comorbidity information, virtually displayed information, the implant or trial 304 , the gap 308 , etc.
- the patient AR display 301 may be controlled by the clinician AR display 300 .
- the clinician may select aspects of the clinician AR display 300 to be shared with the patient AR display 301 .
- the clinician may, for example drag aspects of the clinician AR display 300 onto the virtual representation 310 of the patient view. Dragged aspects may then be displayed on the patient AR display 301 .
- the clinician may select a program or procedure which may then automatically display virtual components in the clinician AR display 300 or in the patient AR display 301 .
- FIG. 4 illustrates an AR display 400 including virtual range of motion components in accordance with some embodiments.
- the AR display 400 includes a femur 402 and a tibia 404 for illustration purposes of a knee replacement, but other bones may be displayed (e.g., for a shoulder replacement or hip replacement).
- the AR display 400 may include one or more planes or axes, such as planes 406 - 410 .
- the planes 406 - 410 may be used for range of motion tests.
- plane 408 may be displayed for extension or flexion range of motion testing.
- the plane 408 may be displayed virtually for the range of motion test to be conducted while keeping the knee within the plane 408 .
- the planes 406 - 410 may be displayed virtually to guide a clinician in conducting a range of motion test.
- FIG. 5 illustrates a three dimensional rendering of a virtual component from two dimensional images in an augmented reality display in accordance with some embodiments.
- the AR display 502 A includes two images that may be taken of anatomy of a patient, for example using an x-ray, an MRI, a CT scan, or the like. The images may be two-dimensional.
- the AR display 502 A may not include virtual components.
- the AR display 502 B may illustrate an animation to create a three-dimensional virtual representation of the patient anatomy (e.g., a bone) from the images.
- the AR display 502 B illustrates a transition stage in the animation between the two-dimensional images displayed in the AR display 502 A and the three-dimensional virtual representation illustrated in AR display 502 C.
- the AR displays 502 A- 502 C may represent three stages of an animation to create the three-dimensional virtual representation from the two-dimensional images.
- the AR display 502 C illustrates the three-dimensional virtual representation of the patient anatomy.
- the three-dimensional virtual representation may be interacted with by a clinician viewing the AR display 502 C, for example using a button, a remote, a gesture, etc.
- the interaction may manipulate the three-dimensional virtual representation, for example rotate, move, zoom, etc., the three-dimensional virtual representation.
- the virtual representation may be created using a 3D modeler running on a computer, such as that described in U.S.
- the clinician may detect impingement or potential impingement.
- the AR display 502 C may illustrate, highlight, or otherwise display the impingement or potential impingement virtually.
- the impingement or potential impingement may be highlighted in red.
- an impingement or potential impingement may be automatically detected, such as by virtually moving bones and detecting whether impingement has occurred.
- two bones may be displayed and moved relative to one another, and a camera may be used to detect if there is overlap between the two bones.
- the bones may be automatically moved for the clinician to detect impingement or potential impingement.
- FIG. 6 illustrates a system 600 for diagnosis using augmented reality in accordance with some embodiments.
- the system 600 may be used to perform the technique 700 described in relation to FIG. 7 below, for example, by using the processor 602 .
- the system includes an augmented reality device 601 that may be in communication with a database 616 .
- the augmented reality device 601 includes a processor 602 , memory 604 , an AR display 608 , and a camera 606 .
- the augmented reality device 601 may include a sensor 610 , a speaker 612 , or a haptic controller 614 .
- the database 616 may include image storage 618 or preoperative plan storage 620 .
- the augmented reality device 601 may be a HoloLens manufactured by Microsoft of Redmond, Wash.
- the processor 602 of the augmented reality device 601 includes an augmented reality modeler 603 .
- the augmented reality modeler 603 may be used by the processor 602 to create the augmented reality environment.
- the augmented reality modeler 603 may receive dimensions of a room, such as from the camera 606 or sensor 610 , and create the augmented reality environment to fit within the physical structure of the room.
- physical objects may be present in the room and the augmented reality modeler 603 may use the physical objects to present virtual objects in the augmented reality environment.
- the augmented reality modeler 603 may use or detect a table present in the room and present a virtual object as resting on the table.
- the AR display 608 may display the AR environment overlaid on a real environment.
- the display 608 may show a virtual object, using the AR device 601 , such as in a fixed position in the AR environment.
- the augmented reality device 601 may include a sensor 610 , such as an infrared sensor.
- the camera 606 or the sensor 610 may be used to detect movement, such as a gesture by a surgeon or other user, that may be interpreted by the processor 602 as attempted or intended interaction by the user with the virtual target.
- the processor 602 may identify an object in a real environment, such as through processing information received using the camera 606 .
- the AR display 608 may present, such as within a surgical field while permitting the surgical field to be viewed through the augmented reality display, a virtual feature corresponding to a physical feature hidden by an anatomical aspect of a patient.
- the virtual feature may have a virtual position or orientation corresponding to a first physical position or orientation of the physical feature.
- the virtual position or orientation of the virtual feature may include an offset from the first physical position or orientation of the physical feature.
- the offset may include a predetermined distance from the augmented reality display, a relative distance from the augmented reality display to the anatomical aspect, or the like.
- the AR device 601 may include a virtual surgical implant creation system 615 to create an implant, for example based on an image.
- the implant may be displayed on the augmented reality display as a virtual component.
- FIG. 7 illustrates a flow chart showing a technique 700 for diagnosis using augmented reality in accordance with some embodiments.
- the technique 700 includes an operation 702 to obtain an image of a patient, for example by using an imaging device (e.g., in real time).
- the virtual feature may be created using two or more two dimensional images, for example as obtained by the imaging device.
- the technique 700 includes an operation 704 to present, while permitting the patient to be viewed through an AR display, a virtual feature corresponding to anatomy of the patient obtained from the image.
- the virtual feature may include a bone or a set of bones, an implant, a skeletal model, for example presented overlaid on the patient, or the like.
- the skeleton may be configured to move when the patient moves (e.g., the virtual skeleton may be visually linked to the real anatomy of the patient).
- the technique 700 may include an operation 706 to detect impingement at the virtual feature (e.g., of the anatomy of the patient).
- the virtual feature may correspond to a bone of the patient and may be displayed virtually with other virtual bones. The bone may interact with one of the other virtual bones, indicating impingement.
- the impingement may be detected automatically, such as using a camera (e.g., by visually detecting the interaction using one or more images) or using a processor (e.g., by detecting the interaction from imaging data).
- the impingement may be identified by a clinician.
- the technique 700 may include receiving input from the clinician indicating the impingement. When the impingement is detected or the clinician input is received, the virtual feature or a portion of the virtual feature may be changed (e.g., highlighted, pulsed, flashed, changed color, etc.).
- the technique 700 may include an operation to determine, such as from a plurality of images (which may be captured by the imaging device described above), that the anatomy of the patient has moved.
- this operation may include determining that the impingement has resolved (e.g., there is no impingement in this orientation/configuration of the patient's anatomy, the anatomy is unimpinged, etc.), and in response to this determination, stop or pause display of an indication of the impingement at the virtual feature.
- the operation may include showing the indication faded or offset from the virtual feature (e.g., to suggest that the impingement exists in the prior orientation/configuration but is not present in a current orientation/configuration).
- the technique 700 may include an operation to present a plane or axis for orientation for performing a range of motion test on the patient.
- the technique 700 may include an operation to present one or more Spinopelvic angles corresponding to the virtual feature.
- the technique 700 may include an operation to create an implant, for example based on the image.
- the implant may be displayed on the augmented reality display as a virtual component.
- the AR device may be used to communicate with a second AR device.
- the AR device may be used to control aspects of operation of the second AR device (e.g., what is displayed).
- the AR device may be a surgeon's device and the second AR device may be a patient's device.
- the patient sees less details in a display of the second AR device than the surgeon sees in the AR display of the AR device.
- a corresponding virtual component displayed in a second AR display of the second AR device may be adjusted, moved, or changed accordingly.
- the technique 700 may include projecting a plane to allow the anatomy of the patient to be kept within the plane while performing a range of motion test (e.g., rotation, flexion, etc.).
- the plane may be used for any biomechanical measurement that is used in a diagnostic exam (e.g., extension or rotation in the shoulder, a scoliosis diagnosis, or the like).
- the technique 700 may include outputting an indication of impingement (e.g., of the anatomy of the patient, at the virtual feature, etc.), such as for display on the augmented reality display.
- the indication of the impingement may be displayed by the augmented reality display by changing an attribute of the virtual feature.
- the virtual feature may be highlighted, encircled, flash, change color, be augmented by one or more additional features (e.g., highlight, circle, color change, etc.), or the like.
- the virtual feature may be a hip or patient anatomy in an area of a hip.
- the indication of the impingement may include a red circle around the hip or the area, overlaying a color on the hip or the area (i.e., making the hip or the area appear to be a different color), flashing a light at or near the hip or the area, or the like.
- FIG. 8 illustrates generally an example of a block diagram of a machine 800 upon which any one or more of the techniques (e.g., methodologies) discussed herein may perform in accordance with some embodiments.
- the machine 800 may operate as a standalone device or may be connected (e.g., networked) to other machines.
- the machine 800 may operate in the capacity of a server machine, a client machine, or both in server-client network environments.
- the machine 800 may be a personal computer (PC), a tablet PC, a personal digital assistant (PDA), a mobile telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine.
- PC personal computer
- PDA personal digital assistant
- machine shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations.
- cloud computing software as a service
- SaaS software as a service
- Examples, as described herein, may include, or may operate on, logic or a number of components, modules, or like mechanisms.
- Such mechanisms are tangible entities (e.g., hardware) capable of performing specified operations when operating.
- the hardware may be specifically configured to carry out a specific operation (e.g., hardwired).
- the hardware may include configurable execution units (e.g., transistors, circuits, etc.) and a computer readable medium containing instructions, where the instructions configure the execution units to carry out a specific operation when in operation.
- the configuring may occur under the direction of the executions units or a loading mechanism.
- the execution units are communicatively coupled to the computer readable medium when the device is operating.
- the execution units may be configured by a first set of instructions to implement a first set of features at one point in time and reconfigured by a second set of instructions to implement a second set of features.
- Machine 800 may include a hardware processor 802 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 804 and a static memory 806 , some or all of which may communicate with each other via an interlink (e.g., bus) 808 .
- the machine 800 may further include a display unit 810 , an alphanumeric input device 812 (e.g., a keyboard), and a user interface (UI) navigation device 814 (e.g., a mouse).
- the display unit 810 , alphanumeric input device 812 and UI navigation device 814 may be a touch screen display.
- the display unit 810 may include goggles, glasses, or other AR or VR display components.
- the display unit may be worn on a head of a user and may provide a heads-up-display to the user.
- the alphanumeric input device 812 may include a virtual keyboard (e.g., a keyboard displayed virtually in a VR or AR setting.
- the machine 800 may additionally include a storage device (e.g., drive unit) 816 , a signal generation device 818 (e.g., a speaker), a network interface device 820 , and one or more sensors 821 , such as a global positioning system (GPS) sensor, compass, accelerometer, or other sensor.
- the machine 800 may include an output controller 828 , such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices.
- a serial e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.
- the storage device 816 may include a machine readable medium 822 that is non-transitory on which is stored one or more sets of data structures or instructions 824 (e.g., software) embodying or utilized by any one or more of the techniques or functions described herein.
- the instructions 824 may also reside, completely or at least partially, within the main memory 804 , within static memory 806 , or within the hardware processor 802 during execution thereof by the machine 800 .
- one or any combination of the hardware processor 802 , the main memory 804 , the static memory 806 , or the storage device 816 may constitute machine readable media.
- machine readable medium 822 is illustrated as a single medium, the term “machine readable medium” may include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) configured to store the one or more instructions 824 .
- machine readable medium may include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) configured to store the one or more instructions 824 .
- machine readable medium may include any medium that is capable of storing, encoding, or carrying instructions for execution by the machine 800 and that cause the machine 800 to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions.
- Non-limiting machine readable medium examples may include solid-state memories, and optical and magnetic media.
- machine readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.
- non-volatile memory such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices
- EPROM Electrically Programmable Read-Only Memory
- EEPROM Electrically Erasable Programmable Read-Only Memory
- flash memory devices e.g., electrically Erasable Programmable Read-Only Memory (EEPROM)
- EPROM Electrically Programmable Read-Only Memory
- EEPROM Electrically Erasable Programmable Read-Only Memory
- flash memory devices e.g., electrically Era
- the instructions 824 may further be transmitted or received over a communications network 826 using a transmission medium via the network interface device 820 utilizing any one of a number of transfer protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.).
- transfer protocols e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.
- Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), mobile telephone networks (e.g., cellular networks), Plain Old Telephone (POTS) networks, and wireless data networks (e.g., Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards known as Wi-Fi®, as the personal area network family of standards known as Bluetooth® that are promulgated by the Bluetooth Special Interest Group, peer-to-peer (P2P) networks, among others.
- the network interface device 820 may include one or more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or one or more antennas to connect to the communications network 826 .
- the network interface device 820 may include a plurality of antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques.
- SIMO single-input multiple-output
- MIMO multiple-input multiple-output
- MISO multiple-input single-output
- transmission medium shall be taken to include any intangible medium that is capable of storing, encoding or carrying instructions for execution by the machine 800 , and includes digital or analog communications signals or other intangible medium to facilitate communication of such software.
- Example 1 is an augmented reality device for use during patient diagnosis, comprising: an imaging device to capture a series of images of the patient; an augmented reality display to present, while permitting the patient to be viewed through the augmented reality display, a virtual feature corresponding to anatomy of the patient obtained from at least one image of the series of images; processing circuitry to: detect impingement of the anatomy of the patient based on a plurality of images from the series of images; and output, for display on the augmented reality display, an indication of the impingement at the virtual feature by changing an attribute of the virtual feature.
- Example 2 the subject matter of Example 1 includes, wherein the processing circuitry is further to create a virtual implant, based on the at least one image, and output the virtual implant for display on the augmented reality display.
- Example 3 the subject matter of Examples 1-2 includes, wherein the augmented reality device is in communication with a second augmented reality device, and wherein processing circuitry of the augmented reality device is further to output a command to control an aspect of operation of the second augmented reality device.
- Example 4 the subject matter of Example 3 includes, wherein the command causes the second augmented reality device to display fewer details in a display of the second augmented reality device than the presented in the augmented reality display of the augmented reality device.
- Example 5 the subject matter of Examples 3-4 includes, wherein the processing circuitry is further to: receive an input adjusting, moving, or changing a virtual component displayed in the augmented reality display; and output a command to cause a corresponding virtual component displayed in the second augmented reality display of the second augmented reality device to be adjusted, moved, or changed accordingly.
- Example 6 the subject matter of Examples 1-5 includes, wherein the augmented reality display is to project a virtual plane to allow a surgeon to keep the anatomy of the patient within a corresponding plane while performing a range of motion test, wherein the virtual plane is used for a biomechanical measurement used in a diagnostic exam.
- Example 7 the subject matter of Examples 1-6 includes, wherein the processing circuitry is further to receive an input manipulating the virtual feature.
- Example 8 the subject matter of Examples 1-7 includes, wherein the processing circuitry is further to: determine, from a second plurality of images from the series of images, that the anatomy of the patient has moved; and in response to the anatomy of the patient moving, determine that the impingement has resolved; and wherein, in response to determining that the impingement has resolved, the augmented reality display is to stop displaying the indication of the impingement at the virtual feature.
- Example 9 the subject matter of Examples 1-8 includes, wherein the virtual feature includes at least one of a virtual bone or set of virtual bones, a virtual implant, or a virtual skeletal model presented overlaid on the patient, the virtual skeletal model configured to move when the patient moves.
- Example 10 the subject matter of Examples 1-9 includes, wherein the augmented reality display presents one or more spinopelvic angles corresponding to the virtual feature.
- Example 11 is a method for using an augmented reality device during patient diagnosis, the method comprising: capturing, using an imaging device, a series of images of the patient; presenting using an augmented reality display, while permitting the patient to be viewed through the augmented reality display, a virtual feature corresponding to anatomy of the patient obtained from at least one image of the series of images; detecting, using processing circuitry, impingement of the anatomy of the patient based on a plurality of images from the series of images; and displaying on the augmented reality display, an indication of the impingement at the virtual feature by changing an attribute of the virtual feature.
- Example 12 the subject matter of Example 11 includes, creating a virtual implant, based on the at least one image, and outputting the virtual implant for display on the augmented reality display.
- Example 13 the subject matter of Examples 11-12 includes, controlling a second augmented reality device using the processing circuitry of the augmented reality device by outputting a command to control an aspect of operation of the second augmented reality device.
- Example 14 the subject matter of Example 13 includes, wherein the command causes the second augmented reality device to display fewer details in a display of the second augmented reality device than the presented in the augmented reality display of the augmented reality device.
- Example 15 the subject matter of Examples 13-14 includes, receiving an input adjusting, moving, or changing a virtual component displayed in the augmented reality display; and outputting a command to cause a corresponding virtual component displayed in the second augmented reality display of the second augmented reality device to be adjusted, moved, or changed accordingly.
- Example 16 the subject matter of Examples 11-15 includes, projecting, using the augmented reality display, a virtual plane to allow a surgeon to keep the anatomy of the patient within a corresponding plane while performing a range of motion test, and wherein the virtual plane is used for a biomechanical measurement used in a diagnostic exam.
- Example 17 the subject matter of Examples 11-16 includes, wherein the virtual feature includes at least one of a virtual bone or set of virtual bones, a virtual implant, or a virtual skeletal model presented overlaid on the patient, the virtual skeletal model configured to move when the patient moves.
- the virtual feature includes at least one of a virtual bone or set of virtual bones, a virtual implant, or a virtual skeletal model presented overlaid on the patient, the virtual skeletal model configured to move when the patient moves.
- Example 18 the subject matter of Examples 11-17 includes, determining, from a second plurality of images from the series of images, that the anatomy of the patient has moved; determining, in response to the anatomy of the patient moving, that the impingement has resolved; and in response to determining that the impingement has resolved, stop displaying the indication of the impingement at the virtual feature on the augmented reality display.
- Example 19 is at least one machine-readable medium including instructions for using an augmented reality device during patient diagnosis, which when executed by a machine, cause the machine to: capture, using an imaging device, a series of images of the patient; present using an augmented reality display, while permitting the patient to be viewed through the augmented reality display, a virtual feature corresponding to anatomy of the patient obtained from at least one image of the series of images; detect, using processing circuitry, impingement of the anatomy of the patient based on a plurality of images from the series of images; and display on the augmented reality display, an indication of the impingement at the virtual feature by changing an attribute of the virtual feature.
- Example 20 the subject matter of Example 19 includes, wherein the processing circuitry is further to: determine, from a second plurality of images from the series of images, that the anatomy of the patient has moved; in response to the anatomy of the patient moving, determine that the impingement has resolved; and wherein, in response to determining that the impingement has resolved, the augmented reality display is to stop displaying the indication of the impingement at the virtual feature.
- Example 21 is an augmented reality device for use during patient diagnosis, which may comprise: an imaging device to image the patient (e.g., in real time); an augmented reality display to present, while permitting the patient to be viewed through the augmented reality display, a virtual feature corresponding to anatomy of the patient obtained from the image; an impingement detection component (e.g., a processor to detect impingement from image data, a camera to detect impingement from one or more images, or a user interface to receive user input indicating impingement) to identify impingement at the virtual feature; and wherein the augmented reality display is further to present an indication of impingement at the virtual feature (e.g., by highlighting a portion of the virtual feature or changing a color of a portion of the virtual feature).
- an imaging device to image the patient (e.g., in real time)
- an augmented reality display to present, while permitting the patient to be viewed through the augmented reality display, a virtual feature corresponding to anatomy of the patient obtained from the image
- Example 22 the subject matter of Example 21 includes, a virtual surgical implant creation system to create an implant, for example based on the image, wherein the implant may be displayed on the augmented reality display as a virtual component.
- Example 23 the subject matter of Examples 21-22 includes, wherein the augmented reality device is in communication with a second augmented reality device, the augmented reality device to control aspects of operation of the second augmented reality device (e.g., what is displayed); wherein the augmented reality device may be a surgeon's device and the second augmented reality device may be a patient's device.
- Example 24 the subject matter of Example 23 includes, wherein the patient sees less details in a display of the second augmented reality device than the surgeon sees in the augmented reality display of the augmented reality device.
- Example 25 the subject matter of Examples 23-24 includes, wherein when the surgeon adjusts, moves, or changes a virtual component displayed in the augmented reality display, a corresponding virtual component displayed in a second augmented reality display of the second augmented reality device is adjusted, moved, or changed accordingly.
- Example 26 the subject matter of Examples 21-25 includes, wherein the augmented reality display is to project a plane to allow the anatomy of the patient to be kept within the plane while performing a range of motion test (e.g., rotation, flexion, etc.).
- a range of motion test e.g., rotation, flexion, etc.
- Example 27 the subject matter of Example 26 includes, wherein the plane is used for any biomechanical measurement that is used in a diagnostic exam (e.g., extension and rotation in the shoulder, a scoliosis diagnosis, or the like).
- a diagnostic exam e.g., extension and rotation in the shoulder, a scoliosis diagnosis, or the like.
- Example 28 the subject matter of Examples 21-27 includes, wherein the virtual feature is created using two or more two dimensional images, for example as obtained by the imaging device.
- Example 29 the subject matter of Examples 21-28 includes, wherein the virtual feature may include a bone or set of bones; wherein the virtual feature may include an implant; or wherein the virtual feature may include a skeletal model presented overlaid on the patient that is configured to move when the patient moves (e.g., linked).
- Example 30 the subject matter of Examples 21-29 includes, wherein the augmented reality display may present a plane or axis for orientation for performing a range of motion test on the patient.
- Example 31 the subject matter of Examples 21-30 includes, wherein the augmented reality display may present one or more Spinopelvic angles corresponding to the virtual feature.
- Example 32 is at least one machine-readable medium including instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations to implement of any of Examples 1-31.
- Example 33 is an apparatus comprising means to implement of any of Examples 1-31.
- Example 34 is a system to implement of any of Examples 1-31.
- Example 35 is a method to implement of any of Examples 1-31.
- Method examples described herein may be machine or computer-implemented at least in part. Some examples may include a computer-readable medium or machine-readable medium encoded with instructions operable to configure an electronic device to perform methods as described in the above examples.
- An implementation of such methods may include code, such as microcode, assembly language code, a higher-level language code, or the like. Such code may include computer readable instructions for performing various methods. The code may form portions of computer program products. Further, in an example, the code may be tangibly stored on one or more volatile, non-transitory, or non-volatile tangible computer-readable media, such as during execution or at other times.
- Examples of these tangible computer-readable media may include, but are not limited to, hard disks, removable magnetic disks, removable optical disks (e.g., compact disks and digital video disks), magnetic cassettes, memory cards or sticks, random access memories (RAMs), read only memories (ROMs), and the like.
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Surgery (AREA)
- Public Health (AREA)
- Medical Informatics (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Molecular Biology (AREA)
- Veterinary Medicine (AREA)
- Animal Behavior & Ethology (AREA)
- Heart & Thoracic Surgery (AREA)
- Robotics (AREA)
- Epidemiology (AREA)
- Pathology (AREA)
- Primary Health Care (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Radiology & Medical Imaging (AREA)
- Databases & Information Systems (AREA)
- Gynecology & Obstetrics (AREA)
- Data Mining & Analysis (AREA)
- Business, Economics & Management (AREA)
- General Business, Economics & Management (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Processing Or Creating Images (AREA)
Abstract
Description
- This application claims the benefit of priority to U.S. Provisional Application Ser. No. 62/470,819, filed Mar. 13, 2017, titled “Augmented Reality Diagnosis Guidance” and U.S. Provisional Application Ser. No. 62/470,690, filed Mar. 13, 2017, titled “Augmented Reality Surgical Technique Guidance” which are hereby incorporated herein by reference in their entirety.
- Diagnostics are used to evaluate a patient to determine whether the patient needs a surgical procedure, such as a total hip arthroplasty, ligament repair, knee replacement, or the like. These procedures are performed hundreds of thousands of times a year in the United States. Surgical advancements have allowed surgeons to use preoperative planning, display devices, and imaging, to improve diagnoses and surgical outcomes. While these advances have allowed for quicker and more successful diagnoses, these diagnostic techniques are highly technical and difficult for patients to understand.
- In the drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views. Like numerals having different letter suffixes may represent different instances of similar components. The drawings illustrate generally, by way of example, but not by way of limitation, various embodiments discussed in the present document.
-
FIG. 1 illustrates a clinician and a patient using augmented reality in accordance with some embodiments. -
FIGS. 2A-2F illustrate diagnostic augmented reality displays in accordance with some embodiments. -
FIG. 3 illustrates a clinician augmented reality display and a patient augmented reality display in accordance with some embodiments. -
FIG. 4 illustrates a display including virtual range of motion components in accordance with some embodiments. -
FIG. 5 illustrates a three dimensional rendering of a virtual component from two dimensional images in an augmented reality display in accordance with some embodiments. -
FIG. 6 illustrates a system for diagnosis using augmented reality in accordance with some embodiments. -
FIG. 7 illustrates a flow chart showing a technique for diagnosis using augmented reality in accordance with some embodiments. -
FIG. 8 illustrates generally an example of a block diagram of a machine upon which any one or more of the techniques discussed herein may perform in accordance with some embodiments. - Systems and methods for using an augmented reality device for diagnosis are described herein. The systems and methods herein describe uses for the augmented reality device, such as to display virtual components or representations of real objects overlaid on a real environment. An augmented reality (AR) device allows a user to view displayed virtual objects that appear to be projected into the real environment, which is also visible. AR devices typically include two display lenses or screens, including one for each eye of a user. Light is permitted to pass through the two display lenses such that aspects of the real environment are visible while also projecting light to make virtual elements visible to the user of the AR device.
- An AR device includes an AR display which may be used to display a virtual component while allowing real objects to be viewed. The virtual component may include measurements, angles, an identification of an impingement (e.g., a femoroacetabular impingement) or a potential impingement, a representation of an implant, a representation of patient anatomy, a range of motion plane or axis, or the like. For example, alignment angles may be shown virtually on an image of a patient or on a virtually presented skeleton, for example overlaid on the patient, using the AR display.
- Patient imaging may be performed (e.g., using a CT scan, an Mill, an X-ray, or the like) to create at least one patient image. In an example, two images may be used (e.g., two-dimensional images) to create a three-dimensional representation of an aspect of a patient's anatomy. The three-dimensional representation may be displayed virtually using an AR display.
- When displayed virtually, the three-dimensional representation may be manipulated or observed in the AR display (e.g., rotated, moved, zoomed in or out on, virtually cut, etc.).
- Two or more AR devices may be used in a coordinated manner, for example with a first AR device controlling one or more additional AR devices, or in a system with defined roles. For example, when activating an AR device, a user may select a role (e.g., clinician or patient) and the AR device may display information relevant to that role. For example, the AR display for a clinician may include more information or more complex or technical information than the AR display for a patient, which may be simplified for ease of understanding. In an example, the clinician AR device may control what is viewed on the patient AR device.
-
FIG. 1 illustrates aclinician 102 and apatient 108 using augmented reality in accordance with some embodiments. The clinician is wearing an augmented reality (AR)device 104 which may be used to display a virtual object to theclinician 102. The virtual object may not be visible to others if they are not wearing an AR device. Even if another person (e.g., the patient 108) is using an AR device (e.g., theAR device 110, which may have a camera 112), the person may not be able to see the virtual object. In another example, thepatient 108 may be able to see the virtual object in a shared augmented reality with theclinician 102, or may be able to see a modified version of the virtual object (e.g., according to customizations unique to theclinician 102 or the patient 108). Augmented reality is explained in more detail below. - Augmented reality is a technology for displaying virtual or “augmented” objects or visual effects overlaid on a real environment. The real environment may include a room or specific area, or may be more general to include the world at large. The virtual aspects overlaid on the real environment may be represented as anchored or in a set position relative to one or more aspects of the real environment. For example, a virtual object may be configured to appear to be resting on a table. An AR system may present virtual aspects that are fixed to a real object without regard to a perspective of a viewer or viewers of the AR system (e.g., the clinician 102). For example, the virtual object may exist in a room, visible to a viewer of the AR system within the room and not visible to a viewer of the AR system outside the room. The virtual object in the room may be displayed to the viewer outside the room when the viewer enters the room. In this example, the room may act as a real object that the virtual object is fixed to in the AR system.
- The
AR device 104 may include one or more screens, such as a single screen or two screens (e.g., one per eye of a user). The screens may allow light to pass through the screens such that aspects of the real environment are visible while displaying the virtual object. The virtual object may be made visible to theclinician 102 by projecting light. The virtual object may appear to have a degree of transparency or may be opaque (i.e., blocking aspects of the real environment). - An AR system may be viewable to one or more viewers, and may include differences among views available for the one or more viewers while retaining some aspects as universal among the views. For example, a heads-up display may change between two views while virtual objects may be fixed to a real object or area in both views. Aspects such as a color of an object, lighting, or other changes may be made among the views without changing a fixed position of at least one virtual object.
- A user may see the virtual object presented in an AR system as opaque or as including some level of transparency. In an example, the user may interact with the virtual object, such as by moving the virtual object from a first position to a second position. For example, the user may move an object with his or her hand. This may be done in the AR system virtually by determining that the hand has moved into a position coincident or adjacent to the object (e.g., using one or more cameras, which may be mounted on an AR device, such as
AR device camera 106 or separate, and which may be static or may be controlled to move), and causing the object to move in response. Virtual aspects may include virtual representations of real world objects or may include visual effects, such as lighting effects, etc. The AR system may include rules to govern the behavior of virtual objects, such as subjecting a virtual object to gravity or friction, or may include other predefined rules that defy real world physical constraints (e.g., floating objects, perpetual motion, etc.). AnAR device 104 may include acamera 106 on theAR device 104. TheAR device camera 106 may include an infrared camera, an infrared filter, a visible light filter, a plurality of cameras, a depth camera, etc. TheAR device 104 may project virtual items over a representation of a real environment, which may be viewed by a user. Theclinician 102 may control a virtual object using theAR device 104, a remote controller for theAR device 104, or by interacting with the virtual object (e.g., using a hand to “interact” with the virtual object or a gesture recognized by thecamera 106 of the AR device 104). - The
AR device 104 and theAR device 110 may be part of a shared AR system. The shared AR system may allow the AR devices to view the same or similar content, to be in communication with each other, for one to control the other, or the like. For example, theclinician 102 may control theAR device 110 of thepatient 108 using the clinician's own AR device 104 (e.g., using a remote, a gesture, a button on theAR device 104, or the like). In an example, theclinician 102 and thepatient 108 may select their respective roles when initiating theAR devices - By assigning roles to the AR devices, the AR devices may automatically provide appropriate virtual objects to be displayed. For example, the
clinician 102 may select the clinician role for theAR device 104. Once the clinician role is selected, theclinician 102 may be presented with a plurality of choices for display, such as diagnostic tools, viewing imaging of thepatient 108, selecting a patient file, education information, or the like. When theclinician 102 selects a choice, theAR device 104 may display virtual objects or information corresponding to the choice. In an example, theAR device 110 may automatically display virtual objects or information (which may correspond to the choice, to the clinician-displayed virtual objects or information, or may be derived from the choice or the clinician-displayed virtual objects or information) for view by thepatient 108. For example, theclinician 102 may choose to view an image of a bone of thepatient 108. The image may be previously obtained. TheAR device 104 may display, in response to the choice, a three-dimensional view of the bone, and for example, additional clinical information (e.g., technical data). TheAR device 110 may automatically display, in response to the choice a three-dimensional view of the bone without the additional clinical information or with basic educational information (e.g., labels for names of parts of the bone). In an example, theclinician 102 may manipulate the virtual representation of the bone (e.g., using a button or gesture to rotate or zoom the bone) and the changes to the bone may be automatically reflected in theAR device 110 of thepatient 108. - The
AR device 104 of theclinician 102 may automatically identify a bony impingement (e.g., using thecamera 106 or by analyzing an image). A virtual representation of the impingement may be displayed (e.g., as a red virtually-highlighted area) using theAR device 104 or theAR device 110, for example on an image, presented on the patient 108 (e.g., with a three-dimensional skeletal overlay on the patient 108), or the like. The virtual representation of the impingement may move as thepatient 108 moves, and may disappear or appear to fade away if the impingement is (temporarily) removed (e.g., if it is temporarily not present when thepatient 108 sits or stands). -
FIGS. 2A-2F illustrate diagnostic augmented reality displays in accordance with some embodiments.FIG. 2A illustrates afirst view 202A of a patient without augmented or virtual components and asecond view 202B of the patient with a virtual skeleton overlaid on the patient. The virtual skeleton may be based on imaging of the patient. For example, the skeleton may be created by using two images (e.g., X-rays) of the patient to create a three-dimensional model of the patient's skeleton, which may be displayed virtually using an AR display as shown in thesecond view 202B. In another example, the skeleton may be determined in real time from an image of the patient using a camera of an AR device that is then used to display the skeleton, based on a generic or template skeleton model scaled or deformed to fit the image of the patient taken by the AR device. - For example, one or more landmarks on the virtual skeleton may be associated with bony landmarks of the patient (such as at the knee joint). The landmarks may be identified and tracked by the AR device and used to align the virtual skeleton with the real-life moving patient anatomy. In an example, the one or more landmarks may be used to align and scale the virtual skeleton to the patient (e.g., a foot landmark, a head landmark, a hip landmark, a hand landmark, etc.). The landmarks may be identified using a registration device, such as a pointer or finger within an AR view. For example, a clinician may point to or touch each identified landmark to register the landmark to the patient. After registration, the virtual skeleton may be applied to the patient in the AR view. For example, a generic or template skeleton may be overlaid on the patient. In another example, a skeleton based on the two images (e.g., X-rays) may be used as a virtual skeleton. A clinician may toggle between the views using an AR device.
-
FIG. 2B illustrates anAR display 203 of a patient. TheAR display 203 may include technical information displayed virtually, such as height, weight, BMI, co-morbidities (e.g., hypertension, cardiovascular disease, peripheral vascular disease, congestive heart failure, renal function impairment, diabetes, respiratory disease, etc.) for a patient being viewed through theAR display 203. TheAR display 203 may include patient information (e.g., name, age, allergies, potential patient-specific complications, etc.), readiness information, time or counters, or the like. In an example, theAR display 203 may include virtually displayed diagnosis type information, such as information from a patient file, reported pain, any previously identified impingement, referral notes (e.g., from a general practitioner), family medical history information (e.g., parental impingement procedures or co-morbidities), or the like. In an example, the clinician may interact with the virtually displayed diagnosis type information, for example by zooming in or out, selecting the virtually displayed diagnosis type information (e.g., if additional information is available, such as a diagnosis recommendation), or the like. The displayed information may be accessed or changed in real time, including updating the information for display in theAR display 203. In an example, a clinician using theAR display 203 may change the information displayed, such as by using a gesture or input device, or may save the information or a snapshot of theAR display 203, such as to a patient file. -
FIG. 2C illustrates an AR display 205 including a virtual skeleton overlaid on a patient. The AR display 205 may include aspects described above in relation toAR display 203. The virtual skeleton overlaid may be generated from an image of the patient or using a camera of an AR device including the AR display 205. The virtual skeleton may be virtually coupled to the patient, for example if the patient moves, then the skeleton may move accordingly. -
FIG. 2D illustrates an AR display presenting afirst view 204A of a patient image and asecond view 204B of the patient image. Thefirst view 204A includes a virtually highlightedarea 212 of an image or of a virtual representation of anatomy of a patient. The image may be obtained using x-ray, MRI, CT scan, or the like. The virtual representation may be a representation of an obtained image or may be a three-dimensional representation that may be created from, for example, two or more images used to create a three-dimensional model that may then be displayed virtually. The virtually highlightedarea 212 may correspond with an impingement. The impingement may be identified from an image of anatomy of the patient or form virtually projected anatomy of the patient. For example, the impingement may be automatically identified and the virtually highlightedarea 212 may be projected based on the automatic identification. The patient may report pain in an area, which may be evaluated for impingement. - In an example, a clinician may virtually view the anatomy (e.g., at least part of a skeleton) and identify an impingement or a potential impingement. In response to the clinician identifying the impingement or the potential impingement, the AR display may highlight the impingement or the potential impingement (e.g., in red) virtually (e.g., display the virtually highlighted area 212). The virtually displayed anatomy may be manipulated by the clinician (e.g., rotated, zoomed, moved, etc.). The manipulation may be performed using a gesture, a button (e.g., on a remote or on the AR device), or the like. The manipulation may allow the clinician to better identify the impingement. In an example, the clinician may manipulate a portion of the virtually displayed anatomy while leaving other portions of the virtually displayed anatomy static (or move other portions in a different direction relative to the first portion). For example, the clinician may cause a virtually displayed knee to bend or hip to swivel or shoulder to rotate. By viewing the movement of a bone in the virtually displayed anatomy relative to another bone, impingement may be detected by the clinician or automatically by the AR device (e.g., by identifying overlap in the virtually displayed anatomy). The impingement may then be displayed as the virtually highlighted
area 212. - The
second view 204B may include one or morevirtual angles 206 or virtual lines used to indicate, for example, potential acetabular component position for a surgical intervention, an implant alignment, anatomy (e.g., bone, muscle, ligament, other soft tissue, or the like) or position, orientation, or location of anatomy, or may include Spinopelvic angles, described in more detail below with respect toFIG. 2E . Thevirtual angles 206 may be automatically detected and projected onto the image or virtual representation, for example using the highlighted portion as an origin location. -
FIG. 2E illustrates anAR display 207 including one or morevirtual angles 208. Thevirtual angles 208, referred to herein as Spinopelvic angles, are displayed virtually in theAR display 207. The Spinopelvic angles may include a pelvic incidence (PI), a sacral tilt (ST), a pelvic femoral angle (PFA), an anteinclination (AI) of the pre-operative acetabulum, and a sacral acetabular angle (SAA). The PI and SAA may be static numbers that are the same standing and sitting. The Spinopelvic angles are described in more detail in “Spinopelvic mobility and acetabular component position for total hip arthroplasty,” Bone Joint J 2017; 99-B (1 Supple A):37-45, Stefl, et. al, which is hereby incorporated herein by reference in its entirety. -
FIG. 2F illustrates afirst AR display 210A and asecond AR display 210B showing virtual components displayed in a real environment. Thefirst AR display 210A includes a virtual component (e.g., an impingement or potential impingement) highlighted on a virtual skeleton of a patient walking on a treadmill. Thesecond AR display 210B includes a virtual component (e.g., an impingement or potential impingement) highlighted on a virtual skeleton of a patient sitting on a chair. The virtual skeleton may be virtually coupled to the patient, for example if the patient moves, then the skeleton may move accordingly. Thus thefirst AR display 210A may allow a clinician to observe skeletal (or other anatomical aspects which may be displayed virtually) action when the patient walks, for example to examine range of motion, potential impingement, or other clinical analysis. Thesecond AR display 210B may allow a clinician to make observations or an examination of the patient in a seated position. The impingement or potential impingement (the highlighted virtual component) of AR displays 210A or 210B may be detected automatically (e.g., using an AR device or a camera) or may be identified by a clinician. When detected automatically, the highlighting may be automatically applied, and when identified by the clinician, the highlighting may be applied in response to receiving input from the clinician indicating the identification. -
FIG. 3 illustrates a clinician augmentedreality display 300 and a patient augmentedreality display 301 in accordance with some embodiments. Theclinician AR display 300 includes virtually displayed information, and example virtually displayed components or augmented images. For example,bones trial 304 may be displayed virtually on thebone 302. Agap 308 may be displayed for the clinician on theclinician AR display 300. Theclinician AR display 300 may include avirtual representation 310 of thepatient AR display 301. Thepatient AR display 301 includes a simplified version of the view available in theclinician AR display 300. For example, thebones patient AR display 301 without additional comorbidity information, virtually displayed information, the implant ortrial 304, thegap 308, etc. - The
patient AR display 301 may be controlled by theclinician AR display 300. For example, the clinician may select aspects of theclinician AR display 300 to be shared with thepatient AR display 301. In an example, the clinician may, for example drag aspects of theclinician AR display 300 onto thevirtual representation 310 of the patient view. Dragged aspects may then be displayed on thepatient AR display 301. In another example, the clinician may select a program or procedure which may then automatically display virtual components in theclinician AR display 300 or in thepatient AR display 301. -
FIG. 4 illustrates an AR display 400 including virtual range of motion components in accordance with some embodiments. The AR display 400 includes a femur 402 and a tibia 404 for illustration purposes of a knee replacement, but other bones may be displayed (e.g., for a shoulder replacement or hip replacement). The AR display 400 may include one or more planes or axes, such as planes 406-410. The planes 406-410 may be used for range of motion tests. For example, plane 408 may be displayed for extension or flexion range of motion testing. The plane 408 may be displayed virtually for the range of motion test to be conducted while keeping the knee within the plane 408. Thus the planes 406-410 may be displayed virtually to guide a clinician in conducting a range of motion test. -
FIG. 5 illustrates a three dimensional rendering of a virtual component from two dimensional images in an augmented reality display in accordance with some embodiments. TheAR display 502A includes two images that may be taken of anatomy of a patient, for example using an x-ray, an MRI, a CT scan, or the like. The images may be two-dimensional. TheAR display 502A may not include virtual components. TheAR display 502B may illustrate an animation to create a three-dimensional virtual representation of the patient anatomy (e.g., a bone) from the images. TheAR display 502B illustrates a transition stage in the animation between the two-dimensional images displayed in theAR display 502A and the three-dimensional virtual representation illustrated inAR display 502C. The AR displays 502A-502C may represent three stages of an animation to create the three-dimensional virtual representation from the two-dimensional images. TheAR display 502C illustrates the three-dimensional virtual representation of the patient anatomy. The three-dimensional virtual representation may be interacted with by a clinician viewing theAR display 502C, for example using a button, a remote, a gesture, etc. The interaction may manipulate the three-dimensional virtual representation, for example rotate, move, zoom, etc., the three-dimensional virtual representation. In an example, the virtual representation may be created using a 3D modeler running on a computer, such as that described in U.S. Publication Number 2013/0191099, titled “Process for generating a computer-accessible medium including information on the functioning of a joint,” which is hereby incorporated by reference herein, in its entirety. By manipulating the three-dimensional virtual representation, the clinician may detect impingement or potential impingement. In response to receiving an indication from the clinician that impingement or potential impingement has been detected, theAR display 502C may illustrate, highlight, or otherwise display the impingement or potential impingement virtually. For example, the impingement or potential impingement may be highlighted in red. In another example, an impingement or potential impingement may be automatically detected, such as by virtually moving bones and detecting whether impingement has occurred. For example, two bones may be displayed and moved relative to one another, and a camera may be used to detect if there is overlap between the two bones. In yet another example, the bones may be automatically moved for the clinician to detect impingement or potential impingement. -
FIG. 6 illustrates asystem 600 for diagnosis using augmented reality in accordance with some embodiments. Thesystem 600 may be used to perform the technique 700 described in relation toFIG. 7 below, for example, by using theprocessor 602. The system includes anaugmented reality device 601 that may be in communication with adatabase 616. Theaugmented reality device 601 includes aprocessor 602,memory 604, anAR display 608, and acamera 606. Theaugmented reality device 601 may include asensor 610, aspeaker 612, or ahaptic controller 614. Thedatabase 616 may includeimage storage 618 orpreoperative plan storage 620. In an example, theaugmented reality device 601 may be a HoloLens manufactured by Microsoft of Redmond, Wash. - The
processor 602 of theaugmented reality device 601 includes anaugmented reality modeler 603. Theaugmented reality modeler 603 may be used by theprocessor 602 to create the augmented reality environment. For example, theaugmented reality modeler 603 may receive dimensions of a room, such as from thecamera 606 orsensor 610, and create the augmented reality environment to fit within the physical structure of the room. In another example, physical objects may be present in the room and theaugmented reality modeler 603 may use the physical objects to present virtual objects in the augmented reality environment. For example, theaugmented reality modeler 603 may use or detect a table present in the room and present a virtual object as resting on the table. TheAR display 608 may display the AR environment overlaid on a real environment. Thedisplay 608 may show a virtual object, using theAR device 601, such as in a fixed position in the AR environment. - The
augmented reality device 601 may include asensor 610, such as an infrared sensor. Thecamera 606 or thesensor 610 may be used to detect movement, such as a gesture by a surgeon or other user, that may be interpreted by theprocessor 602 as attempted or intended interaction by the user with the virtual target. Theprocessor 602 may identify an object in a real environment, such as through processing information received using thecamera 606. - The
AR display 608, for example during a surgical procedure, may present, such as within a surgical field while permitting the surgical field to be viewed through the augmented reality display, a virtual feature corresponding to a physical feature hidden by an anatomical aspect of a patient. The virtual feature may have a virtual position or orientation corresponding to a first physical position or orientation of the physical feature. In an example, the virtual position or orientation of the virtual feature may include an offset from the first physical position or orientation of the physical feature. The offset may include a predetermined distance from the augmented reality display, a relative distance from the augmented reality display to the anatomical aspect, or the like. TheAR device 601 may include a virtual surgicalimplant creation system 615 to create an implant, for example based on an image. The implant may be displayed on the augmented reality display as a virtual component. -
FIG. 7 illustrates a flow chart showing a technique 700 for diagnosis using augmented reality in accordance with some embodiments. The technique 700 includes anoperation 702 to obtain an image of a patient, for example by using an imaging device (e.g., in real time). The virtual feature may be created using two or more two dimensional images, for example as obtained by the imaging device. - The technique 700 includes an
operation 704 to present, while permitting the patient to be viewed through an AR display, a virtual feature corresponding to anatomy of the patient obtained from the image. The virtual feature may include a bone or a set of bones, an implant, a skeletal model, for example presented overlaid on the patient, or the like. The skeleton may be configured to move when the patient moves (e.g., the virtual skeleton may be visually linked to the real anatomy of the patient). - The technique 700 may include an
operation 706 to detect impingement at the virtual feature (e.g., of the anatomy of the patient). For example, the virtual feature may correspond to a bone of the patient and may be displayed virtually with other virtual bones. The bone may interact with one of the other virtual bones, indicating impingement. The impingement may be detected automatically, such as using a camera (e.g., by visually detecting the interaction using one or more images) or using a processor (e.g., by detecting the interaction from imaging data). In an example, the impingement may be identified by a clinician. The technique 700 may include receiving input from the clinician indicating the impingement. When the impingement is detected or the clinician input is received, the virtual feature or a portion of the virtual feature may be changed (e.g., highlighted, pulsed, flashed, changed color, etc.). - In an example, the technique 700 may include an operation to determine, such as from a plurality of images (which may be captured by the imaging device described above), that the anatomy of the patient has moved. In response to determining that the anatomy has moved, this operation may include determining that the impingement has resolved (e.g., there is no impingement in this orientation/configuration of the patient's anatomy, the anatomy is unimpinged, etc.), and in response to this determination, stop or pause display of an indication of the impingement at the virtual feature. In an example, instead of or in addition to stopping or pausing display of the indication, the operation may include showing the indication faded or offset from the virtual feature (e.g., to suggest that the impingement exists in the prior orientation/configuration but is not present in a current orientation/configuration).
- The technique 700 may include an operation to present a plane or axis for orientation for performing a range of motion test on the patient. The technique 700 may include an operation to present one or more Spinopelvic angles corresponding to the virtual feature. The technique 700 may include an operation to create an implant, for example based on the image. In an example, the implant may be displayed on the augmented reality display as a virtual component.
- The AR device may be used to communicate with a second AR device. The AR device may be used to control aspects of operation of the second AR device (e.g., what is displayed). In an example, the AR device may be a surgeon's device and the second AR device may be a patient's device. In an example, the patient sees less details in a display of the second AR device than the surgeon sees in the AR display of the AR device. In an example, when the surgeon adjusts, moves, or changes a virtual component displayed in the AR display, a corresponding virtual component displayed in a second AR display of the second AR device may be adjusted, moved, or changed accordingly. In an example, the technique 700 may include projecting a plane to allow the anatomy of the patient to be kept within the plane while performing a range of motion test (e.g., rotation, flexion, etc.). The plane may be used for any biomechanical measurement that is used in a diagnostic exam (e.g., extension or rotation in the shoulder, a scoliosis diagnosis, or the like).
- In an example, the technique 700 may include outputting an indication of impingement (e.g., of the anatomy of the patient, at the virtual feature, etc.), such as for display on the augmented reality display. The indication of the impingement may be displayed by the augmented reality display by changing an attribute of the virtual feature. For example, the virtual feature may be highlighted, encircled, flash, change color, be augmented by one or more additional features (e.g., highlight, circle, color change, etc.), or the like. In an example, the virtual feature may be a hip or patient anatomy in an area of a hip. In this example, the indication of the impingement may include a red circle around the hip or the area, overlaying a color on the hip or the area (i.e., making the hip or the area appear to be a different color), flashing a light at or near the hip or the area, or the like.
-
FIG. 8 illustrates generally an example of a block diagram of amachine 800 upon which any one or more of the techniques (e.g., methodologies) discussed herein may perform in accordance with some embodiments. In alternative embodiments, themachine 800 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, themachine 800 may operate in the capacity of a server machine, a client machine, or both in server-client network environments. Themachine 800 may be a personal computer (PC), a tablet PC, a personal digital assistant (PDA), a mobile telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations. - Examples, as described herein, may include, or may operate on, logic or a number of components, modules, or like mechanisms. Such mechanisms are tangible entities (e.g., hardware) capable of performing specified operations when operating. In an example, the hardware may be specifically configured to carry out a specific operation (e.g., hardwired). In an example, the hardware may include configurable execution units (e.g., transistors, circuits, etc.) and a computer readable medium containing instructions, where the instructions configure the execution units to carry out a specific operation when in operation. The configuring may occur under the direction of the executions units or a loading mechanism. Accordingly, the execution units are communicatively coupled to the computer readable medium when the device is operating. For example, under operation, the execution units may be configured by a first set of instructions to implement a first set of features at one point in time and reconfigured by a second set of instructions to implement a second set of features.
- Machine (e.g., computer system) 800 may include a hardware processor 802 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a
main memory 804 and astatic memory 806, some or all of which may communicate with each other via an interlink (e.g., bus) 808. Themachine 800 may further include adisplay unit 810, an alphanumeric input device 812 (e.g., a keyboard), and a user interface (UI) navigation device 814 (e.g., a mouse). In an example, thedisplay unit 810,alphanumeric input device 812 andUI navigation device 814 may be a touch screen display. Thedisplay unit 810 may include goggles, glasses, or other AR or VR display components. For example, the display unit may be worn on a head of a user and may provide a heads-up-display to the user. Thealphanumeric input device 812 may include a virtual keyboard (e.g., a keyboard displayed virtually in a VR or AR setting. - The
machine 800 may additionally include a storage device (e.g., drive unit) 816, a signal generation device 818 (e.g., a speaker), anetwork interface device 820, and one ormore sensors 821, such as a global positioning system (GPS) sensor, compass, accelerometer, or other sensor. Themachine 800 may include anoutput controller 828, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices. - The
storage device 816 may include a machinereadable medium 822 that is non-transitory on which is stored one or more sets of data structures or instructions 824 (e.g., software) embodying or utilized by any one or more of the techniques or functions described herein. Theinstructions 824 may also reside, completely or at least partially, within themain memory 804, withinstatic memory 806, or within thehardware processor 802 during execution thereof by themachine 800. In an example, one or any combination of thehardware processor 802, themain memory 804, thestatic memory 806, or thestorage device 816 may constitute machine readable media. - While the machine
readable medium 822 is illustrated as a single medium, the term “machine readable medium” may include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) configured to store the one ormore instructions 824. - The term “machine readable medium” may include any medium that is capable of storing, encoding, or carrying instructions for execution by the
machine 800 and that cause themachine 800 to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions. Non-limiting machine readable medium examples may include solid-state memories, and optical and magnetic media. Specific examples of machine readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. - The
instructions 824 may further be transmitted or received over acommunications network 826 using a transmission medium via thenetwork interface device 820 utilizing any one of a number of transfer protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), mobile telephone networks (e.g., cellular networks), Plain Old Telephone (POTS) networks, and wireless data networks (e.g., Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards known as Wi-Fi®, as the personal area network family of standards known as Bluetooth® that are promulgated by the Bluetooth Special Interest Group, peer-to-peer (P2P) networks, among others. In an example, thenetwork interface device 820 may include one or more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or one or more antennas to connect to thecommunications network 826. In an example, thenetwork interface device 820 may include a plurality of antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. The term “transmission medium” shall be taken to include any intangible medium that is capable of storing, encoding or carrying instructions for execution by themachine 800, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software. - Each of these non-limiting examples may stand on its own, or may be combined in various permutations or combinations with one or more of the other examples.
- Example 1 is an augmented reality device for use during patient diagnosis, comprising: an imaging device to capture a series of images of the patient; an augmented reality display to present, while permitting the patient to be viewed through the augmented reality display, a virtual feature corresponding to anatomy of the patient obtained from at least one image of the series of images; processing circuitry to: detect impingement of the anatomy of the patient based on a plurality of images from the series of images; and output, for display on the augmented reality display, an indication of the impingement at the virtual feature by changing an attribute of the virtual feature.
- In Example 2, the subject matter of Example 1 includes, wherein the processing circuitry is further to create a virtual implant, based on the at least one image, and output the virtual implant for display on the augmented reality display.
- In Example 3, the subject matter of Examples 1-2 includes, wherein the augmented reality device is in communication with a second augmented reality device, and wherein processing circuitry of the augmented reality device is further to output a command to control an aspect of operation of the second augmented reality device.
- In Example 4, the subject matter of Example 3 includes, wherein the command causes the second augmented reality device to display fewer details in a display of the second augmented reality device than the presented in the augmented reality display of the augmented reality device.
- In Example 5, the subject matter of Examples 3-4 includes, wherein the processing circuitry is further to: receive an input adjusting, moving, or changing a virtual component displayed in the augmented reality display; and output a command to cause a corresponding virtual component displayed in the second augmented reality display of the second augmented reality device to be adjusted, moved, or changed accordingly.
- In Example 6, the subject matter of Examples 1-5 includes, wherein the augmented reality display is to project a virtual plane to allow a surgeon to keep the anatomy of the patient within a corresponding plane while performing a range of motion test, wherein the virtual plane is used for a biomechanical measurement used in a diagnostic exam.
- In Example 7, the subject matter of Examples 1-6 includes, wherein the processing circuitry is further to receive an input manipulating the virtual feature.
- In Example 8, the subject matter of Examples 1-7 includes, wherein the processing circuitry is further to: determine, from a second plurality of images from the series of images, that the anatomy of the patient has moved; and in response to the anatomy of the patient moving, determine that the impingement has resolved; and wherein, in response to determining that the impingement has resolved, the augmented reality display is to stop displaying the indication of the impingement at the virtual feature.
- In Example 9, the subject matter of Examples 1-8 includes, wherein the virtual feature includes at least one of a virtual bone or set of virtual bones, a virtual implant, or a virtual skeletal model presented overlaid on the patient, the virtual skeletal model configured to move when the patient moves.
- In Example 10, the subject matter of Examples 1-9 includes, wherein the augmented reality display presents one or more spinopelvic angles corresponding to the virtual feature.
- Example 11 is a method for using an augmented reality device during patient diagnosis, the method comprising: capturing, using an imaging device, a series of images of the patient; presenting using an augmented reality display, while permitting the patient to be viewed through the augmented reality display, a virtual feature corresponding to anatomy of the patient obtained from at least one image of the series of images; detecting, using processing circuitry, impingement of the anatomy of the patient based on a plurality of images from the series of images; and displaying on the augmented reality display, an indication of the impingement at the virtual feature by changing an attribute of the virtual feature.
- In Example 12, the subject matter of Example 11 includes, creating a virtual implant, based on the at least one image, and outputting the virtual implant for display on the augmented reality display.
- In Example 13, the subject matter of Examples 11-12 includes, controlling a second augmented reality device using the processing circuitry of the augmented reality device by outputting a command to control an aspect of operation of the second augmented reality device.
- In Example 14, the subject matter of Example 13 includes, wherein the command causes the second augmented reality device to display fewer details in a display of the second augmented reality device than the presented in the augmented reality display of the augmented reality device.
- In Example 15, the subject matter of Examples 13-14 includes, receiving an input adjusting, moving, or changing a virtual component displayed in the augmented reality display; and outputting a command to cause a corresponding virtual component displayed in the second augmented reality display of the second augmented reality device to be adjusted, moved, or changed accordingly.
- In Example 16, the subject matter of Examples 11-15 includes, projecting, using the augmented reality display, a virtual plane to allow a surgeon to keep the anatomy of the patient within a corresponding plane while performing a range of motion test, and wherein the virtual plane is used for a biomechanical measurement used in a diagnostic exam.
- In Example 17, the subject matter of Examples 11-16 includes, wherein the virtual feature includes at least one of a virtual bone or set of virtual bones, a virtual implant, or a virtual skeletal model presented overlaid on the patient, the virtual skeletal model configured to move when the patient moves.
- In Example 18, the subject matter of Examples 11-17 includes, determining, from a second plurality of images from the series of images, that the anatomy of the patient has moved; determining, in response to the anatomy of the patient moving, that the impingement has resolved; and in response to determining that the impingement has resolved, stop displaying the indication of the impingement at the virtual feature on the augmented reality display.
- Example 19 is at least one machine-readable medium including instructions for using an augmented reality device during patient diagnosis, which when executed by a machine, cause the machine to: capture, using an imaging device, a series of images of the patient; present using an augmented reality display, while permitting the patient to be viewed through the augmented reality display, a virtual feature corresponding to anatomy of the patient obtained from at least one image of the series of images; detect, using processing circuitry, impingement of the anatomy of the patient based on a plurality of images from the series of images; and display on the augmented reality display, an indication of the impingement at the virtual feature by changing an attribute of the virtual feature.
- In Example 20, the subject matter of Example 19 includes, wherein the processing circuitry is further to: determine, from a second plurality of images from the series of images, that the anatomy of the patient has moved; in response to the anatomy of the patient moving, determine that the impingement has resolved; and wherein, in response to determining that the impingement has resolved, the augmented reality display is to stop displaying the indication of the impingement at the virtual feature.
- Example 21 is an augmented reality device for use during patient diagnosis, which may comprise: an imaging device to image the patient (e.g., in real time); an augmented reality display to present, while permitting the patient to be viewed through the augmented reality display, a virtual feature corresponding to anatomy of the patient obtained from the image; an impingement detection component (e.g., a processor to detect impingement from image data, a camera to detect impingement from one or more images, or a user interface to receive user input indicating impingement) to identify impingement at the virtual feature; and wherein the augmented reality display is further to present an indication of impingement at the virtual feature (e.g., by highlighting a portion of the virtual feature or changing a color of a portion of the virtual feature).
- In Example 22, the subject matter of Example 21 includes, a virtual surgical implant creation system to create an implant, for example based on the image, wherein the implant may be displayed on the augmented reality display as a virtual component.
- In Example 23, the subject matter of Examples 21-22 includes, wherein the augmented reality device is in communication with a second augmented reality device, the augmented reality device to control aspects of operation of the second augmented reality device (e.g., what is displayed); wherein the augmented reality device may be a surgeon's device and the second augmented reality device may be a patient's device.
- In Example 24, the subject matter of Example 23 includes, wherein the patient sees less details in a display of the second augmented reality device than the surgeon sees in the augmented reality display of the augmented reality device.
- In Example 25, the subject matter of Examples 23-24 includes, wherein when the surgeon adjusts, moves, or changes a virtual component displayed in the augmented reality display, a corresponding virtual component displayed in a second augmented reality display of the second augmented reality device is adjusted, moved, or changed accordingly.
- In Example 26, the subject matter of Examples 21-25 includes, wherein the augmented reality display is to project a plane to allow the anatomy of the patient to be kept within the plane while performing a range of motion test (e.g., rotation, flexion, etc.).
- In Example 27, the subject matter of Example 26 includes, wherein the plane is used for any biomechanical measurement that is used in a diagnostic exam (e.g., extension and rotation in the shoulder, a scoliosis diagnosis, or the like).
- In Example 28, the subject matter of Examples 21-27 includes, wherein the virtual feature is created using two or more two dimensional images, for example as obtained by the imaging device.
- In Example 29, the subject matter of Examples 21-28 includes, wherein the virtual feature may include a bone or set of bones; wherein the virtual feature may include an implant; or wherein the virtual feature may include a skeletal model presented overlaid on the patient that is configured to move when the patient moves (e.g., linked).
- In Example 30, the subject matter of Examples 21-29 includes, wherein the augmented reality display may present a plane or axis for orientation for performing a range of motion test on the patient.
- In Example 31, the subject matter of Examples 21-30 includes, wherein the augmented reality display may present one or more Spinopelvic angles corresponding to the virtual feature.
- Example 32 is at least one machine-readable medium including instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations to implement of any of Examples 1-31.
- Example 33 is an apparatus comprising means to implement of any of Examples 1-31.
- Example 34 is a system to implement of any of Examples 1-31.
- Example 35 is a method to implement of any of Examples 1-31.
- Method examples described herein may be machine or computer-implemented at least in part. Some examples may include a computer-readable medium or machine-readable medium encoded with instructions operable to configure an electronic device to perform methods as described in the above examples. An implementation of such methods may include code, such as microcode, assembly language code, a higher-level language code, or the like. Such code may include computer readable instructions for performing various methods. The code may form portions of computer program products. Further, in an example, the code may be tangibly stored on one or more volatile, non-transitory, or non-volatile tangible computer-readable media, such as during execution or at other times. Examples of these tangible computer-readable media may include, but are not limited to, hard disks, removable magnetic disks, removable optical disks (e.g., compact disks and digital video disks), magnetic cassettes, memory cards or sticks, random access memories (RAMs), read only memories (ROMs), and the like.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/919,139 US10575905B2 (en) | 2017-03-13 | 2018-03-12 | Augmented reality diagnosis guidance |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201762470690P | 2017-03-13 | 2017-03-13 | |
US201762470819P | 2017-03-13 | 2017-03-13 | |
US15/919,139 US10575905B2 (en) | 2017-03-13 | 2018-03-12 | Augmented reality diagnosis guidance |
Publications (2)
Publication Number | Publication Date |
---|---|
US20180256258A1 true US20180256258A1 (en) | 2018-09-13 |
US10575905B2 US10575905B2 (en) | 2020-03-03 |
Family
ID=61952963
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/919,139 Active US10575905B2 (en) | 2017-03-13 | 2018-03-12 | Augmented reality diagnosis guidance |
Country Status (5)
Country | Link |
---|---|
US (1) | US10575905B2 (en) |
EP (1) | EP3596658A1 (en) |
AU (1) | AU2018236172B2 (en) |
CA (1) | CA3055941C (en) |
WO (1) | WO2018169891A1 (en) |
Cited By (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180082480A1 (en) * | 2016-09-16 | 2018-03-22 | John R. White | Augmented reality surgical technique guidance |
US20190385367A1 (en) * | 2018-06-14 | 2019-12-19 | Robert Labron | Virtual reality software system and method |
US10726626B2 (en) * | 2017-11-22 | 2020-07-28 | Google Llc | Interaction between a viewer and an object in an augmented reality environment |
US10849692B2 (en) * | 2016-06-17 | 2020-12-01 | Zimmer, Inc. | System and method for intraoperative surgical planning |
US11205306B2 (en) * | 2019-05-21 | 2021-12-21 | At&T Intellectual Property I, L.P. | Augmented reality medical diagnostic projection |
US20220020219A1 (en) * | 2020-07-15 | 2022-01-20 | Orthosoft Ulc | Augmented reality bone landmark display |
US11432877B2 (en) | 2017-08-02 | 2022-09-06 | Medtech S.A. | Surgical field camera system that only uses images from cameras with an unobstructed sight line for tracking |
WO2022232250A1 (en) * | 2021-04-27 | 2022-11-03 | Zimmer Us, Inc. | Augmented reality patient assessment module |
WO2023019299A1 (en) * | 2021-08-17 | 2023-02-23 | The EduPod Pty Ltd | A sole and method of determining attributes thereof |
US11941174B1 (en) * | 2019-02-19 | 2024-03-26 | Meta Platforms Technologies, Llc | Finger pinch detection |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6138566B2 (en) * | 2013-04-24 | 2017-05-31 | 川崎重工業株式会社 | Component mounting work support system and component mounting method |
WO2018169891A1 (en) | 2017-03-13 | 2018-09-20 | Zimmer, Inc. | Augmented reality diagnosis guidance |
EP3810012A1 (en) | 2018-06-19 | 2021-04-28 | Tornier, Inc. | Mixed reality-aided education using virtual models or virtual representations for orthopedic surgical procedures |
CA3191645A1 (en) * | 2020-09-04 | 2022-03-10 | George GRAMMATOPOULOS | Hip arthroplasty planning method |
Family Cites Families (62)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5610811A (en) | 1992-11-09 | 1997-03-11 | Niti-On Medical Supply Co., Ltd. | Surgical instrument file system |
US6701174B1 (en) | 2000-04-07 | 2004-03-02 | Carnegie Mellon University | Computer-aided bone distraction |
FR2816200A1 (en) * | 2000-11-06 | 2002-05-10 | Praxim | DETERMINING THE POSITION OF A KNEE PROSTHESIS |
DE10063089C1 (en) | 2000-12-18 | 2002-07-25 | Siemens Ag | User-controlled linking of information within an augmented reality system |
US6919867B2 (en) | 2001-03-29 | 2005-07-19 | Siemens Corporate Research, Inc. | Method and apparatus for augmented reality visualization |
US7383073B1 (en) | 2001-10-16 | 2008-06-03 | Z-Kat Inc. | Digital minimally invasive surgery system |
US8010180B2 (en) | 2002-03-06 | 2011-08-30 | Mako Surgical Corp. | Haptic guidance system and method |
US20040044295A1 (en) | 2002-08-19 | 2004-03-04 | Orthosoft Inc. | Graphical user interface for computer-assisted surgery |
CA2460119A1 (en) | 2004-03-04 | 2005-09-04 | Orthosoft Inc. | Graphical user interface for computer-assisted surgery |
US7118029B2 (en) | 2004-08-27 | 2006-10-10 | Sdgi Holdings, Inc. | Smart instrument tray RFID reader |
US7896869B2 (en) | 2004-12-29 | 2011-03-01 | Depuy Products, Inc. | System and method for ensuring proper medical instrument use in an operating room |
US9492240B2 (en) | 2009-06-16 | 2016-11-15 | Intuitive Surgical Operations, Inc. | Virtual measurement tool for minimally invasive surgery |
US8100133B2 (en) | 2005-06-30 | 2012-01-24 | Intuitive Surgical Operations | Indicator for tool state and communication in multi-arm robotic telesurgery and method of use |
US20070179626A1 (en) | 2005-11-30 | 2007-08-02 | De La Barrera Jose L M | Functional joint arthroplasty method |
US7949386B2 (en) * | 2006-03-21 | 2011-05-24 | A2 Surgical | Computer-aided osteoplasty surgery system |
US10008017B2 (en) | 2006-06-29 | 2018-06-26 | Intuitive Surgical Operations, Inc. | Rendering tool information as graphic overlays on displayed images of tools |
US9718190B2 (en) | 2006-06-29 | 2017-08-01 | Intuitive Surgical Operations, Inc. | Tool position and identification indicator displayed in a boundary area of a computer display screen |
US8600478B2 (en) | 2007-02-19 | 2013-12-03 | Medtronic Navigation, Inc. | Automatic identification of instruments used with a surgical navigation system |
US20090017430A1 (en) | 2007-05-15 | 2009-01-15 | Stryker Trauma Gmbh | Virtual surgical training tool |
US20100159434A1 (en) | 2007-10-11 | 2010-06-24 | Samsun Lampotang | Mixed Simulator and Uses Thereof |
US8491664B2 (en) * | 2008-02-25 | 2013-07-23 | Smith & Nephew, Inc. | System of orienting femoral head for acetabular prosthesis alignment |
US9248000B2 (en) | 2008-08-15 | 2016-02-02 | Stryker European Holdings I, Llc | System for and method of visualizing an interior of body |
US8884618B2 (en) | 2009-02-25 | 2014-11-11 | Zimmer, Inc. | Method of generating a patient-specific bone shell |
US8662900B2 (en) | 2009-06-04 | 2014-03-04 | Zimmer Dental Inc. | Dental implant surgical training simulation system |
US8860760B2 (en) | 2010-09-25 | 2014-10-14 | Teledyne Scientific & Imaging, Llc | Augmented reality (AR) system and method for tracking parts and visually cueing a user to identify and locate parts in a scene |
US9847044B1 (en) | 2011-01-03 | 2017-12-19 | Smith & Nephew Orthopaedics Ag | Surgical implement training process |
US20130191099A1 (en) * | 2012-01-19 | 2013-07-25 | Peter Roelof Krekel | Process for generating a computer-accessible medium including information on the functioning of a joint |
NL2008437C2 (en) | 2012-01-19 | 2013-07-22 | Clinical Graphics B V | Process to generate a computer-accessible medium comprising information on the functioning of a joint. |
US20140188240A1 (en) * | 2012-02-07 | 2014-07-03 | Conformis, Inc. | Methods and devices related to patient-adapted hip joint implants |
US20130267838A1 (en) | 2012-04-09 | 2013-10-10 | Board Of Regents, The University Of Texas System | Augmented Reality System for Use in Medical Procedures |
KR20140136517A (en) | 2012-05-02 | 2014-11-28 | 엠파이어 테크놀로지 디벨롭먼트 엘엘씨 | Four dimensional image registration using dynamical model for augmented reality in medical applications |
US20140081659A1 (en) | 2012-09-17 | 2014-03-20 | Depuy Orthopaedics, Inc. | Systems and methods for surgical and interventional planning, support, post-operative follow-up, and functional recovery tracking |
KR20140112207A (en) | 2013-03-13 | 2014-09-23 | 삼성전자주식회사 | Augmented reality imaging display system and surgical robot system comprising the same |
US20140272863A1 (en) | 2013-03-15 | 2014-09-18 | Peter Kim | User Interface For Virtual Reality Surgical Training Simulator |
EP3022672A2 (en) | 2013-07-16 | 2016-05-25 | Seiko Epson Corporation | Information processing apparatus, information processing method, and information processing system |
JP6188477B2 (en) | 2013-08-02 | 2017-08-30 | オリンパス株式会社 | Image processing apparatus, image processing method, and program |
CA2925299C (en) | 2013-09-25 | 2022-07-26 | Zimmer, Inc. | Patient specific instrumentation (psi) for orthopedic surgery and systems and methods for using x-rays to produce same |
US20150366628A1 (en) | 2014-06-18 | 2015-12-24 | Covidien Lp | Augmented surgical reality environment system |
US9665960B1 (en) | 2014-12-22 | 2017-05-30 | Amazon Technologies, Inc. | Image-based item location identification |
US10154239B2 (en) | 2014-12-30 | 2018-12-11 | Onpoint Medical, Inc. | Image-guided surgery with surface reconstruction and augmented reality visualization |
US9538962B1 (en) | 2014-12-31 | 2017-01-10 | Verily Life Sciences Llc | Heads-up displays for augmented reality network in a medical environment |
US10235807B2 (en) | 2015-01-20 | 2019-03-19 | Microsoft Technology Licensing, Llc | Building holographic content using holographic tools |
US10013808B2 (en) | 2015-02-03 | 2018-07-03 | Globus Medical, Inc. | Surgeon head-mounted display apparatuses |
US10908681B2 (en) * | 2015-02-20 | 2021-02-02 | Covidien Lp | Operating room and surgical site awareness |
US10271917B2 (en) | 2015-02-27 | 2019-04-30 | Flex Operating Room, Llc | Surgical tray efficiency system and related methods |
CN107847289A (en) | 2015-03-01 | 2018-03-27 | 阿里斯医疗诊断公司 | The morphology operation of reality enhancing |
US20160324580A1 (en) | 2015-03-23 | 2016-11-10 | Justin Esterberg | Systems and methods for assisted surgical navigation |
USRE49930E1 (en) | 2015-03-26 | 2024-04-23 | Universidade De Coimbra | Methods and systems for computer-aided surgery using intra-operative video acquired by a free moving camera |
US9861446B2 (en) * | 2016-03-12 | 2018-01-09 | Philipp K. Lang | Devices and methods for surgery |
US20170312032A1 (en) | 2016-04-27 | 2017-11-02 | Arthrology Consulting, Llc | Method for augmenting a surgical field with virtual guidance content |
US10194990B2 (en) | 2016-04-27 | 2019-02-05 | Arthrology Consulting, Llc | Method for augmenting a surgical field with virtual guidance content |
US20170337402A1 (en) | 2016-05-18 | 2017-11-23 | Hand Held Products, Inc. | Tool verification systems and methods for a workflow process |
US10582971B2 (en) * | 2016-06-16 | 2020-03-10 | Medtech S.A. | Robotized system for femoroacetabular impingement resurfacing |
WO2018052966A1 (en) | 2016-09-16 | 2018-03-22 | Zimmer, Inc. | Augmented reality surgical technique guidance |
US20180090029A1 (en) | 2016-09-29 | 2018-03-29 | Simbionix Ltd. | Method and system for medical simulation in an operating room in a virtual reality or augmented reality environment |
WO2018083687A1 (en) | 2016-10-07 | 2018-05-11 | Simbionix Ltd | Method and system for rendering a medical simulation in an operating room in virtual reality or augmented reality environment |
EP3534847B1 (en) * | 2016-11-02 | 2020-09-09 | Zimmer, Inc. | Device for sensing implant location and impingement |
US9836654B1 (en) | 2017-02-28 | 2017-12-05 | Kinosis Ltd. | Surgical tracking and procedural map analysis tool |
EP4278956A3 (en) * | 2017-03-10 | 2024-02-21 | Biomet Manufacturing, LLC | Augmented reality supported knee surgery |
WO2018169891A1 (en) | 2017-03-13 | 2018-09-20 | Zimmer, Inc. | Augmented reality diagnosis guidance |
US9892564B1 (en) | 2017-03-30 | 2018-02-13 | Novarad Corporation | Augmenting real-time views of a patient with three-dimensional data |
US11432877B2 (en) | 2017-08-02 | 2022-09-06 | Medtech S.A. | Surgical field camera system that only uses images from cameras with an unobstructed sight line for tracking |
-
2018
- 2018-03-12 WO PCT/US2018/022074 patent/WO2018169891A1/en unknown
- 2018-03-12 EP EP18717148.3A patent/EP3596658A1/en active Pending
- 2018-03-12 CA CA3055941A patent/CA3055941C/en active Active
- 2018-03-12 US US15/919,139 patent/US10575905B2/en active Active
- 2018-03-12 AU AU2018236172A patent/AU2018236172B2/en active Active
Cited By (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11490965B2 (en) | 2016-06-17 | 2022-11-08 | Zimmer, Inc. | System and method for intraoperative surgical planning |
US10849692B2 (en) * | 2016-06-17 | 2020-12-01 | Zimmer, Inc. | System and method for intraoperative surgical planning |
US20180082480A1 (en) * | 2016-09-16 | 2018-03-22 | John R. White | Augmented reality surgical technique guidance |
US11432877B2 (en) | 2017-08-02 | 2022-09-06 | Medtech S.A. | Surgical field camera system that only uses images from cameras with an unobstructed sight line for tracking |
US10726626B2 (en) * | 2017-11-22 | 2020-07-28 | Google Llc | Interaction between a viewer and an object in an augmented reality environment |
US11263819B2 (en) | 2017-11-22 | 2022-03-01 | Google Llc | Interaction between a viewer and an object in an augmented reality environment |
US20190385367A1 (en) * | 2018-06-14 | 2019-12-19 | Robert Labron | Virtual reality software system and method |
US10885709B2 (en) * | 2018-06-14 | 2021-01-05 | Robert Labron | Virtual reality software system and method for treating medical condition in user |
US11941174B1 (en) * | 2019-02-19 | 2024-03-26 | Meta Platforms Technologies, Llc | Finger pinch detection |
US11205306B2 (en) * | 2019-05-21 | 2021-12-21 | At&T Intellectual Property I, L.P. | Augmented reality medical diagnostic projection |
US20220020219A1 (en) * | 2020-07-15 | 2022-01-20 | Orthosoft Ulc | Augmented reality bone landmark display |
WO2022232250A1 (en) * | 2021-04-27 | 2022-11-03 | Zimmer Us, Inc. | Augmented reality patient assessment module |
WO2023019299A1 (en) * | 2021-08-17 | 2023-02-23 | The EduPod Pty Ltd | A sole and method of determining attributes thereof |
Also Published As
Publication number | Publication date |
---|---|
WO2018169891A1 (en) | 2018-09-20 |
CA3055941A1 (en) | 2018-09-20 |
US10575905B2 (en) | 2020-03-03 |
CA3055941C (en) | 2023-06-20 |
AU2018236172A8 (en) | 2019-10-24 |
EP3596658A1 (en) | 2020-01-22 |
AU2018236172A1 (en) | 2019-10-03 |
AU2018236172B2 (en) | 2021-03-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10575905B2 (en) | Augmented reality diagnosis guidance | |
US20230017128A1 (en) | Surgical field camera system that only uses images from cameras with an unobstructed sight line for tracking | |
US11490965B2 (en) | System and method for intraoperative surgical planning | |
US11275249B2 (en) | Augmented visualization during surgery | |
US11903654B2 (en) | Augmented reality supported knee surgery | |
US10359916B2 (en) | Virtual object display device, method, program, and system | |
US20180082480A1 (en) | Augmented reality surgical technique guidance | |
JP6463038B2 (en) | Image alignment apparatus, method and program | |
CN109069208A (en) | Ultra wide band for wireless ultrasound Tracking And Communications positions | |
US20100312143A1 (en) | Human body measurement system and information provision method using the same | |
JP2021128794A (en) | Stereoscopic image generation system, stereoscopic image generation method, and stereoscopic image generation program | |
CN112533556A (en) | System method and computer program product for computer-assisted surgery | |
Izard et al. | NextMed: How to enhance 3D radiological images with Augmented and Virtual Reality | |
WO2021100513A1 (en) | Radiation imaging system, radiation imaging method, image processing device, and program | |
US20170228877A1 (en) | Device and method for image registration, and a nontransitory recording medium | |
US20240212292A1 (en) | Augmented reality patient assessment module | |
JP6392190B2 (en) | Image registration device, method of operating image registration device, and program | |
Inácio et al. | Augmented Reality in Surgery: A New Approach to Enhance the Surgeon's Experience |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: ZIMMER, INC., INDIANA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:NASH, SETH ANDERSON;WHITE, JOHN R.;GALLOWAY, ANNELISE;SIGNING DATES FROM 20180327 TO 20180515;REEL/FRAME:047186/0157 |
|
AS | Assignment |
Owner name: ZIMMER, INC., INDIANA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:CLAYPOOL, JODY L.;REEL/FRAME:047387/0768 Effective date: 20170321 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: AWAITING TC RESP., ISSUE FEE NOT PAID |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |