US20160353055A1 - Simulated transparent display with augmented reality for remote collaboration - Google Patents
Simulated transparent display with augmented reality for remote collaboration Download PDFInfo
- Publication number
- US20160353055A1 US20160353055A1 US15/167,011 US201615167011A US2016353055A1 US 20160353055 A1 US20160353055 A1 US 20160353055A1 US 201615167011 A US201615167011 A US 201615167011A US 2016353055 A1 US2016353055 A1 US 2016353055A1
- Authority
- US
- United States
- Prior art keywords
- image
- annotation
- computing device
- display
- current image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000003190 augmentative effect Effects 0.000 title abstract description 13
- 238000000034 method Methods 0.000 claims abstract description 47
- 238000004891 communication Methods 0.000 claims description 21
- 238000013507 mapping Methods 0.000 claims description 16
- 238000009877 rendering Methods 0.000 claims description 5
- 238000005259 measurement Methods 0.000 claims 2
- 230000015654 memory Effects 0.000 description 13
- 230000003993 interaction Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 238000004873 anchoring Methods 0.000 description 6
- 230000002708 enhancing effect Effects 0.000 description 5
- 238000000605 extraction Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000001356 surgical procedure Methods 0.000 description 5
- 238000013459 approach Methods 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 239000008186 active pharmaceutical agent Substances 0.000 description 2
- 238000003491 array Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 230000001934 delay Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000002559 palpation Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- ZCYVEMRRCGMTRW-UHFFFAOYSA-N 7553-56-2 Chemical compound [I] ZCYVEMRRCGMTRW-UHFFFAOYSA-N 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000002874 hemostatic agent Substances 0.000 description 1
- 229910052740 iodine Inorganic materials 0.000 description 1
- 239000011630 iodine Substances 0.000 description 1
- 238000002357 laparoscopic surgery Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/141—Systems for two-way working between two video terminals, e.g. videophone
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
-
- G06F17/241—
-
- G06F19/3425—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/169—Annotation, e.g. comment data or footnotes
-
- G06K9/6215—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G06T7/0042—
-
- G06T7/0051—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/107—Static hand or arm
- G06V40/113—Recognition of static hand signs
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H80/00—ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/366—Image reproducers using viewer tracking
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B2090/364—Correlation of different images or relation of image positions in respect to the body
- A61B2090/365—Correlation of different images or relation of image positions in respect to the body augmented reality, i.e. correlating a live optical image with another image
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B2090/364—Correlation of different images or relation of image positions in respect to the body
- A61B2090/368—Correlation of different images or relation of image positions in respect to the body changing the image on a display according to the operator's position
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
- A61B2090/371—Surgical systems with images on a monitor during operation with simultaneous use of two cameras
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
- A61B2090/373—Surgical systems with images on a monitor during operation using light, e.g. by using optical scanners
Definitions
- the present disclosure generally relates to displays, and in particular to systems and methods for enhancing remote collaboration using simulated transparent displays with augmented reality.
- Remote collaboration and remote instruction have a number of useful applications such as telementoring.
- telementoring is surgical telementoring, which has the potential to abstract away the geographic distance between a patient in need of expert surgical care and the surgeon with the required expertise. For example, if a patient urgently needing a complex procedure for which a rural hospital does not have a specialist, telementoring could enable the rural surgeon to perform the procedure under the guidance of a remote expert, without the delays associated with transporting the patient to a major surgical center. If a surgeon were deployed to a location where its operating base has limited resources, the surgeon could provide urgent specialized surgical care with the help of an expert surgeon remotely located, possibly thousands of miles away. Further, if an innovative surgical technique were available but not yet widely adopted, a surgeon could disseminate the novel procedure through telementoring.
- a remote mentor might annotate a video feed of a surgical operation using a telestrator.
- the annotated video is sent back to the operating room where it is displayed on a nearby monitor, then a local surgeon performing the operation needs to shift focus frequently between the operating field and the nearby monitor to acquire and apply the instructions from the remote mentor.
- the local surgeon first has to parse and understand the instructions on the monitor, memorize the instructions, and finally after shifting his focus back to the surgery, the local surgeon has to, temporally and spatially, project those instructions into the real-world context of the surgery.
- FIG. 1 is a graphical representation of annotated photographs of components of an example environment of a telementoring system in accordance with at least one aspect of the present disclosure.
- FIG. 2 is a graphical representation of an example view of a simulated transparent display with augmented reality by the trainee as used in FIG. 1 in accordance with at least one aspect of the present disclosure.
- FIG. 3 is an example block diagram of an embodiment of an enhanced remote collaboration system, e.g., using a simulated transparent display with augmented reality, in accordance with at least one aspect of the present disclosure.
- FIG. 4 is a graphical representation of an example annotation authoring interface as displayed on a remote mentor system in accordance with at least one aspect of the present disclosure.
- FIG. 5 is an example table of annotation categories in accordance with at least one aspect of the present disclosure.
- FIG. 6 is a graphical representation of an example diagram of the annotation orientation in the remote mentor system using a two-touch interaction orientation tool in accordance with at least one aspect of the present disclosure.
- FIG. 7 is an example diagram of the annotation scaling in the remote mentor system sing a two-touch interaction scaling tool in accordance with at least one aspect of the present disclosure.
- FIG. 8 is an example flowchart 800 of a process for providing anchored annotations over an image for enhancing remote collaboration utilizing a simulated transparent display with augmented reality in accordance with at least one aspect of the present disclosure.
- FIG. 9 is an example reference frame preprocessing algorithm in accordance with at least one aspect of the present disclosure.
- FIG. 10 is an example annotation anchoring algorithm in accordance with at least one aspect of the present disclosure.
- FIG. 11 is an example keypoint matching algorithm in accordance with at least one aspect of the present disclosure.
- FIG. 12 is an example parameter extraction algorithm in accordance with at least one aspect of the present disclosure.
- FIG. 13 is an example setup for a simulated transparent display in accordance with at least one aspect of the present disclosure.
- Example embodiments in accordance with the present disclosure includes telementoring which can, e.g., remotely provide enhanced instructions, e.g., surgical telementoring, which can be used to demonstrate an apparatus, method and system.
- the annotations are superimposed directly onto an image of a view field where telementoring or telecorroboration is desired such as a surgery area, using a simulated transparent display with augmented reality (“AR”).
- AR augmented reality
- Telemedicine and telementoring applications rely on effective communication of medical expertise. AR has potential of enhancing telementoring either as an interface or as an environment.
- a virtualized interface may allow for more intuitive interaction between an operating surgeon and relevant medical information provided.
- the live video which is the image of the view field, may be displayed to the telementoring surgeon in conjunction with a view of the operating room. Additional viewpoints may provide greater visual context to both trainee and mentor.
- any function, component, device, module, system and alike herein described may not be limited to a physical component but may also be realized with a set of executable instructions stored in a non-transitory memory device, e.g., a computer-readable memory, to be executed by one or more processors, or be realized with a combination of one or more physical components with such instructions.
- a non-transitory memory device e.g., a computer-readable memory
- Computer-readable media described herein include computer storage media and/or communication media.
- Computer storage media includes tangible storage units such as volatile memory, nonvolatile memory, and/or other persistent and/or auxiliary computer storage media, removable and non-removable computer storage media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data.
- Computer storage media includes tangible or physical forms of media included in a device or hardware component that is pan of a device or external to a device, including but not limited to random access memory (RAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), phase change memory (PRAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, compact disc read-only memory (CD-ROM), digital versatile disks (DVDs), optical cards or other optical storage media, magnetic cassettes, magnetic tape, magnetic disk storage, magnetic cards or other magnetic storage devices or media, solid-state memory devices, storage arrays, network attached storage, storage area networks, hosted computer storage or memories, storage, devices, and/or storage media that can be used to store and maintain information for access by a computing device 102 or 104 .
- RAM random access memory
- SRAM static random-access memory
- DRAM dynamic random-access memory
- PRAM phase change memory
- ROM read-only memory
- communication media can embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, or other transmission mechanism.
- a modulated data signal such as a carrier wave, or other transmission mechanism.
- computer storage media does not include communication media.
- FIG. 1 is an example telementoring, system environment 100 in accordance with at least one aspect of the present disclosure.
- the local surgeon, or a trainee, 104 sees the view field 106 as the annotated w field 108 on a trainee system 110 , shown as a simulated transparent display device, suspended in his field of view.
- the trainee system 110 is connected to a remote mentor system 112 located at a remote mentor site 114 via a communication network 116 .
- a remote mentor 118 sees an image 120 of the view field 106 , and enters an annotation 122 to the remote mentor system 112 .
- Information related to the annotation 122 is communicated to the trainee system 110 via the communication network 116 , and an appropriate annotation is displayed on the trainee system 110 in real, or near real, time.
- FIG. 2 is an example view 200 by the trainee 104 of the view field 106 and the surrounding areas in accordance with at least one aspect of the present disclosure.
- the trainee's view includes the trainee system 110 showing a live image of the view field 106 under the trainee system 110 except for the area where the annotation 202 created by a remote mentor is displayed.
- the annotation 202 indicates the precise placement of an additional surgical clamp 204 along with the instructions 206 to “Add Clamp.”
- the simulated transparent display image of the view field 106 displayed on the trainee system 110 represents, or simulate, a direct view of the view field 106 seen by the trainee 104 , allowing the trainee 104 to see his hands 208 , 210 , the surgical instruments 212 , 214 , 216 , and the actual view field 106 under the trainee system 110 as if seeing through the trainee system 110 .
- the part of the view field 106 seen by the trainee 104 through the trainee system 110 is continued with the surrounding region of the view field 106 that the trainee 104 sees directly.
- the annotation 202 remains anchored to the view field elements for which they were defined even as the trainee system 110 is repositioned, as the head position of the trainee 104 changes, and/or as the view field 106 changes over time.
- the simulated transparent display with augmented reality approach has the potential to bypass the shortcomings of the conventional telestrator-based approach by integrating annotations into the view field, allowing the local surgeon to benefit from the annotations without shifting focus.
- the alignment between the displayed image and the peripheral regions of the view field preserves the natural hand-eye coordination on which surgeons rely.
- the annotations are anchored to the view field elements and remain anchored to the elements as the viewpoint and/or view field change causing the elements to shift or move.
- the anchoring of the annotations to the elements reduces the need for the remote mentor to draw annotations that have drifted out of place, improving the continuity of the visual guidance provided to the trainee.
- the functions and utilities of the simulated transparent display 110 at the trainee site may be accomplished by using a conventional tablet that is capable of displaying the live image, or video stream, acquired by its back-facing camera.
- the live image may be sent wirelessly to the mentor site where it is displayed on the mentor's tablet.
- the mentor may add graphical and textual annotations to a frame of the video stream.
- the annotations are sent back to the trainee site where they are overlaid on the trainee's tablet display to provide guidance to the trainee.
- FIG. 3 is an example block diagram 300 of an embodiment of an enhanced remote collaboration system using a simulated transparent display with augmented reality in accordance with at least one aspect of the present disclosure.
- the trainee system 310 or other local computing device shown as the display device 110 in FIGS. 1 and 2 , may be implemented using a tablet. Although a tablet is used for example purposes, other types of a computing device equipped with a display and a camera, which may or may not be integrated in the computing device, may be utilized.
- a processor, or processors, 312 of the trainee system 310 may be coupled to various components and modules in the tablet 310 and enables functions associated with those components and modules.
- the modules can additionally or alternatively represent computer program instructions operable by the processor(s) 312 to perform the desired functions, or any combination of modules coupled to or operable by the processor(s) 312 to perform the desired functions.
- a camera 314 of the trainee system 310 acquires a video stream of the view field 106 , and displays each acquired frame on the display 316 of the trainee system 310 .
- a transceiver 318 may wirelessly send each acquired frame through a communication network 320 to a remote mentor system 330 or other remote computing device to be processed for annotation anchoring.
- the communication network 320 may be any kind of communication network, wired or wireless, capable of communicating required data between the trainee system 310 and the remote mentor system 330 , such as cellular, Wi-Fi, Bluetooth, Ethernet communication networks and alike.
- the remote mentor system 330 which may be another computing device, comprises a processor, or processors, 332 which may be coupled to various components and modules of the remote mentor system 330 and enables functions associated with those components and modules.
- a transceiver 334 receives the video stream comprising of continuously updated frames via the communication network 320 , and each frame is sequentially displayed on a display 336 which may be a touchscreen.
- the display 336 at the remote mentor's site may also be a large screen device to project the life size, or enlarged size, of the view field.
- the remote mentor system 330 can include at least one input device 370 , integral or peripheral to system 330 .
- the at least one input device 370 can be user-operable.
- Examples of input devices 370 can include, e.g., a keyboard, keypad, a mouse, a trackball, a pen sensor or smart pen, a light pen or light gun, a game controller such as a joystick or game pad, a voice input device such as a microphone, voice-recognition device, or speech-recognition device, a touch input device, a gestural input device such as a touchscreen, a grip sensor, an accelerometer, another haptic input, a visual input device such as one or more cameras or image sensors, and the like.
- Input device 370 can be arranged with respect to, or operably coupled with, display 336 to provide a user interface via which the mentor can view images and provide annotations.
- input device 370 can be a touch sensor overlaid on or otherwise associated with display 336 to form a touchscreen.
- An annotation module 338 may present various annotation choices to the mentor via a user interface.
- the mentor may select a reference frame from the video stream and define selected annotations to the reference frame, e.g., by providing inputs via input device 370 of the user interface.
- the annotation module 338 generates and displays the selected annotation, or annotations, over the reference frame displayed on the display 336 .
- a feature detector module 340 in the remote mentor system 330 may then detect image features in the reference frame in the neighborhood of the annotations.
- the image features are also referred to herein as “features” or “salient features,” though that term does not imply any particular standard for determining salience, and may be automatically detected in the reference frame based on a predefined description such as pixel intensities.
- a descriptor extractor module 342 may then compute a respective signature, e.g., a unique signature, for each detected feature.
- the descriptor in some examples, is a bit string that describes the pixel intensities at each pixel in an image patch surrounding a feature point. This allows comparing the descriptors from the reference frame to descriptors of future frames.
- the reference frame data comprising the annotations, reference frame features, and associated descriptors, are then sent to the trainee system 310 via the communication network 320 .
- a feature detector 344 detects image features of a current frame of the video stream being displayed on the display 316 , and the features are enhanced with descriptors extracted by a descriptor extractor 346 .
- a descriptor match module 348 matches the extracted current frame's descriptors with the corresponding reference frame's descriptors received from the remote mentor system 330 where the annotations were defined. Because the trainee's environment may change due to conditions such as the trainee system 310 being repositioned, the view field geometry changing, or the view field becoming partially occluded due to the surgeon's hands or newly added instruments, the comparison and matching of the reference and current frame features ensure that the annotations are placed and anchored to the intended areas of the view field defined by the mentor annotations.
- a homography relates any two images of the same planar surface in space and can be used to relate the reference image to the current image.
- a mapping module 350 may derive a homography for each annotation, and by using the derived homographies, transforms the annotations from the reference frame to the current frame. Techniques other than homography may be utilized to relate the reference and current images.
- An annotation rendering module 352 then renders the transformed annotations superimposed over the current frame displayed on the display 316 . The annotation appears anchored to the view field.
- the trainee system 310 may comprise a depth acquisition module 354 to obtain accurate geometry of the view field 106 to supplement the information obtained by the camera 314 .
- the geometry information may be transmitted to the remote mentor's system 330 along with the reference image from the trainee's system 310 to enhance the reference frame data.
- the feature detector module 340 of the remote mentor's system 330 may detect at least some of image features in the reference image relative to the annotation based, at least in part, on the geometry information. For example, features may be detected at corners or other points, such as edges, peaks, and valleys, with a high-magnitude spatial derivative of depth information.
- the depth acquisition module 354 may comprise a depth acquisition device such as a depth camera, a motion detector, an infrared depth sensor, or any other device or a combination of devices that are capable of acquiring accurate information of the geometry of the view field 106 .
- the depth acquisition module 354 may be integrated in the trainee system 310 .
- the trainee system 310 may additionally comprise a user tracking module 356 to adjust and re-project the images based on the trainee's view point for a better simulation of transparency.
- the tracking module 356 may include a plurality of user-facing cameras which may be used to triangulate the trainee's head position.
- the tracking module 356 may also be, or include, a depth acquisition device similar to the ones described above for the depth acquisition module 354 .
- the tracking module 356 may be integrated in the trainee system 310 , or may be an external module connected to the trainee system 310 . As the trainee's view point changes, missing color samples, due to the limitations of the field of view and/or occlusion changes as the view-point changes from that of the video camera to that of the trainee, may be corrected by filling in the missing color samples with samples from older frames.
- the trainee system and the remote mentor system may also include speaker-and-microphone sets, shown as set 358 for the trainee system and set 360 for the remote mentor system, to allow audio communications between the trainee 104 and the remote mentor 118 .
- the remote mentor system 330 sends to the trainee system 310 only the type of annotations and their position in the reference frame. This compact encoding of annotations saves bandwidth and is sufficient to recreate the annotations at the trainee system based on a local copy of the set of sprites. In other examples, the remote mentor system 330 sends the types and positions of annotations, and also other information.
- Applications of the enhanced remote collaboration system 300 is not limited to a surgical setting, and may be utilized in any remote instruction settings where interactions between a local operator, who is performing an act following instructions, and a remote instructor, who is providing the instructions and viewing the act as seen by the operator, are desirable.
- the ability to mark or annotate an object at the local operator's site and to follow the object and the annotation as seen by the operator is especially beneficial for hands-on training and instruction activities where the operator's act is dynamic and his view point may shift as the act progresses.
- Such training and instruction activities may include medical diagnostics, mechanical and/or electrical diagnostics and repairs, navigation where a route to a destination is annotated and updated, and other situations.
- the role of the remote instructor may also be realized with a computer program or artificial intelligence (AI) interacting with the local operator.
- AI artificial intelligence
- the remote mentor's system 330 may be virtualized at a location accessible by the trainee's system 310 including memory in the trainee's system storing the instructions, when executed by the processor(s) 312 , to realize the AI.
- FIG. 4 is an example block diagram 400 of an annotation module 338 of FIG. 3 as displayed on a remote mentor system in accordance with at least one aspect of the present disclosure.
- the annotation module 338 is shown as a user interface for a tablet with a touchscreen.
- the annotation module 338 may provide multiple selectable symbols such as icon-labeled buttons 402 on the display 336 of the remote mentor system 330 .
- the selectable symbols may be selected by a mouse click, voice command, touch, or any other compatible use input mechanisms, e.g., input device 370 , FIG. 3 .
- the remote mentor 118 may tap to select a particular button of icon-labeled buttons 402 .
- Each icon-labeled button may be associated with a predefined function and may be organized into various annotation categories.
- FIG. 5 is an example table of annotation categories 500 in accordance with at least one aspect of the present disclosure.
- the categories may include drawing shapes 502 , surgical tools 504 , text labels 506 , and hand gestures 508 .
- the predefined text labels in the text labels 506 may include, e.g., “close,” “incision,” “palpation,” “remove,” and “stitch.”
- the types of drawing shapes in the drawing shapes 502 may include: points, lines, and polygons. Each shape may be defined with one or multiple points.
- the mentor may draw on a touchscreen or tablet, draw by dragging with a mouse, or otherwise provide path information via input device 370 .
- the path information can include information of, e.g., at least one of location, pressure, direction, or duration of a user input.
- the mentor can indicate an incision by drawing a series of points, and the path information can include the locations of those points.
- the mentor can indicate a stitch by drawing a line or curve, and the path information can include, e.g., the locations of Bezier control points in a decomposition of the drawn curve into at least one Bezier-curve segment.
- the mentor can indicate palpation by drawing a closed figure, such as a conic section or polygon, and the path information. These specific shapes are not limiting. For example, an incision can be indicated by a curve and a stitch can be indicated by a series of points.
- the mentor may also select and drag a certain button to indicate a desired operation.
- the types of surgical tools the surgical tools 504 may include bag-valve-mask (BVM), endotracheal tube (ET), hemostat, iodine swab, longhook, retractor, scalpel, scissors, stethoscope, surgical tape, syringe, tweezers, and other surgical instruments.
- BVM bag-valve-mask
- ET endotracheal tube
- hemostat iodine swab
- longhook retractor
- scalpel scalpel
- scissors stethoscope
- surgical tape surgical tape
- syringe syringe
- tweezers tweezers
- the hand gesture annotations 508 illustrate typical manual actions performed by the surgeon and may include actions such as palpating, pointing, and stretching.
- Surgical tools, text labels, and hand gesture icons may be positioned based on a reference point, for example, the tip of the scalpel's blade. These selectable icons are displayed on the display 336 as an image with transparent background.
- FIG. 6 is an example diagram 600 of the annotation orientation in the remote mentor system 330 using a two-touch interaction orientation tool in accordance with the present disclosure.
- the annotation 602 may be positioned using a single-touch drag and drop interaction, and may be orientated using a two-touch interaction: one touch for defining the center of rotation 604 and the other touch for dragging motion for defining the rotation angle, shown as the starting point 606 and the ending point 608 .
- the upper portion 610 shows the starting position of the annotation 602 , from center of rotation 604 to starting point 606
- the lower portion 612 shows the ending position of the annotation 602 , from center of rotation 604 to ending point 608 .
- FIG. 7 is an example diagram 700 of the annotation scaling using a two-touch interaction scaling tool in accordance with the present disclosure.
- Scaling of the annotation 702 may be performed by using a two finger pinch-and-zoom interaction: one finger for defining the fixed end of scaling 704 and the other finger for defining the starting point 706 and the ending point 708 .
- the upper portion 708 shows the starting position of the annotation 702 , from fixed end of scaling 704 to starting point 706
- the lower portion 710 shows the ending position of the annotation 702 , from fixed end of scaling 704 to ending point 708 .
- FIG. 8 is an example flowchart 800 of a process in accordance with at least one aspect of the present disclosure.
- the process is a process for providing anchored annotations over art image, e.g., for enhancing remote collaboration utilizing a simulated transparent display with augmented reality.
- a trainee's system 816 acquires a reference image and geometry information of operating view field and d splays it on the trainee's system as described above with reference to the camera 314 and the depth acquisition module 354 .
- the trainee's system 816 transmits the reference image and the geometry information, as described above with reference to the transceiver 318 , to the remote mentor's system via a communication network, as described above with reference to the communication network 320 .
- the communication network may be any kind of communication network, wired or wireless, capable of communicating required data between the trainee system and the remote mentor system, such as cellular, Bluetooth, Ethernet communication networks and the like.
- the mentor's system 818 receives the reference image and geometry information and displays the reference image on the remote mentor's system as described above with reference to the transceiver 334 and the display 336 .
- the remote mentor's system 818 generates an annotation and displays over the reference image on the remote mentor's system as described above with reference to the annotation module 338 .
- the annotation may be received via a user interface such as a selectable icon with a predefined annotation as described in FIGS. 4 and 5 .
- the remote mentor's system 818 generates reference frame data based on the reference image, the geometry information, and the annotation as described above with reference to the feature detector module 340 and the descriptor extractor 342 .
- the reference frame data may be generated from detecting a plurality of salient features in the reference image relative to the annotation and computing a descriptor for each of the plurality of the salient features.
- the remote mentor's system 818 transmits the reference frame data and annotation to the trainee's system via the communication network as described above with reference to the transceiver 334 .
- the trainee's system 816 anchors the annotation to a current image of the view field based on the reference frame data received.
- the trainee's system 816 may detect a plurality of salient features in the current image of the view field displayed on the first display device as described above with reference to the feature detector 344 , compute a descriptor for each of the plurality of the salient features in the current image as described above with reference to the descriptor extractor 346 , and match the current image descriptors with the reference image descriptors as described above with reference to the descriptor match module 348 .
- the trainee's system 816 may then derive a homography or other mapping for the annotation based on the matched descriptors, transform the annotation from the reference image to the current image based on the derived homography as described above with reference to the mapping module 350 , and render the transformed annotation over the current image displayed on the trainee's system as described above with reference to the annotation rendering module 352 .
- blocks 802 , 804 , and 814 are included in a trainee's system 816 .
- blocks 806 , 808 , 810 , and 812 are included in a mentor's system 818 .
- systems 816 and 818 are physically spaced apart from each other, e.g., within a room such as art operating room, in different rooms in a building such as a teaching hospital, or in different buildings (e.g., during a telesurgery session).
- the trainee's system may track a position of a view point of the trainee, such as the position of head or eyes, as described above with reference to the tracking module 356 , and adjust the reference image of the view field based on the position of the view point of the trainee and the geometry information as described above with reference to the depth acquisition module 354 , such that a new image of the view field displayed on the first display device continues to represent a view of the view field by the trainee.
- Annotation anchoring is performed in two major stages, in some examples.
- the first stage preprocesses the reference frame where annotations are defined to prepare for annotation anchoring in future frames.
- the second stage uses the preprocessed reference frame and processes the current frame to anchor the annotation.
- FIG. 9 is an example reference frame preprocessing algorithm 900 in accordance with at least one aspect of the present disclosure.
- the region R of the annotation is defined with an axis aligned rectangle that is obtained by enlarging the 2D axis aligned bounding box of the annotation.
- R would be a rectangle that surrounds the area of interest such as the view field 106 .
- feature points are identified in the region R using the ORB (Oriented FAST (Features from Accelerated Segment Test) and Rotated BRIEF (Binary Robust Independent Elementary Features)) feature detection algorithm, which uses FAST feature detection along with image pyramids to find multiscale features.
- ORB Oriented FAST (Features from Accelerated Segment Test) and Rotated BRIEF (Binary Robust Independent Elementary Features)
- a descriptor is computed for each feature point using the ORB descriptor extraction algorithm as described above with reference to the descriptor extractor 342 .
- the descriptor is a bit string that describes the pixel intensities at each pixel in an image patch surrounding the keypoint in some examples. This allows comparing the descriptors from the reference frame to descriptors of future frames, for example, from the reference frame to the current image.
- the annotation with its set of descriptors is sent to the trainee system 310 where the annotation is tracked and displayed.
- FIG. 10 is an example annotation anchoring algorithm 1000 in accordance with at least one aspect of the present disclosure.
- the current frame is first processed similarly to the reference frame: at step 1002 , features are detected as described above with reference to the feature detector 344 , and then at step 1004 , enhanced with descriptor data as described above with reference to the descriptor extractor 346 .
- descriptor computation may fail. Because descriptor extraction, in some examples, involves reading the intensities of pixels in a ring surrounding the feature, if that ring extended beyond the edges of the image, there would be insufficient information to complete the descriptor extraction.
- Features for which descriptor computation fails may be omitted from further processing of the features of the respective image.
- the reference frame's descriptors are matched to the current frame's descriptors using an all-pairs brute-force matching algorithm as described above with reference to the descriptor match module 348 .
- Each reference frame descriptor d 0i is matched against each current frame descriptor d j , selecting the match with the lowest Hamming distance between the descriptors.
- the matched descriptors are used to define a homography H, as described above with reference to the mapping module 350 , from the reference frame to the current frame using a RANSAC (RANdom SAmple Consensus)-based algorithm at step 1008 .
- RANSAC Random SAmple Consensus
- a RANSAC-based algorithm permits estimating parameters from a set of observed data which contains outliers and determining matching points between the reference frame and the current image.
- this homography computation method takes as one of its parameters a reprojection threshold, which determines whether a match is considered to be an inlier or an outlier.
- the threshold value is scaled based on the downsample factor of the input frame; otherwise, a smaller image with a relatively larger reprojection threshold would allow too many outliers to find a good homography.
- H maps a reference frame point to a current frame point.
- the homography is applied to each annotation point p i , positioning the annotation in the current frame as described above with reference to the mapping module 350 .
- the annotation is rendered with F as background at the position defined by the transformed points p 0i as described above with reference to the annotation rendering module 352 .
- FIG. 11 is an example keypoint matching algorithm 1100 in accordance with at least one aspect of the present disclosure.
- the system may automatically detect the image features in the template, or reference, image as A at step 1102 (Line 1 of the algorithm 1100 ), e.g., as described above with reference to the feature detector module 340 , 344 .
- the locations of those image features are saved as T in step 1104 (Line 2 of the algorithm 1100 ) along with the annotation points, A, made on the reference image.
- step 1106 Line 3 of the algorithm 1100
- a subsequent image has its feature points likewise detected and stored in a second keypoint array S.
- each of the sets are compared to find matching sets between the two keypoint arrays, T and S, e.g., as described above with reference to the descriptor match module 348 .
- an array M of matching indexes is generated, and the algorithm 1100 ends at step 1112 (Line 13 of the algorithm 1100 ).
- FIG. 12 is an example parameter extraction algorithm 1200 in accordance with at least one aspect of the present disclosure.
- the algorithm 1200 finds the changes in pan shift, rotation, and scale between two images.
- the algorithm 1200 starts at step 1202 (Line 1 of the algorithm 1200 ).
- step 1204 Lines 2 - 8 of the algorithm 1200
- the distances between every point pair, D T and D S , and the difference in angles between each corresponding point pair across ⁇ is determined.
- step 1206 Lines 9 and 10 of the algorithm 1200
- the ratio r of sizes between the template and current scene comes from the median distances in D T and D S .
- step 1208 the system then finds the centroids of each of the matched points clouds. All these values are used to find the projection locations of the annotations P by applying Equation 1210 to each of k annotation points as described above with reference to the mapping module 350 .
- FIG. 13 is an example setup 1300 for a simulated transparent display in accordance with at least one aspect of the present disclosure.
- a transparent display such as the display 316 of the trainee's device 310
- LCD liquid crystal display
- the display occlusion shadow 1304 the part of the scene obstructed by the LCD 1302 , may be captured with a camera 1306 . Placing the camera 1306 at the user's viewpoint 1308 may not beneficial because the camera's view would also be obstructed by the LCD, in addition to the disadvantage of the user having to wear the camera.
- the camera 1306 may need to be placed at a different viewpoint, beyond the LCD 1302 , such that the scene would be captured without occlusions.
- the frame captured by the camera 1306 would then be reprojected to the user's viewpoint 1308 , which requires knowledge of scene geometry.
- Some parts of the scene in the display occlusion shadow 1304 may be acquired with a color camera 1306 and a depth camera 1310 , e.g., as described above with reference to the camera 314 and a depth acquisition module 354 .
- the color camera 1306 and the depth camera 1310 may be fixed with respect to each other, and their relative position and orientation may be pre-calibrated, e.g., using a black and white checkerboard that is seen by both cameras, or other predetermined test targets.
- Depth may also be acquired in a real-time by the depth camera 1310 . Depth information is then used to triangulate the image plane 1314 or otherwise determine a mesh or other computational representation of surfaces included in the display occlusion shadow 1304 .
- the user's viewpoint 1308 may be acquired with a tracker 1312 that triangulates the position of the user's head as described above with reference to the tracking module 356 .
- the color and depth data may then be rendered from the user's viewpoint 1308 to ate transparency.
- the color data may be texture-mapped over polygons (e.g., triangles) or other surfaces in the computational representation of the surfaces.
- a method for enhancing remote collaboration comprising superimposing at least one annotation onto an image of the real world using an augmented reality transparent (ART) display.
- ART augmented reality transparent
- annotations are generated through embodied interaction (e.g. gestures) and anchored using an algorithm comprising the features of: generating the annotations in an initial image using translation, scaling, and rotation; establishing a mapping between the initial image and a new image; positioning the annotations in the new image by transforming the position of the annotations in the initial image using the mapping between the initial and the new images.
- N The method of any of paragraphs A-M, further comprising communicating between at least two parties.
- All of the methods and processes described above can be embodied in, and fully automated via, software code modules executed by one or more general purpose computers or processors.
- the code modules can be stored in any type of computer-readable storage medium or other computer storage device. Some or all of the methods can be embodied in specialized computer hardware.
- conjunctive language such as the phrases “X, Y, or Z,” “X, Y, and/or Z,” “at least X, Y, or Z,” or “at least one of X, Y or Z,” unless specifically stated otherwise, is to be understood as signifying that an item, term, etc., can be either X, Y, or Z, or a combination thereof.
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Surgery (AREA)
- Life Sciences & Earth Sciences (AREA)
- Medical Informatics (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Biomedical Technology (AREA)
- Pathology (AREA)
- Public Health (AREA)
- Molecular Biology (AREA)
- Multimedia (AREA)
- Veterinary Medicine (AREA)
- Animal Behavior & Ethology (AREA)
- Heart & Thoracic Surgery (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Human Computer Interaction (AREA)
- Computer Graphics (AREA)
- Gynecology & Obstetrics (AREA)
- Radiology & Medical Imaging (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Signal Processing (AREA)
- Epidemiology (AREA)
- Primary Health Care (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Processing Or Creating Images (AREA)
Abstract
Description
- This application is a non-provisional of, and claims priority to and the benefit of, U.S. Provisional Patent Application Ser. No. 62/168,438, filed May 29, 2015 and entitled “AUGMENTED REALITY TRANSPARENT DISPLAY FOR TELEMENTORING AND TELEPROCTORING,” the entirety of which is incorporated herein by reference.
- This invention was made with government support under W81XWH-14-1-0042 awarded by the U.S. Army Medical Research Acquisition Activity. The government has certain rights in the invention.
- The present disclosure generally relates to displays, and in particular to systems and methods for enhancing remote collaboration using simulated transparent displays with augmented reality.
- Remote collaboration and remote instruction have a number of useful applications such as telementoring. One example of telementoring is surgical telementoring, which has the potential to abstract away the geographic distance between a patient in need of expert surgical care and the surgeon with the required expertise. For example, if a patient urgently needing a complex procedure for which a rural hospital does not have a specialist, telementoring could enable the rural surgeon to perform the procedure under the guidance of a remote expert, without the delays associated with transporting the patient to a major surgical center. If a surgeon were deployed to a location where its operating base has limited resources, the surgeon could provide urgent specialized surgical care with the help of an expert surgeon remotely located, possibly thousands of miles away. Further, if an innovative surgical technique were available but not yet widely adopted, a surgeon could disseminate the novel procedure through telementoring.
- However, the current systems fall short of realizing the possible potential of surgical telementoring. In the current systems, a remote mentor might annotate a video feed of a surgical operation using a telestrator. The annotated video is sent back to the operating room where it is displayed on a nearby monitor, then a local surgeon performing the operation needs to shift focus frequently between the operating field and the nearby monitor to acquire and apply the instructions from the remote mentor. The local surgeon first has to parse and understand the instructions on the monitor, memorize the instructions, and finally after shifting his focus back to the surgery, the local surgeon has to, temporally and spatially, project those instructions into the real-world context of the surgery. This indirect approach to acquiring and applying mentor instruction translates to a significant additional cognitive load for the trainee and interferes with natural hand-eye coordination, which can lead to surgery delays or even errors. Further, the annotations used in the current systems are static and can become disassociated from the operating field elements for which they were associated. For example, an incision line drawn by the remote mentor can move away from its intended location as the operating field changes.
- The Detailed Description is set forth with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical items.
-
FIG. 1 is a graphical representation of annotated photographs of components of an example environment of a telementoring system in accordance with at least one aspect of the present disclosure. -
FIG. 2 is a graphical representation of an example view of a simulated transparent display with augmented reality by the trainee as used inFIG. 1 in accordance with at least one aspect of the present disclosure. -
FIG. 3 is an example block diagram of an embodiment of an enhanced remote collaboration system, e.g., using a simulated transparent display with augmented reality, in accordance with at least one aspect of the present disclosure. -
FIG. 4 is a graphical representation of an example annotation authoring interface as displayed on a remote mentor system in accordance with at least one aspect of the present disclosure. -
FIG. 5 is an example table of annotation categories in accordance with at least one aspect of the present disclosure. -
FIG. 6 is a graphical representation of an example diagram of the annotation orientation in the remote mentor system using a two-touch interaction orientation tool in accordance with at least one aspect of the present disclosure. -
FIG. 7 is an example diagram of the annotation scaling in the remote mentor system sing a two-touch interaction scaling tool in accordance with at least one aspect of the present disclosure. -
FIG. 8 is anexample flowchart 800 of a process for providing anchored annotations over an image for enhancing remote collaboration utilizing a simulated transparent display with augmented reality in accordance with at least one aspect of the present disclosure. -
FIG. 9 is an example reference frame preprocessing algorithm in accordance with at least one aspect of the present disclosure. -
FIG. 10 is an example annotation anchoring algorithm in accordance with at least one aspect of the present disclosure. -
FIG. 11 is an example keypoint matching algorithm in accordance with at least one aspect of the present disclosure. -
FIG. 12 is an example parameter extraction algorithm in accordance with at least one aspect of the present disclosure. -
FIG. 13 is an example setup for a simulated transparent display in accordance with at least one aspect of the present disclosure. - The present disclosure provides, in various aspects, an apparatus, method and system for remote collaboration and remote instructions. Example embodiments in accordance with the present disclosure includes telementoring which can, e.g., remotely provide enhanced instructions, e.g., surgical telementoring, which can be used to demonstrate an apparatus, method and system. In the embodiments, the annotations are superimposed directly onto an image of a view field where telementoring or telecorroboration is desired such as a surgery area, using a simulated transparent display with augmented reality (“AR”). Telemedicine and telementoring applications rely on effective communication of medical expertise. AR has potential of enhancing telementoring either as an interface or as an environment. For example, a virtualized interface may allow for more intuitive interaction between an operating surgeon and relevant medical information provided. In laparoscopic surgery where the operating surgeon and the telementoring surgeon can share the same real-time laparoscopic video, the live video, which is the image of the view field, may be displayed to the telementoring surgeon in conjunction with a view of the operating room. Additional viewpoints may provide greater visual context to both trainee and mentor.
- The fundamental challenge in using AR in surgical environments and other telementoring environments is integrating synthetic overlays seamlessly within a real-world scene. Many existing systems would require the trainee to look at a screen with a display that does not align with the trainee's actual view of the scene outside the screen. Systems that use AR head-mounted displays may interfere with the vision or the trainee's head motion and cause ocular fatigue. In addition, it is important for an augmented image to avoid obscuring important real-world detail, while ensuring that the information provided by the AR is readily accessible to the trainee.
- Generally, any function, component, device, module, system and alike herein described may not be limited to a physical component but may also be realized with a set of executable instructions stored in a non-transitory memory device, e.g., a computer-readable memory, to be executed by one or more processors, or be realized with a combination of one or more physical components with such instructions.
- Computer-readable media described herein include computer storage media and/or communication media. Computer storage media includes tangible storage units such as volatile memory, nonvolatile memory, and/or other persistent and/or auxiliary computer storage media, removable and non-removable computer storage media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes tangible or physical forms of media included in a device or hardware component that is pan of a device or external to a device, including but not limited to random access memory (RAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), phase change memory (PRAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, compact disc read-only memory (CD-ROM), digital versatile disks (DVDs), optical cards or other optical storage media, magnetic cassettes, magnetic tape, magnetic disk storage, magnetic cards or other magnetic storage devices or media, solid-state memory devices, storage arrays, network attached storage, storage area networks, hosted computer storage or memories, storage, devices, and/or storage media that can be used to store and maintain information for access by a
computing device - In contrast to computer storage media, communication media can embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, or other transmission mechanism. As defined herein, computer storage media does not include communication media.
- The terms “local” and “remote” do not imply any specific distance between the operator and the instructor; various aspects are useful with any positive, nonzero distance between the operator and the instructor.
-
FIG. 1 is an example telementoring,system environment 100 in accordance with at least one aspect of the present disclosure. At the local surgeon site, or trainee site, 102, the local surgeon, or a trainee, 104 sees theview field 106 as the annotatedw field 108 on atrainee system 110, shown as a simulated transparent display device, suspended in his field of view. Thetrainee system 110 is connected to aremote mentor system 112 located at aremote mentor site 114 via acommunication network 116. Aremote mentor 118 sees animage 120 of theview field 106, and enters anannotation 122 to theremote mentor system 112. Information related to theannotation 122 is communicated to thetrainee system 110 via thecommunication network 116, and an appropriate annotation is displayed on thetrainee system 110 in real, or near real, time. -
FIG. 2 is anexample view 200 by thetrainee 104 of theview field 106 and the surrounding areas in accordance with at least one aspect of the present disclosure. The trainee's view includes thetrainee system 110 showing a live image of theview field 106 under thetrainee system 110 except for the area where theannotation 202 created by a remote mentor is displayed. In this view, theannotation 202 indicates the precise placement of an additionalsurgical clamp 204 along with the instructions 206 to “Add Clamp.” The simulated transparent display image of theview field 106 displayed on thetrainee system 110 represents, or simulate, a direct view of theview field 106 seen by thetrainee 104, allowing thetrainee 104 to see hishands surgical instruments actual view field 106 under thetrainee system 110 as if seeing through thetrainee system 110. The part of theview field 106 seen by thetrainee 104 through thetrainee system 110 is continued with the surrounding region of theview field 106 that thetrainee 104 sees directly. Theannotation 202 remains anchored to the view field elements for which they were defined even as thetrainee system 110 is repositioned, as the head position of thetrainee 104 changes, and/or as theview field 106 changes over time. The simulated transparent display with augmented reality approach has the potential to bypass the shortcomings of the conventional telestrator-based approach by integrating annotations into the view field, allowing the local surgeon to benefit from the annotations without shifting focus. The alignment between the displayed image and the peripheral regions of the view field preserves the natural hand-eye coordination on which surgeons rely. The annotations are anchored to the view field elements and remain anchored to the elements as the viewpoint and/or view field change causing the elements to shift or move. The anchoring of the annotations to the elements reduces the need for the remote mentor to draw annotations that have drifted out of place, improving the continuity of the visual guidance provided to the trainee. - The functions and utilities of the simulated
transparent display 110 at the trainee site may be accomplished by using a conventional tablet that is capable of displaying the live image, or video stream, acquired by its back-facing camera. The live image may be sent wirelessly to the mentor site where it is displayed on the mentor's tablet. Using the tablet's touch-based user interface, the mentor may add graphical and textual annotations to a frame of the video stream. The annotations are sent back to the trainee site where they are overlaid on the trainee's tablet display to provide guidance to the trainee. -
FIG. 3 is an example block diagram 300 of an embodiment of an enhanced remote collaboration system using a simulated transparent display with augmented reality in accordance with at least one aspect of the present disclosure. Thetrainee system 310 or other local computing device, shown as thedisplay device 110 inFIGS. 1 and 2 , may be implemented using a tablet. Although a tablet is used for example purposes, other types of a computing device equipped with a display and a camera, which may or may not be integrated in the computing device, may be utilized. A processor, or processors, 312 of thetrainee system 310 may be coupled to various components and modules in thetablet 310 and enables functions associated with those components and modules. The modules can additionally or alternatively represent computer program instructions operable by the processor(s) 312 to perform the desired functions, or any combination of modules coupled to or operable by the processor(s) 312 to perform the desired functions. - A
camera 314 of thetrainee system 310 acquires a video stream of theview field 106, and displays each acquired frame on thedisplay 316 of thetrainee system 310. Atransceiver 318 may wirelessly send each acquired frame through acommunication network 320 to aremote mentor system 330 or other remote computing device to be processed for annotation anchoring. Thecommunication network 320 may be any kind of communication network, wired or wireless, capable of communicating required data between thetrainee system 310 and theremote mentor system 330, such as cellular, Wi-Fi, Bluetooth, Ethernet communication networks and alike. - The
remote mentor system 330, which may be another computing device, comprises a processor, or processors, 332 which may be coupled to various components and modules of theremote mentor system 330 and enables functions associated with those components and modules. In theremote mentor system 330, atransceiver 334 receives the video stream comprising of continuously updated frames via thecommunication network 320, and each frame is sequentially displayed on adisplay 336 which may be a touchscreen. Thedisplay 336 at the remote mentor's site may also be a large screen device to project the life size, or enlarged size, of the view field. - In some examples, the
remote mentor system 330 can include at least oneinput device 370, integral or peripheral tosystem 330. The at least oneinput device 370 can be user-operable. Examples ofinput devices 370 can include, e.g., a keyboard, keypad, a mouse, a trackball, a pen sensor or smart pen, a light pen or light gun, a game controller such as a joystick or game pad, a voice input device such as a microphone, voice-recognition device, or speech-recognition device, a touch input device, a gestural input device such as a touchscreen, a grip sensor, an accelerometer, another haptic input, a visual input device such as one or more cameras or image sensors, and the like.Input device 370 can be arranged with respect to, or operably coupled with,display 336 to provide a user interface via which the mentor can view images and provide annotations. For example,input device 370 can be a touch sensor overlaid on or otherwise associated withdisplay 336 to form a touchscreen. - An
annotation module 338 may present various annotation choices to the mentor via a user interface. The mentor may select a reference frame from the video stream and define selected annotations to the reference frame, e.g., by providing inputs viainput device 370 of the user interface. Theannotation module 338 generates and displays the selected annotation, or annotations, over the reference frame displayed on thedisplay 336. To generate the reference frame data, afeature detector module 340 in theremote mentor system 330 may then detect image features in the reference frame in the neighborhood of the annotations. The image features are also referred to herein as “features” or “salient features,” though that term does not imply any particular standard for determining salience, and may be automatically detected in the reference frame based on a predefined description such as pixel intensities. A descriptor extractor module 342 may then compute a respective signature, e.g., a unique signature, for each detected feature. The descriptor, in some examples, is a bit string that describes the pixel intensities at each pixel in an image patch surrounding a feature point. This allows comparing the descriptors from the reference frame to descriptors of future frames. The reference frame data, comprising the annotations, reference frame features, and associated descriptors, are then sent to thetrainee system 310 via thecommunication network 320. - In the
trainee system 310, afeature detector 344 detects image features of a current frame of the video stream being displayed on thedisplay 316, and the features are enhanced with descriptors extracted by adescriptor extractor 346. Adescriptor match module 348 matches the extracted current frame's descriptors with the corresponding reference frame's descriptors received from theremote mentor system 330 where the annotations were defined. Because the trainee's environment may change due to conditions such as thetrainee system 310 being repositioned, the view field geometry changing, or the view field becoming partially occluded due to the surgeon's hands or newly added instruments, the comparison and matching of the reference and current frame features ensure that the annotations are placed and anchored to the intended areas of the view field defined by the mentor annotations. A homography relates any two images of the same planar surface in space and can be used to relate the reference image to the current image. Amapping module 350 may derive a homography for each annotation, and by using the derived homographies, transforms the annotations from the reference frame to the current frame. Techniques other than homography may be utilized to relate the reference and current images. Anannotation rendering module 352 then renders the transformed annotations superimposed over the current frame displayed on thedisplay 316. The annotation appears anchored to the view field. - To enhance the simulated transparency, the
trainee system 310 may comprise adepth acquisition module 354 to obtain accurate geometry of theview field 106 to supplement the information obtained by thecamera 314. The geometry information may be transmitted to the remote mentor'ssystem 330 along with the reference image from the trainee'ssystem 310 to enhance the reference frame data. For example, thefeature detector module 340 of the remote mentor'ssystem 330 may detect at least some of image features in the reference image relative to the annotation based, at least in part, on the geometry information. For example, features may be detected at corners or other points, such as edges, peaks, and valleys, with a high-magnitude spatial derivative of depth information. Thedepth acquisition module 354 may comprise a depth acquisition device such as a depth camera, a motion detector, an infrared depth sensor, or any other device or a combination of devices that are capable of acquiring accurate information of the geometry of theview field 106. Thedepth acquisition module 354 may be integrated in thetrainee system 310. Thetrainee system 310 may additionally comprise auser tracking module 356 to adjust and re-project the images based on the trainee's view point for a better simulation of transparency. Thetracking module 356 may include a plurality of user-facing cameras which may be used to triangulate the trainee's head position. Thetracking module 356 may also be, or include, a depth acquisition device similar to the ones described above for thedepth acquisition module 354. Thetracking module 356 may be integrated in thetrainee system 310, or may be an external module connected to thetrainee system 310. As the trainee's view point changes, missing color samples, due to the limitations of the field of view and/or occlusion changes as the view-point changes from that of the video camera to that of the trainee, may be corrected by filling in the missing color samples with samples from older frames. - The trainee system and the remote mentor system may also include speaker-and-microphone sets, shown as
set 358 for the trainee system and set 360 for the remote mentor system, to allow audio communications between thetrainee 104 and theremote mentor 118. - The
remote mentor system 330, in some examples, sends to thetrainee system 310 only the type of annotations and their position in the reference frame. This compact encoding of annotations saves bandwidth and is sufficient to recreate the annotations at the trainee system based on a local copy of the set of sprites. In other examples, theremote mentor system 330 sends the types and positions of annotations, and also other information. - Applications of the enhanced
remote collaboration system 300 is not limited to a surgical setting, and may be utilized in any remote instruction settings where interactions between a local operator, who is performing an act following instructions, and a remote instructor, who is providing the instructions and viewing the act as seen by the operator, are desirable. The ability to mark or annotate an object at the local operator's site and to follow the object and the annotation as seen by the operator is especially beneficial for hands-on training and instruction activities where the operator's act is dynamic and his view point may shift as the act progresses. Such training and instruction activities may include medical diagnostics, mechanical and/or electrical diagnostics and repairs, navigation where a route to a destination is annotated and updated, and other situations. The role of the remote instructor may also be realized with a computer program or artificial intelligence (AI) interacting with the local operator. With the AI as the remote instructor, the remote mentor'ssystem 330 may be virtualized at a location accessible by the trainee'ssystem 310 including memory in the trainee's system storing the instructions, when executed by the processor(s) 312, to realize the AI. -
FIG. 4 is an example block diagram 400 of anannotation module 338 ofFIG. 3 as displayed on a remote mentor system in accordance with at least one aspect of the present disclosure. In this example embodiment, theannotation module 338 is shown as a user interface for a tablet with a touchscreen. Theannotation module 338 may provide multiple selectable symbols such as icon-labeledbuttons 402 on thedisplay 336 of theremote mentor system 330. The selectable symbols may be selected by a mouse click, voice command, touch, or any other compatible use input mechanisms, e.g.,input device 370,FIG. 3 . Theremote mentor 118 may tap to select a particular button of icon-labeledbuttons 402. Each icon-labeled button may be associated with a predefined function and may be organized into various annotation categories. -
FIG. 5 is an example table ofannotation categories 500 in accordance with at least one aspect of the present disclosure. The categories may include drawingshapes 502,surgical tools 504, text labels 506, and hand gestures 508. The predefined text labels in the text labels 506 may include, e.g., “close,” “incision,” “palpation,” “remove,” and “stitch.” - The types of drawing shapes in the drawing shapes 502 may include: points, lines, and polygons. Each shape may be defined with one or multiple points. In some examples, the mentor may draw on a touchscreen or tablet, draw by dragging with a mouse, or otherwise provide path information via
input device 370. The path information can include information of, e.g., at least one of location, pressure, direction, or duration of a user input. In the three columns ofFIG. 5 , the mentor can indicate an incision by drawing a series of points, and the path information can include the locations of those points. The mentor can indicate a stitch by drawing a line or curve, and the path information can include, e.g., the locations of Bezier control points in a decomposition of the drawn curve into at least one Bezier-curve segment. The mentor can indicate palpation by drawing a closed figure, such as a conic section or polygon, and the path information. These specific shapes are not limiting. For example, an incision can be indicated by a curve and a stitch can be indicated by a series of points. The mentor may also select and drag a certain button to indicate a desired operation. - The types of surgical tools the
surgical tools 504 may include bag-valve-mask (BVM), endotracheal tube (ET), hemostat, iodine swab, longhook, retractor, scalpel, scissors, stethoscope, surgical tape, syringe, tweezers, and other surgical instruments. - The
hand gesture annotations 508 illustrate typical manual actions performed by the surgeon and may include actions such as palpating, pointing, and stretching. Surgical tools, text labels, and hand gesture icons may be positioned based on a reference point, for example, the tip of the scalpel's blade. These selectable icons are displayed on thedisplay 336 as an image with transparent background. -
FIG. 6 is an example diagram 600 of the annotation orientation in theremote mentor system 330 using a two-touch interaction orientation tool in accordance with the present disclosure. Once selected, theannotation 602 may be positioned using a single-touch drag and drop interaction, and may be orientated using a two-touch interaction: one touch for defining the center ofrotation 604 and the other touch for dragging motion for defining the rotation angle, shown as thestarting point 606 and theending point 608. Theupper portion 610 shows the starting position of theannotation 602, from center ofrotation 604 tostarting point 606, and thelower portion 612 shows the ending position of theannotation 602, from center ofrotation 604 to endingpoint 608. -
FIG. 7 is an example diagram 700 of the annotation scaling using a two-touch interaction scaling tool in accordance with the present disclosure. Scaling of theannotation 702 may be performed by using a two finger pinch-and-zoom interaction: one finger for defining the fixed end of scaling 704 and the other finger for defining thestarting point 706 and theending point 708. Theupper portion 708 shows the starting position of theannotation 702, from fixed end of scaling 704 tostarting point 706, and thelower portion 710 shows the ending position of theannotation 702, from fixed end of scaling 704 to endingpoint 708. -
FIG. 8 is anexample flowchart 800 of a process in accordance with at least one aspect of the present disclosure. In some examples, the process is a process for providing anchored annotations over art image, e.g., for enhancing remote collaboration utilizing a simulated transparent display with augmented reality. - At block 802, a trainee's
system 816 acquires a reference image and geometry information of operating view field and d splays it on the trainee's system as described above with reference to thecamera 314 and thedepth acquisition module 354. Atblock 804, the trainee'ssystem 816 transmits the reference image and the geometry information, as described above with reference to thetransceiver 318, to the remote mentor's system via a communication network, as described above with reference to thecommunication network 320. The communication network may be any kind of communication network, wired or wireless, capable of communicating required data between the trainee system and the remote mentor system, such as cellular, Bluetooth, Ethernet communication networks and the like. - At
block 806, the mentor'ssystem 818 receives the reference image and geometry information and displays the reference image on the remote mentor's system as described above with reference to thetransceiver 334 and thedisplay 336. Atblock 808, the remote mentor'ssystem 818 generates an annotation and displays over the reference image on the remote mentor's system as described above with reference to theannotation module 338. The annotation may be received via a user interface such as a selectable icon with a predefined annotation as described inFIGS. 4 and 5 . Atblock 810, the remote mentor'ssystem 818 generates reference frame data based on the reference image, the geometry information, and the annotation as described above with reference to thefeature detector module 340 and the descriptor extractor 342. The reference frame data may be generated from detecting a plurality of salient features in the reference image relative to the annotation and computing a descriptor for each of the plurality of the salient features. Atblock 812, the remote mentor'ssystem 818 transmits the reference frame data and annotation to the trainee's system via the communication network as described above with reference to thetransceiver 334. - At
block 814, the trainee'ssystem 816 anchors the annotation to a current image of the view field based on the reference frame data received. To anchor the annotation to the current image, the trainee'ssystem 816 may detect a plurality of salient features in the current image of the view field displayed on the first display device as described above with reference to thefeature detector 344, compute a descriptor for each of the plurality of the salient features in the current image as described above with reference to thedescriptor extractor 346, and match the current image descriptors with the reference image descriptors as described above with reference to thedescriptor match module 348. The trainee'ssystem 816 may then derive a homography or other mapping for the annotation based on the matched descriptors, transform the annotation from the reference image to the current image based on the derived homography as described above with reference to themapping module 350, and render the transformed annotation over the current image displayed on the trainee's system as described above with reference to theannotation rendering module 352. - In the illustrated example, blocks 802, 804, and 814 are included in a trainee's
system 816. In the illustrated example, blocks 806, 808, 810, and 812 are included in a mentor'ssystem 818. In some examples,systems - As the trainee's system is repositioned, as the view field geometry changes, and/or as the view field becomes partially occluded due to the surgeon's/trainee's hands and due to new instruments added to the view field, the annotations have to be repositioned to remain overlaid onto the associated view field elements. To compensate for these changes, the trainee's system may track a position of a view point of the trainee, such as the position of head or eyes, as described above with reference to the
tracking module 356, and adjust the reference image of the view field based on the position of the view point of the trainee and the geometry information as described above with reference to thedepth acquisition module 354, such that a new image of the view field displayed on the first display device continues to represent a view of the view field by the trainee. - Annotation anchoring is performed in two major stages, in some examples. The first stage preprocesses the reference frame where annotations are defined to prepare for annotation anchoring in future frames. The second stage uses the preprocessed reference frame and processes the current frame to anchor the annotation.
-
FIG. 9 is an example reference frame preprocessing algorithm 900 in accordance with at least one aspect of the present disclosure. - At
step 902, the region R of the annotation is defined with an axis aligned rectangle that is obtained by enlarging the 2D axis aligned bounding box of the annotation. R would be a rectangle that surrounds the area of interest such as theview field 106. Atstep 904 as described above with reference to thefeature detector module 340, feature points are identified in the region R using the ORB (Oriented FAST (Features from Accelerated Segment Test) and Rotated BRIEF (Binary Robust Independent Elementary Features)) feature detection algorithm, which uses FAST feature detection along with image pyramids to find multiscale features. Atstep 906, a descriptor is computed for each feature point using the ORB descriptor extraction algorithm as described above with reference to the descriptor extractor 342. The descriptor is a bit string that describes the pixel intensities at each pixel in an image patch surrounding the keypoint in some examples. This allows comparing the descriptors from the reference frame to descriptors of future frames, for example, from the reference frame to the current image. The annotation with its set of descriptors is sent to thetrainee system 310 where the annotation is tracked and displayed. -
FIG. 10 is an exampleannotation anchoring algorithm 1000 in accordance with at least one aspect of the present disclosure. The current frame is first processed similarly to the reference frame: atstep 1002, features are detected as described above with reference to thefeature detector 344, and then atstep 1004, enhanced with descriptor data as described above with reference to thedescriptor extractor 346. For some features near the edges of the frame, descriptor computation may fail. Because descriptor extraction, in some examples, involves reading the intensities of pixels in a ring surrounding the feature, if that ring extended beyond the edges of the image, there would be insufficient information to complete the descriptor extraction. Features for which descriptor computation fails may be omitted from further processing of the features of the respective image. - Next, at
step 1006, the reference frame's descriptors are matched to the current frame's descriptors using an all-pairs brute-force matching algorithm as described above with reference to thedescriptor match module 348. Each reference frame descriptor d0i is matched against each current frame descriptor dj, selecting the match with the lowest Hamming distance between the descriptors. The matched descriptors are used to define a homography H, as described above with reference to themapping module 350, from the reference frame to the current frame using a RANSAC (RANdom SAmple Consensus)-based algorithm atstep 1008. A RANSAC-based algorithm permits estimating parameters from a set of observed data which contains outliers and determining matching points between the reference frame and the current image. In some examples, this homography computation method takes as one of its parameters a reprojection threshold, which determines whether a match is considered to be an inlier or an outlier. The threshold value is scaled based on the downsample factor of the input frame; otherwise, a smaller image with a relatively larger reprojection threshold would allow too many outliers to find a good homography. H maps a reference frame point to a current frame point. Atstep 1010, the homography is applied to each annotation point pi, positioning the annotation in the current frame as described above with reference to themapping module 350. Finally, atstep 1012, the annotation is rendered with F as background at the position defined by the transformed points p0i as described above with reference to theannotation rendering module 352. -
FIG. 11 is an examplekeypoint matching algorithm 1100 in accordance with at least one aspect of the present disclosure. When the remote mentor selects a reference image, or template, the system may automatically detect the image features in the template, or reference, image as A at step 1102 (Line 1 of the algorithm 1100), e.g., as described above with reference to thefeature detector module Line 2 of the algorithm 1100) along with the annotation points, A, made on the reference image. Then, in step 1106 (Line 3 of the algorithm 1100), for each iteration of the computational thread, a subsequent image has its feature points likewise detected and stored in a second keypoint array S. At step 1108 (Lines 4-9 of the algorithm 1100), each of the sets are compared to find matching sets between the two keypoint arrays, T and S, e.g., as described above with reference to thedescriptor match module 348. At step 1110 (Lines 10-12 of the algorithm 1100), an array M of matching indexes is generated, and thealgorithm 1100 ends at step 1112 (Line 13 of the algorithm 1100). -
FIG. 12 is an exampleparameter extraction algorithm 1200 in accordance with at least one aspect of the present disclosure. Using the set of matches M, along with T and S, thealgorithm 1200 finds the changes in pan shift, rotation, and scale between two images. Thealgorithm 1200 starts at step 1202 (Line 1 of the algorithm 1200). At step 1204 (Lines 2-8 of the algorithm 1200), for each cloud of matched keypoints, the distances between every point pair, DT and DS, and the difference in angles between each corresponding point pair across θ is determined. In step 1206 (Lines Equation 1210 to each of k annotation points as described above with reference to themapping module 350. -
FIG. 13 is anexample setup 1300 for a simulated transparent display in accordance with at least one aspect of the present disclosure. To simulate a transparent display, such as thedisplay 316 of the trainee'sdevice 310, using a conventional liquid crystal display (LCD) 1302 or other electronic display, the image that the user would see in the absence of thedisplay 1302 needs to be displayed. Thedisplay occlusion shadow 1304, the part of the scene obstructed by theLCD 1302, may be captured with acamera 1306. Placing thecamera 1306 at the user'sviewpoint 1308 may not beneficial because the camera's view would also be obstructed by the LCD, in addition to the disadvantage of the user having to wear the camera. Consequently, thecamera 1306 may need to be placed at a different viewpoint, beyond theLCD 1302, such that the scene would be captured without occlusions. The frame captured by thecamera 1306 would then be reprojected to the user'sviewpoint 1308, which requires knowledge of scene geometry. - Some parts of the scene in the
display occlusion shadow 1304 may be acquired with acolor camera 1306 and adepth camera 1310, e.g., as described above with reference to thecamera 314 and adepth acquisition module 354. Thecolor camera 1306 and thedepth camera 1310 may be fixed with respect to each other, and their relative position and orientation may be pre-calibrated, e.g., using a black and white checkerboard that is seen by both cameras, or other predetermined test targets. Depth may also be acquired in a real-time by thedepth camera 1310. Depth information is then used to triangulate theimage plane 1314 or otherwise determine a mesh or other computational representation of surfaces included in thedisplay occlusion shadow 1304. The user'sviewpoint 1308 may be acquired with atracker 1312 that triangulates the position of the user's head as described above with reference to thetracking module 356. The color and depth data may then be rendered from the user'sviewpoint 1308 to ate transparency. For example, the color data may be texture-mapped over polygons (e.g., triangles) or other surfaces in the computational representation of the surfaces. - A: A method for enhancing remote collaboration, comprising superimposing at least one annotation onto an image of the real world using an augmented reality transparent (ART) display.
- B: The method of paragraph A, wherein the ART display is suspended into a users field of view.
- C: The method of paragraph A or B, wherein the ART display is mounted on a user's head.
- D: The method of any of paragraphs A-C, wherein the ART display is implemented using a tablet computer and its video camera.
- E: The method of any of paragraphs A-D, wherein the ART display is implemented using a tablet computer and its video camera, a head tracker, and a depth camera.
- F: The method of any of paragraphs A-E, wherein the ART display is implemented with a physically transparent display, e.g. a transparent OLED.
- G: The method of any of paragraphs A-F, wherein the ART display is mounted using at least one of a mechanical arm or a robotic arm.
- H: The method of any of paragraphs A-G, wherein at least one annotation is anchored to the real-world entity it describes, as the real world changes, as the user head moves, and as the ART display is repositioned.
- I: The method of any of paragraphs A-H, wherein the annotations are generated through embodied interaction (e.g. gestures) and anchored using an algorithm comprising the features of: generating the annotations in an initial image using translation, scaling, and rotation; establishing a mapping between the initial image and a new image; positioning the annotations in the new image by transforming the position of the annotations in the initial image using the mapping between the initial and the new images.
- J: The method of paragraph I, further comprising matching a plurality of features in the new image with a plurality of initial features in the initial image.
- K: The method of any of paragraphs A-J, wherein the remote collaboration is telementoring.
- L: The method of any of paragraphs A-K, wherein the remote collaboration is surgical telementoring.
- M: The method of any of paragraphs A-L, wherein the remote collaboration is remote instruction.
- N: The method of any of paragraphs A-M, further comprising communicating between at least two parties.
- O: The method of any of paragraphs A-N, wherein the at least two parties comprise a human party and artificial party.
- P: The method of any of paragraphs A-O, wherein the at least two parties comprise at least two human parties.
- Q: The method of any of paragraphs A-P, further comprising orienting the annotations based at least in part on that a geometric transformation and a current frame.
- Although the techniques have been described in language particular to structural features or methodological acts, it is to be understood that the appended claims are not necessarily limited to the features or acts described. Rather, the features and acts are described as example implementations of such techniques.
- All of the methods and processes described above can be embodied in, and fully automated via, software code modules executed by one or more general purpose computers or processors. The code modules can be stored in any type of computer-readable storage medium or other computer storage device. Some or all of the methods can be embodied in specialized computer hardware.
- Conditional language such as, among others, “can,” “could,” “might” and/or “may,” unless specifically stated otherwise, are understood within the context to present that certain examples include, while other examples do not include, certain features, elements and/or steps. Thus, such conditional language is not generally intended to imply that certain features, elements arid/or steps are in any way required for one or more examples and/or that one or more examples necessarily include logic for deciding, with and/or without user input and/or prompting, whether certain features, elements and/or steps are included and/or are to be performed in any particular example. The word “or” and the phrase “and/or” are used herein in art inclusive sense unless specifically stated otherwise. Accordingly, conjunctive language such as the phrases “X, Y, or Z,” “X, Y, and/or Z,” “at least X, Y, or Z,” or “at least one of X, Y or Z,” unless specifically stated otherwise, is to be understood as signifying that an item, term, etc., can be either X, Y, or Z, or a combination thereof.
- The disclosure includes combinations of the examples described herein. References to a particular “example” and the like refer to features that are present in at least one example or configuration of what is within the scope of the disclosed subject matter. Separate references to “an example” or “particular examples” or the like do not necessarily refer to the same example or examples; however, such examples are not mutually exclusive, unless specifically indicated. The use of singular or plural in referring to “example,” “examples,” “method,” “methods” and the like is not limiting. Moreover, in the claims, any reference to a group of items provided by a preceding claim clause is a reference to at least some of the items in the group of items, unless specifically stated otherwise.
- Various embodiments of the present disclosure have been described in detail with particular reference to certain preferred aspects thereof, but it will be understood that variations, combinations, and modifications can be effected by a person of ordinary skill in the art within the spirit and scope of the disclosure.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/167,011 US9503681B1 (en) | 2015-05-29 | 2016-05-27 | Simulated transparent display with augmented reality for remote collaboration |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201562168438P | 2015-05-29 | 2015-05-29 | |
US15/167,011 US9503681B1 (en) | 2015-05-29 | 2016-05-27 | Simulated transparent display with augmented reality for remote collaboration |
Publications (2)
Publication Number | Publication Date |
---|---|
US9503681B1 US9503681B1 (en) | 2016-11-22 |
US20160353055A1 true US20160353055A1 (en) | 2016-12-01 |
Family
ID=57287882
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/167,011 Active US9503681B1 (en) | 2015-05-29 | 2016-05-27 | Simulated transparent display with augmented reality for remote collaboration |
Country Status (1)
Country | Link |
---|---|
US (1) | US9503681B1 (en) |
Cited By (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180046355A1 (en) * | 2016-08-12 | 2018-02-15 | Boston Scientific Scimed, Inc. | Distributed interactive medical visualization system with user interface features |
US10133534B2 (en) * | 2015-11-25 | 2018-11-20 | Tencent Technology (Shenzhen) Company Limited | Image processing method and apparatus for interactive augmented reality |
WO2019119022A1 (en) * | 2017-12-21 | 2019-06-27 | Ehatsystems Pty Ltd | Augmented visual assistance system for assisting a person working at a remote workplace, method and headwear for use therewith |
WO2019160955A1 (en) * | 2018-02-13 | 2019-08-22 | SentiAR, Inc. | Augmented reality display sharing |
KR102051309B1 (en) * | 2019-06-27 | 2019-12-03 | 주식회사 버넥트 | Intelligent technology based augmented reality system |
US10592067B2 (en) | 2016-08-12 | 2020-03-17 | Boston Scientific Scimed, Inc. | Distributed interactive medical visualization system with primary/secondary interaction features |
US10627911B2 (en) | 2017-04-25 | 2020-04-21 | International Business Machines Corporation | Remote interaction with content of a transparent display |
US11160614B2 (en) | 2017-10-25 | 2021-11-02 | Synaptive Medical Inc. | Surgical imaging sensor and display unit, and surgical navigation system associated therewith |
WO2021236245A1 (en) * | 2020-05-17 | 2021-11-25 | James L. Orrington, II D.D.S., P.C. | Protective apparatuses for minimizing risk of transmission of infection and associated systems |
EP3773309A4 (en) * | 2018-03-26 | 2022-06-08 | Covidien LP | Telementoring control assemblies for robotic surgical systems |
US11368670B2 (en) | 2017-10-26 | 2022-06-21 | Yeda Research And Development Co. Ltd. | Augmented reality display system and method |
KR102458491B1 (en) * | 2022-03-17 | 2022-10-26 | 주식회사 메디씽큐 | System for providing remote collaborative treatment for tagging realtime surgical video |
US11521512B2 (en) | 2019-02-19 | 2022-12-06 | Illinois Tool Works Inc. | Systems for simulating joining operations using mobile devices |
US11645936B2 (en) | 2019-11-25 | 2023-05-09 | Illinois Tool Works Inc. | Weld training simulations using mobile devices, modular workpieces, and simulated welding equipment |
US11707806B2 (en) * | 2019-02-12 | 2023-07-25 | Illinois Tool Works Inc. | Virtual markings in welding systems |
US11721231B2 (en) | 2019-11-25 | 2023-08-08 | Illinois Tool Works Inc. | Weld training simulations using mobile devices, modular workpieces, and simulated welding equipment |
JP7393842B1 (en) | 2022-02-09 | 2023-12-07 | リバーフィールド株式会社 | Support system, support device, supported device |
EP4319170A1 (en) * | 2022-08-04 | 2024-02-07 | Koninklijke Philips N.V. | Vendor-agnostic remote-controlled screen overlay for collaboration in a virtualized radiology environment |
WO2024028235A1 (en) * | 2022-08-04 | 2024-02-08 | Koninklijke Philips N.V. | Vendor-agnostic remote-controlled screen overlay for collaboration in a virtualized radiology environment |
Families Citing this family (36)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9566414B2 (en) | 2013-03-13 | 2017-02-14 | Hansen Medical, Inc. | Integrated catheter and guide wire controller |
US9283046B2 (en) | 2013-03-15 | 2016-03-15 | Hansen Medical, Inc. | User interface for active drive apparatus with finite range of motion |
US10849702B2 (en) | 2013-03-15 | 2020-12-01 | Auris Health, Inc. | User input devices for controlling manipulation of guidewires and catheters |
JP6138566B2 (en) * | 2013-04-24 | 2017-05-31 | 川崎重工業株式会社 | Component mounting work support system and component mounting method |
US11020016B2 (en) | 2013-05-30 | 2021-06-01 | Auris Health, Inc. | System and method for displaying anatomy and devices on a movable display |
EP2923669B1 (en) | 2014-03-24 | 2017-06-28 | Hansen Medical, Inc. | Systems and devices for catheter driving instinctiveness |
US10154239B2 (en) | 2014-12-30 | 2018-12-11 | Onpoint Medical, Inc. | Image-guided surgery with surface reconstruction and augmented reality visualization |
KR102413074B1 (en) * | 2015-09-21 | 2022-06-24 | 삼성전자주식회사 | User terminal device, Electronic device, And Method for controlling the user terminal device and the electronic device thereof |
US9861446B2 (en) | 2016-03-12 | 2018-01-09 | Philipp K. Lang | Devices and methods for surgery |
US11037464B2 (en) * | 2016-07-21 | 2021-06-15 | Auris Health, Inc. | System with emulator movement tracking for controlling medical devices |
US10146758B1 (en) * | 2016-09-30 | 2018-12-04 | Amazon Technologies, Inc. | Distributed moderation and dynamic display of content annotations |
CA3049662A1 (en) | 2017-01-16 | 2018-07-19 | Philipp K. Lang | Optical guidance for surgical, medical, and dental procedures |
US10956981B1 (en) | 2017-04-07 | 2021-03-23 | State Farm Mutual Automobile Insurance Company | Systems and methods for visualizing an accident scene |
US10373387B1 (en) * | 2017-04-07 | 2019-08-06 | State Farm Mutual Automobile Insurance Company | Systems and methods for enhancing and developing accident scene visualizations |
US11054894B2 (en) | 2017-05-05 | 2021-07-06 | Microsoft Technology Licensing, Llc | Integrated mixed-input system |
US10895966B2 (en) | 2017-06-30 | 2021-01-19 | Microsoft Technology Licensing, Llc | Selection using a multi-device mixed interactivity system |
US11023109B2 (en) * | 2017-06-30 | 2021-06-01 | Microsoft Techniogy Licensing, LLC | Annotation using a multi-device mixed interactivity system |
US11801114B2 (en) | 2017-09-11 | 2023-10-31 | Philipp K. Lang | Augmented reality display for vascular and other interventions, compensation for cardiac and respiratory motion |
CN110831534B (en) | 2017-12-08 | 2023-04-28 | 奥瑞斯健康公司 | System and method for medical instrument navigation and targeting |
US11348257B2 (en) | 2018-01-29 | 2022-05-31 | Philipp K. Lang | Augmented reality guidance for orthopedic and other surgical procedures |
US10496366B2 (en) * | 2018-04-18 | 2019-12-03 | Comcast Cable Communications, Llc | Method to determine intended direction of a vocal command and target for vocal interaction |
AU2019262082A1 (en) * | 2018-04-30 | 2020-06-04 | Singular Health Pte Ltd | Medical virtual reality and mixed reality collaboration platform |
KR20210010871A (en) | 2018-05-18 | 2021-01-28 | 아우리스 헬스, 인코포레이티드 | Controller for robotic remote operation system |
WO2019245862A1 (en) * | 2018-06-19 | 2019-12-26 | Tornier, Inc. | Visualization of intraoperatively modified surgical plans |
WO2020033208A1 (en) | 2018-08-07 | 2020-02-13 | Intuitive Surgical Operations, Inc. | Multi-modal visualization in computer-assisted tele-operated surgery |
WO2020041228A1 (en) * | 2018-08-20 | 2020-02-27 | Safavi Abbasi Sam | Neuromuscular enhancement system |
US10776619B2 (en) * | 2018-09-27 | 2020-09-15 | The Toronto-Dominion Bank | Systems and methods for augmenting a displayed document |
JP7234264B2 (en) * | 2019-02-06 | 2023-03-07 | マクセル株式会社 | MIXED REALITY DISPLAY DEVICE AND MIXED REALITY DISPLAY METHOD |
US11857378B1 (en) | 2019-02-14 | 2024-01-02 | Onpoint Medical, Inc. | Systems for adjusting and tracking head mounted displays during surgery including with surgical helmets |
US11553969B1 (en) | 2019-02-14 | 2023-01-17 | Onpoint Medical, Inc. | System for computation of object coordinates accounting for movement of a surgical site for spinal and other procedures |
US11872007B2 (en) | 2019-06-28 | 2024-01-16 | Auris Health, Inc. | Console overlay and methods of using same |
CN111915819A (en) * | 2020-08-14 | 2020-11-10 | 中国工商银行股份有限公司 | Remote virtual interaction method, device and system |
US20220198420A1 (en) * | 2020-12-17 | 2022-06-23 | Toshiba Global Commerce Solutions Holdings Corporation | Self-checkout systems using overlapping display devices |
WO2022192585A1 (en) | 2021-03-10 | 2022-09-15 | Onpoint Medical, Inc. | Augmented reality guidance for imaging systems and robotic surgery |
US11995853B2 (en) | 2021-11-04 | 2024-05-28 | Honeywell Federal Manufacturing & Technologies, Llc | System and method for transparent augmented reality |
WO2024090228A1 (en) * | 2022-10-26 | 2024-05-02 | 伊知朗 竹政 | Information processing device, information processing system, and information processing program |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080117225A1 (en) * | 2006-11-21 | 2008-05-22 | Rainer Wegenkittl | System and Method for Geometric Image Annotation |
US8554307B2 (en) * | 2010-04-12 | 2013-10-08 | Inneroptic Technology, Inc. | Image annotation in image-guided medical procedures |
WO2013049386A1 (en) * | 2011-09-27 | 2013-04-04 | Allied Minds Devices Llc | Instruct-or |
US20140176661A1 (en) * | 2012-12-21 | 2014-06-26 | G. Anthony Reina | System and method for surgical telementoring and training with virtualized telestration and haptic holograms, including metadata tagging, encapsulation and saving multi-modal streaming medical imagery together with multi-dimensional [4-d] virtual mesh and multi-sensory annotation in standard file formats used for digital imaging and communications in medicine (dicom) |
US20140198190A1 (en) * | 2013-01-16 | 2014-07-17 | Kris Okumu | Wearable surgical imaging device with semi-transparent screen |
EP2973105B1 (en) * | 2013-03-15 | 2022-08-31 | Arthrex, Inc | Surgical imaging system and method for processing surgical images |
US10154239B2 (en) * | 2014-12-30 | 2018-12-11 | Onpoint Medical, Inc. | Image-guided surgery with surface reconstruction and augmented reality visualization |
US9516255B2 (en) * | 2015-01-21 | 2016-12-06 | Microsoft Technology Licensing, Llc | Communication system |
-
2016
- 2016-05-27 US US15/167,011 patent/US9503681B1/en active Active
Cited By (24)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10133534B2 (en) * | 2015-11-25 | 2018-11-20 | Tencent Technology (Shenzhen) Company Limited | Image processing method and apparatus for interactive augmented reality |
US10592067B2 (en) | 2016-08-12 | 2020-03-17 | Boston Scientific Scimed, Inc. | Distributed interactive medical visualization system with primary/secondary interaction features |
US20180046355A1 (en) * | 2016-08-12 | 2018-02-15 | Boston Scientific Scimed, Inc. | Distributed interactive medical visualization system with user interface features |
US10585552B2 (en) * | 2016-08-12 | 2020-03-10 | Boston Scientific Scimed, Inc. | Distributed interactive medical visualization system with user interface features |
US10627911B2 (en) | 2017-04-25 | 2020-04-21 | International Business Machines Corporation | Remote interaction with content of a transparent display |
US11160614B2 (en) | 2017-10-25 | 2021-11-02 | Synaptive Medical Inc. | Surgical imaging sensor and display unit, and surgical navigation system associated therewith |
US11368670B2 (en) | 2017-10-26 | 2022-06-21 | Yeda Research And Development Co. Ltd. | Augmented reality display system and method |
WO2019119022A1 (en) * | 2017-12-21 | 2019-06-27 | Ehatsystems Pty Ltd | Augmented visual assistance system for assisting a person working at a remote workplace, method and headwear for use therewith |
US11507336B2 (en) | 2018-02-13 | 2022-11-22 | SentiAR, Inc. | Augmented reality display sharing |
US11003410B2 (en) * | 2018-02-13 | 2021-05-11 | SentiAR, Inc. | Augmented reality display sharing |
WO2019160955A1 (en) * | 2018-02-13 | 2019-08-22 | SentiAR, Inc. | Augmented reality display sharing |
EP3773309A4 (en) * | 2018-03-26 | 2022-06-08 | Covidien LP | Telementoring control assemblies for robotic surgical systems |
US11707806B2 (en) * | 2019-02-12 | 2023-07-25 | Illinois Tool Works Inc. | Virtual markings in welding systems |
US11521512B2 (en) | 2019-02-19 | 2022-12-06 | Illinois Tool Works Inc. | Systems for simulating joining operations using mobile devices |
US11967249B2 (en) | 2019-02-19 | 2024-04-23 | Illinois Tool Works Inc. | Systems for simulating joining operations using mobile devices |
KR102051309B1 (en) * | 2019-06-27 | 2019-12-03 | 주식회사 버넥트 | Intelligent technology based augmented reality system |
US11645936B2 (en) | 2019-11-25 | 2023-05-09 | Illinois Tool Works Inc. | Weld training simulations using mobile devices, modular workpieces, and simulated welding equipment |
US11721231B2 (en) | 2019-11-25 | 2023-08-08 | Illinois Tool Works Inc. | Weld training simulations using mobile devices, modular workpieces, and simulated welding equipment |
WO2021236245A1 (en) * | 2020-05-17 | 2021-11-25 | James L. Orrington, II D.D.S., P.C. | Protective apparatuses for minimizing risk of transmission of infection and associated systems |
JP7393842B1 (en) | 2022-02-09 | 2023-12-07 | リバーフィールド株式会社 | Support system, support device, supported device |
KR102458491B1 (en) * | 2022-03-17 | 2022-10-26 | 주식회사 메디씽큐 | System for providing remote collaborative treatment for tagging realtime surgical video |
WO2023177002A1 (en) * | 2022-03-17 | 2023-09-21 | 주식회사 메디씽큐 | Remote collaboration support system in which real-time surgical video can be tagged |
EP4319170A1 (en) * | 2022-08-04 | 2024-02-07 | Koninklijke Philips N.V. | Vendor-agnostic remote-controlled screen overlay for collaboration in a virtualized radiology environment |
WO2024028235A1 (en) * | 2022-08-04 | 2024-02-08 | Koninklijke Philips N.V. | Vendor-agnostic remote-controlled screen overlay for collaboration in a virtualized radiology environment |
Also Published As
Publication number | Publication date |
---|---|
US9503681B1 (en) | 2016-11-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9503681B1 (en) | Simulated transparent display with augmented reality for remote collaboration | |
US11232639B2 (en) | Rendering virtual objects in 3D environments | |
US11507336B2 (en) | Augmented reality display sharing | |
US11730545B2 (en) | System and method for multi-client deployment of augmented reality instrument tracking | |
Polvi et al. | SlidAR: A 3D positioning method for SLAM-based handheld augmented reality | |
Andersen et al. | Virtual annotations of the surgical field through an augmented reality transparent display | |
Kim et al. | Touch and hand gesture-based interactions for directly manipulating 3D virtual objects in mobile augmented reality | |
KR101453815B1 (en) | Device and method for providing user interface which recognizes a user's motion considering the user's viewpoint | |
CA2794898C (en) | Method of rendering and manipulating anatomical images on mobile computing device | |
JP7337104B2 (en) | Model animation multi-plane interaction method, apparatus, device and storage medium by augmented reality | |
US10969872B2 (en) | Gesture interface | |
CN108304075B (en) | Method and device for performing man-machine interaction on augmented reality device | |
US20200004328A1 (en) | Gaze based interface for augmented reality environment | |
KR20130108643A (en) | Systems and methods for a gaze and gesture interface | |
US9269324B2 (en) | Orientation aware application demonstration interface | |
Andersen et al. | Avoiding focus shifts in surgical telementoring using an augmented reality transparent display | |
Camba et al. | From reality to augmented reality: Rapid strategies for developing marker-based AR content using image capturing and authoring tools | |
CN114299809B (en) | Direction information display method, display device, electronic apparatus, and readable storage medium | |
Loescher et al. | An augmented reality approach to surgical telementoring | |
Shi et al. | Error elimination method in moving target tracking in real-time augmented reality | |
JP2013257830A (en) | Information processor | |
JP7464933B2 (en) | Display device and display system | |
Andersen | Effective User Guidance Through Augmented Reality Interfaces: Advances and Applications | |
EP4207090A1 (en) | A method of learning a target object using a virtual viewpoint camera and a method of augmenting a virtual model on a real object that implements the target object using the same | |
Rasool et al. | Modeling arthroscopic camera with haptic devices in image-based virtual environments |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: PURDUE RESEARCH FOUNDATION, INDIANA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:WACHS, JUAN P.;POPESCU, VOICU S.;SIGNING DATES FROM 20160622 TO 20160705;REEL/FRAME:039996/0166 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FEPP | Fee payment procedure |
Free format text: SURCHARGE FOR LATE PAYMENT, SMALL ENTITY (ORIGINAL EVENT CODE: M2554); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YR, SMALL ENTITY (ORIGINAL EVENT CODE: M2551); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY Year of fee payment: 4 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YR, SMALL ENTITY (ORIGINAL EVENT CODE: M2552); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY Year of fee payment: 8 |