Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T19/00—Manipulating 3D models or images for computer graphics
  • G06T19/006—Mixed reality
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/01—Head-up displays
  • G02B27/017—Head mounted
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
  • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
  • G16H20/40—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
  • G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
  • G16H40/63—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for local operation
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H80/00—ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B17/00—Surgical instruments, devices or methods
  • A61B2017/00017—Electrical control of surgical instruments
  • A61B2017/00216—Electrical control of surgical instruments with eye tracking or head position tracking control
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
  • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
  • A61B2034/2046—Tracking techniques
  • A61B2034/2048—Tracking techniques using an accelerometer or inertia sensor
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
  • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
  • A61B2090/364—Correlation of different images or relation of image positions in respect to the body
  • A61B2090/365—Correlation of different images or relation of image positions in respect to the body augmented reality, i.e. correlating a live optical image with another image
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
  • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
  • A61B90/37—Surgical systems with images on a monitor during operation
  • A61B2090/371—Surgical systems with images on a monitor during operation with simultaneous use of two cameras
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
  • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
  • A61B90/37—Surgical systems with images on a monitor during operation
  • A61B2090/372—Details of monitor hardware
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
  • A61B90/50—Supports for surgical instruments, e.g. articulated arms
  • A61B2090/502—Headgear, e.g. helmet, spectacles
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/01—Head-up displays
  • G02B27/0101—Head-up displays characterised by optical features
  • G02B2027/0132—Head-up displays characterised by optical features comprising binocular systems
  • G02B2027/0134—Head-up displays characterised by optical features comprising binocular systems of stereoscopic type
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/01—Head-up displays
  • G02B27/0101—Head-up displays characterised by optical features
  • G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/01—Head-up displays
  • G02B27/0179—Display position adjusting means not related to the information to be displayed
  • G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye

Definitions

• FIG. 7 illustrates a 3D space with a shared virtual object fixed at a location within the 3D for augmented reality for collaborative interventions, in accordance with an aspect of the present disclosure
• FIG. 10 illustrates a control table for augmented reality for collaborative interventions, in accordance with a representative embodiment
• FIG. 14 illustrates timelines for augmented reality for collaborative interventions, in accordance with a representative embodiment
• FIGs. 16B illustrates another process flow for augmented reality for collaborative interventions, in accordance with representative embodiments.
• mapping and registering can also be performed at the time of the AR session.
• a 3D space can also be dynamically mapped after an augmented reality session begins, such as by using the sensors of augmented reality devices to collect parameters of a 3D space and then using image recognition to register the physical parameters and physical items in the 3D space.
• an augmented reality session is initiated.
• An augmented reality session may be considered to start when a first subject wearing a head-mountable device enters a 3D space.
• multiple augmented reality sessions may take place simultaneously, wherein each of multiple different subjects wearing head-mountable devices individually enters the 3D space.
• the head-mountable devices may be pre-programmed or preauthorized to access the shared portion of the 3D space which is occupied by a virtual reality object.
• augmented reality sessions may correspond to a medical intervention occurring in a pre-mapped operating room serving as the 3D space, and each subject authorized to access the virtual reality object may access information displayed via the virtual reality object during the medical intervention.
• the first visual information and the first shared portion are not restricted to head-mountable devices. Rather, head-mountable devices are readily explained in the context of a 3D space such as an operating room and are used as examples for convenience herein.
• the virtual reality object occupying the first shared portion may be displayed to any authorized subject with a view of the 3D space that includes the shared portion. For example, even a remote user watching the 3D space via a camera may have access to the first shared portion when the remote user is so authorized.
• users with access to a first shared portion may use projected images, transparent heads-up displays, and other forms of augmented reality devices.
• the 3D space 300 is an enclosure such as an operating room, and may be pre-mapped so that every physical object in the 3D space 300 is mapped in preparation for the augmented reality for collaborative interventions described herein.
• the pre-mapping for the 3D space may be used therefore to provide each of the five subjects 381-385 with augmented reality. That is, the pre-mapping provides physical constraints that cannot be altered, whereas virtual objects can be provided in the 3D space in locations that do not conflict with the pre-mapped physical constraints.
• a medical equipment computer #1 381 is used to obtain and display medical information from sensors on or around the patient.
• An augmented reality interface 386 is used to selectively provide the medical information to personnel via augmented reality.
• the transparent optical displays 211a, 21 1b may, for example, simultaneously allow subjects to view the physical world and artificially generated virtual objects.
• the transparent optical displays 21 1a, 211b may include, for example, transparent and polarized OLEDs, light- guide optical elements, and similar materials arranged in a matrix that can be individually and logically controlled, i.e., without a projected beam as in FIG. 4.
• Examples of the elements and materials that can be used for transparent optical displays 21 1a, 211b include an electroluminescent display elements, liquid crystal display (LCD) elements, and waveguides, reflective coatings.
• the computer system 600 may operate in the capacity of a server or as a client user computer in a server-client user network environment, or as a peer computer system in a peer-to-peer (or distributed) network environment.
• the computer system 600 can also be implemented as or incorporated into various devices, such as a head-mountable device, a stationary computer, a mobile computer, a personal computer (PC), a laptop computer, a tablet computer, a wireless smart phone, a personal digital assistant (PDA), a communications device, or any other machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.
• different shared regions may be highlighted in different colors, such as a yellow border or shading for a first shared region controlled by a user and a green border or shading for a second shared region controlled by the same user.
• Other screens that are unshared i.e., for the user's personal viewing
• a dedicated shared region may be the same fixed location in the room for all users with access to the dedicated shared region, as was the case with shared virtual object 710 in FIG. 7, and as may be useful when, for example, a virtual screen is overlaid onto an existing physical object in the 3D space. Additionally, a screen within a dedicated shared region may be oriented perpendicular to each user in a 3D space, as determined using sensors of the augmented reality devices used by the users. Additionally, a 3D hologram may be shared as an augmented reality object.
• FIG. 13 illustrates another control process for augmented reality for collaborative interventions, in accordance with a representative embodiment.
• the process starts with a default display of default visual information at SI 302.
• the default visual information may be from a feed designated by a controller of a shared portion of the 3D space, such as a feed from a piece of equipment monitoring a patient or a feed showing fixed images such as X-ray images.

Landscapes

• Engineering & Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Health & Medical Sciences (AREA)
• Human Computer Interaction (AREA)
• Medical Informatics (AREA)
• Public Health (AREA)
• Primary Health Care (AREA)
• Biomedical Technology (AREA)
• Epidemiology (AREA)
• General Health & Medical Sciences (AREA)
• Optics & Photonics (AREA)
• Software Systems (AREA)
• Computer Graphics (AREA)
• Computer Hardware Design (AREA)
• Urology & Nephrology (AREA)
• Surgery (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Business, Economics & Management (AREA)
• General Business, Economics & Management (AREA)
• Pathology (AREA)
• Processing Or Creating Images (AREA)
• User Interface Of Digital Computer (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=62244456

Family Applications (1)

Country Status (5)

Families Citing this family (21)

Citations (3)

Family Cites Families (8)

• 2018
  • 2018-05-09 JP JP2019563618A patent/JP2020520521A/ja not_active Ceased
  • 2018-05-09 US US16/613,607 patent/US11069146B2/en active Active
  • 2018-05-09 EP EP18726934.5A patent/EP3625649A1/en not_active Withdrawn
  • 2018-05-09 CN CN201880047279.5A patent/CN110914789A/zh active Pending
  • 2018-05-09 WO PCT/EP2018/062013 patent/WO2018210656A1/en not_active Ceased

Patent Citations (3)

Also Published As

Similar Documents

Legal Events