WO2014125789A1 - Head mounted display and control method for head mounted display - Google Patents
Head mounted display and control method for head mounted display Download PDFInfo
- Publication number
- WO2014125789A1 WO2014125789A1 PCT/JP2014/000595 JP2014000595W WO2014125789A1 WO 2014125789 A1 WO2014125789 A1 WO 2014125789A1 JP 2014000595 W JP2014000595 W JP 2014000595W WO 2014125789 A1 WO2014125789 A1 WO 2014125789A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- information
- image
- user
- superimposition
- head mounted
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims description 447
- 238000012545 processing Methods 0.000 claims abstract description 263
- 230000008569 process Effects 0.000 claims description 200
- 230000000007 visual effect Effects 0.000 claims description 124
- 238000001514 detection method Methods 0.000 claims description 24
- 238000012937 correction Methods 0.000 claims description 19
- 230000009467 reduction Effects 0.000 claims description 18
- 239000000284 extract Substances 0.000 claims description 14
- 238000006243 chemical reaction Methods 0.000 claims description 11
- 238000009966 trimming Methods 0.000 claims description 10
- 238000005259 measurement Methods 0.000 claims description 9
- 239000003550 marker Substances 0.000 claims description 8
- 230000004044 response Effects 0.000 claims description 4
- 230000003287 optical effect Effects 0.000 description 65
- 238000010586 diagram Methods 0.000 description 48
- 230000000875 corresponding effect Effects 0.000 description 40
- 230000006870 function Effects 0.000 description 23
- 230000005540 biological transmission Effects 0.000 description 21
- 210000003128 head Anatomy 0.000 description 21
- 230000004048 modification Effects 0.000 description 19
- 238000012986 modification Methods 0.000 description 19
- 210000001747 pupil Anatomy 0.000 description 15
- 238000001356 surgical procedure Methods 0.000 description 15
- 238000002591 computed tomography Methods 0.000 description 14
- 230000033001 locomotion Effects 0.000 description 14
- 230000008859 change Effects 0.000 description 13
- 230000006854 communication Effects 0.000 description 11
- 238000004891 communication Methods 0.000 description 11
- 239000000470 constituent Substances 0.000 description 10
- 230000002596 correlated effect Effects 0.000 description 9
- 238000002595 magnetic resonance imaging Methods 0.000 description 6
- 230000005236 sound signal Effects 0.000 description 6
- FFBHFFJDDLITSX-UHFFFAOYSA-N benzyl N-[2-hydroxy-4-(3-oxomorpholin-4-yl)phenyl]carbamate Chemical compound OC1=C(NC(=O)OCC2=CC=CC=C2)C=CC(=C1)N1CCOCC1=O FFBHFFJDDLITSX-UHFFFAOYSA-N 0.000 description 5
- 239000011521 glass Substances 0.000 description 5
- 239000004973 liquid crystal related substance Substances 0.000 description 5
- 238000001931 thermography Methods 0.000 description 5
- 239000002131 composite material Substances 0.000 description 4
- 238000004590 computer program Methods 0.000 description 4
- 238000010276 construction Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 210000001145 finger joint Anatomy 0.000 description 4
- 210000004932 little finger Anatomy 0.000 description 4
- 230000008054 signal transmission Effects 0.000 description 4
- 230000015572 biosynthetic process Effects 0.000 description 3
- 210000001525 retina Anatomy 0.000 description 3
- 206010052143 Ocular discomfort Diseases 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 2
- 208000003464 asthenopia Diseases 0.000 description 2
- 210000004204 blood vessel Anatomy 0.000 description 2
- 210000000988 bone and bone Anatomy 0.000 description 2
- 230000000295 complement effect Effects 0.000 description 2
- 238000006073 displacement reaction Methods 0.000 description 2
- 230000004424 eye movement Effects 0.000 description 2
- 230000004438 eyesight Effects 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000002324 minimally invasive surgery Methods 0.000 description 2
- 230000029058 respiratory gaseous exchange Effects 0.000 description 2
- 239000003381 stabilizer Substances 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 238000002834 transmittance Methods 0.000 description 2
- WQZGKKKJIJFFOK-GASJEMHNSA-N Glucose Natural products OC[C@H]1OC(O)[C@H](O)[C@@H](O)[C@@H]1O WQZGKKKJIJFFOK-GASJEMHNSA-N 0.000 description 1
- 101100259947 Homo sapiens TBATA gene Proteins 0.000 description 1
- 230000001154 acute effect Effects 0.000 description 1
- 230000001174 ascending effect Effects 0.000 description 1
- QVGXLLKOCUKJST-UHFFFAOYSA-N atomic oxygen Chemical compound [O] QVGXLLKOCUKJST-UHFFFAOYSA-N 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000007175 bidirectional communication Effects 0.000 description 1
- 230000004397 blinking Effects 0.000 description 1
- 239000008280 blood Substances 0.000 description 1
- 210000004369 blood Anatomy 0.000 description 1
- 230000036772 blood pressure Effects 0.000 description 1
- 230000001276 controlling effect Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000000994 depressogenic effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 210000003414 extremity Anatomy 0.000 description 1
- 230000004907 flux Effects 0.000 description 1
- 239000000446 fuel Substances 0.000 description 1
- 239000008103 glucose Substances 0.000 description 1
- 230000004886 head movement Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000001939 inductive effect Effects 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 229910052760 oxygen Inorganic materials 0.000 description 1
- 239000001301 oxygen Substances 0.000 description 1
- 238000012856 packing Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 239000011347 resin Substances 0.000 description 1
- 229920005989 resin Polymers 0.000 description 1
- 230000002207 retinal effect Effects 0.000 description 1
- 206010039073 rheumatoid arthritis Diseases 0.000 description 1
- 231100000241 scar Toxicity 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 230000004382 visual function Effects 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
- A61B6/46—Arrangements for interfacing with the operator or the patient
- A61B6/461—Displaying means of special interest
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
- A61B6/46—Arrangements for interfacing with the operator or the patient
- A61B6/461—Displaying means of special interest
- A61B6/462—Displaying means of special interest characterised by constructional features of the display
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
- A61B6/52—Devices using data or image processing specially adapted for radiation diagnosis
- A61B6/5211—Devices using data or image processing specially adapted for radiation diagnosis involving processing of medical diagnostic data
- A61B6/5229—Devices using data or image processing specially adapted for radiation diagnosis involving processing of medical diagnostic data combining image data of a patient, e.g. combining a functional image with an anatomical image
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B27/0172—Head mounted characterised by optical features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
- A61B2090/372—Details of monitor hardware
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/50—Supports for surgical instruments, e.g. articulated arms
- A61B2090/502—Headgear, e.g. helmet, spectacles
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B2027/0178—Eyeglass type
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0179—Display position adjusting means not related to the information to be displayed
- G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
Definitions
- the present invention relates to a head mounted display.
- a head mounted display which is a display mounted on the head is known.
- the HMD generates image light representing an image by using, for example, a liquid crystal display and a light source, and guides the generated image light to a user's eyes by using a projection optical system or a light guiding plate, thereby allowing the user to visually recognize a virtual image.
- PTL 1 discloses a surgical imaging system and a surgical robot using such a head mounted display.
- a user (surgeon) of the head mounted display operates a surgical manipulator while visually recognizing, as a virtual image, an image with an endoscope which is inserted into a body cavity of a patient, thereby performing surgery.
- PTL 2 discloses a face fitting image display device for medical use (head mounted display).
- the face fitting display device when a user of the face fitting display device visually recognizes an endoscope image, the user directs his/her visual line in a horizontal direction and views an LCD disposed in front of a main body.
- the user visually recognizes external scenery, the user directs his/her visual line in a vertical direction and views an observation window disposed under the main body.
- the user of the head mounted display can visually recognize an endoscope image during surgery.
- the user of the head mounted display cannot directly view a surgical site (a patient's body).
- the user of the head mounted display is required to move his/her visual line in the vertical direction in order to directly view the surgical site.
- these problems are not limited to an endoscope image, and are problems common to all cases where information which is not shown in an appearance of an object is provided to a user of the head mounted display, such as, for example, a computed tomography (CT) image, a magnetic resonance imaging (MRI) image, and an X-ray image.
- CT computed tomography
- MRI magnetic resonance imaging
- X-ray image an X-ray image.
- these problems are not limited to a medical head mounted display, and are problems common to head mounted displays for use in other applications (for example, a head mounted display which is used in a construction site and provides information which is not shown in an appearance of a building to a user).
- a head mounted display which enables a real image of an object in a visual field direction and information which is not shown in an appearance of the object to be visually recognized by a user together without the user moving his/her visual line.
- An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following aspects.
- An aspect of the invention provides a head mounted display which allows a user to visually recognize a virtual image and external scenery.
- the head mounted display includes an image display unit that forms the virtual image which is visually recognized by the user; and a superimposition processing unit that causes the image display unit to form the virtual image based on superimposition information for superimposing invisible information which is not shown in an appearance of an object on the object included in the external scenery.
- the image display unit allows the user to visually recognize the virtual image based on the superimposition information which is information for superimposing the invisible information of the object on the object included in the external scenery.
- the head mounted display is a head mounted display which allows the user to visually recognize a virtual image and external scenery.
- the user of the head mounted display can visually recognize an object in the visual field direction, that is, a real image of the object included in the external scenery, and the information (superimposition information) which is not shown in the appearance of the object together, without moving the visual line of the user.
- the superimposition processing unit may cause the image display unit to repeatedly form the virtual image based on the superimposition information corresponding to the object included in the external scenery every predetermined time.
- the superimposition processing unit causes the image display unit to repeatedly form the virtual image based on the superimposition information corresponding to the object included in the external scenery every predetermined time.
- the user can visually recognize the virtual image based on the superimposition information which tracks a motion of the head of the user over time.
- the head mounted display may further include a storage unit that stores the invisible information; and an image acquisition unit that acquires an image of external scenery in a visual field direction of the user when the user wears the head mounted display.
- the superimposition processing unit may extract a feature of the object included in the external scenery through image recognition on the image in the visual field direction acquired by the image acquisition unit, and may correct the invisible information of the object stored in the storage unit on the basis of the extracted feature so as to generate the superimposition information.
- the superimposition processing unit can generate the superimposition information by using the invisible information of the object stored in the storage unit.
- the image acquired by the image acquisition unit is an image obtained by capturing a real image of the object (in other words, an object included in the external scenery) in the visual field direction of the user.
- the superimposition processing unit extracts a feature of the object included in the external scenery from the external scenery image acquired by the image acquisition unit, and generates superimposition information on the basis of the extracted feature. For this reason, the superimposition processing unit can generate the superimposition information which matches the feature of the object (the object included in the external scenery) in the visual field direction of the user.
- the invisible information may be an image indicating a structure of the object.
- the invisible information is an image indicating a structure of the object. For this reason, the user of the head mounted display can visually recognize a real image of the object in the visual field direction and the image indicating a structure of the object together, without moving the visual line of the user.
- the feature may be an edge of the object.
- the superimposition processing unit can extract a contour of the object included in the external scenery by detecting an edge of the object through image recognition. As a result, the superimposition processing unit can generate the superimposition information by correcting the invisible information of the object on the basis of the extracted contour of the object.
- the feature may be a marker attached to the object.
- the superimposition processing unit can extract a position of the object included in the external scenery by detecting markers attached to the object through the image recognition. As a result, the superimposition processing unit can generate the superimposition information by correcting the invisible information of the object on the basis of the extracted position of the object.
- the correction may be performed by performing at least one of enlargement, reduction, rotation, inversion, trimming, distortion, and noise removal on an image indicating the invisible information.
- the superimposition processing unit performs correction by performing at least one of enlargement, reduction, rotation, inversion, trimming, distortion, and noise removal on an image indicating the invisible information of the object stored in the storage unit, so as to generate the superimposition information.
- the correction is performed by dividing an image indicating the invisible information into a plurality of regions and performing at least one of enlargement, reduction, trimming, distortion, and noise removal on each of the plurality of regions.
- the superimposition processing unit performs correction by dividing an image indicating the invisible information of the object stored in the storage unit into a plurality of regions and performing at least one of enlargement, reduction, trimming, distortion, and noise removal on each of the plurality of regions.
- the superimposition processing unit divides the image indicating the invisible information into a plurality of regions and then corrects each region. Therefore, it is possible to improve accuracy of the correction of the invisible information.
- the superimposition processing unit may measure a distance between the user and each part of the object by using triangulation, and may perform the division on the basis of a result of the measurement. According to the head mounted display, the superimposition processing unit can measure a distance between the user and each part of the object by using triangulation, and can divide the region of the invisible information efficiently and in light of the actual situation on the basis of a result of the measurement.
- the superimposition processing unit may change accuracy of the correction on the basis of an information accepting characteristic indicating an extent of visual capability within a visual field of the user. According to the head mounted display, the superimposition processing unit changes accuracy of the correction of the invisible information on the basis of the information accepting characteristic within the visual field of the user. As a result, the superimposition processing unit can efficiently correct the invisible information.
- the head mounted display may further include a visual line detection unit that detects a visual line of the user.
- the image display unit may form both the virtual image corresponding to the right eye of the user and the virtual image corresponding to the left eye of the user.
- the superimposition processing unit may further obtain a steady gaze point distance which is a distance between the user and a steady gaze point at which the user steadily gazes, from the visual line detected by the visual line detection unit, and may generate the superimposition information for the right eye and the superimposition information for the left eye to which a parallax corresponding to the obtained steady gaze point distance is given.
- the superimposition processing unit generates the superimposition information for the right eye and the superimposition information for the left eye to which the parallax corresponding to a distance (steady gaze point distance) between the user and the steady gaze point at which the user gazes steadily is given, and the image display unit allows the user to visually recognize the virtual image based on this superimposition information. As a result, the user can visually recognize the superimposition information at a position of the same distance as the steady gaze point.
- the head mounted display may further include parallax information that correlates a plurality of distances between the user and the virtual image visually recognized by the user with parallaxes which respectively correspond to the plurality of distances.
- the superimposition processing unit may give a parallax corresponding to the steady gaze point distance by using the parallax information. According to the head mounted display, the superimposition processing unit can easily generate the superimposition information for the right eye and the superimposition information for the left eye by using the parallax information which is prepared in advance.
- the parallax information may be provided in a plurality of pieces, and, here, the superimposition processing unit may select one parallax information piece from the plurality of parallax information pieces in accordance with the user. According to the head mounted display, the superimposition processing unit selects one parallax information piece from a plurality of parallax information pieces in accordance with the user. For this reason, the superimposition processing unit can select parallax information matching the characteristics (for example, the dominant eye or the interocular distance) of the user, thereby reducing visual discomfort which the user feels.
- the characteristics for example, the dominant eye or the interocular distance
- the head mounted display may further include a position detection unit that detects a position of the user relative to the object, and, here, the storage unit may store a plurality of invisible information pieces regarding a single object.
- the superimposition processing unit may select one invisible information piece corresponding to the position detected by the position detection unit from the plurality of invisible information pieces stored in the storage unit, and may correct the selected invisible information.
- the superimposition processing unit can select one invisible information piece corresponding to the position of the user relative to the object from the plurality of invisible information pieces stored in the storage unit, and can correct the selected invisible information. For this reason, the superimposition processing unit can generate the superimposition information by using appropriate invisible information on the basis of the positional relationship between the user and the object.
- the superimposition processing unit may further generate enlargement information in which at least a part of the object included in the external scenery is extracted and enlarged, from the image in the visual field direction acquired by the image acquisition unit.
- the image display unit may allow the user to visually recognize the virtual image based on the superimposition information and the enlargement information. According to the head mounted display, the user of the head mounted display can simultaneously visually recognize the enlargement information of at least a part of the object in the visual field direction, in addition to the real image of the object (in other words, the object included in the external scenery) in the visual field direction and the information (the superimposition information) which is not shown in the appearance of the object.
- the superimposition processing unit may not display the virtual image in response to a request from the user.
- the user of the head mounted display can cause the virtual image not to be displayed at a user's own will. For this reason, it is possible to improve convenience to the user of the head mounted display.
- the head mounted display may further include a procedure management unit that generates procedure information which is information regarding content which is to be performed by the user in a current procedure, and, here, the image display unit may allow the user to visually recognize the virtual image based on the superimposition information and the procedure information. According to the head mounted display, the image display unit allows the user to visually recognize the virtual image based on the procedure information which indicates content to be performed by the user in the current procedure. For this reason, it is possible to support work performed by the user.
- the procedure information may include at least one of an image of a scene in which a third party exemplarily performs the procedure and an image indicating content to be performed by the user when the individual specific procedure is performed.
- the head mounted display it is possible to visually show content to be performed by the user, and thus, it is possible to more intelligibly support work performed by the user.
- the image indicating content to be performed by the user when the individual specific procedure is performed is shown, it is possible to support work performed by the user more individually and specifically, and thus it is possible to improve accuracy and efficiency of the work.
- the superimposition processing unit may further generate auxiliary information which is additional information for assisting work of the user.
- the image display unit may allow the user to visually recognize the virtual image based on the superimposition information and the auxiliary information. According to the head mounted display, the user visually recognizes the virtual image based on the information for assisting work of the user. For this reason, it is possible to improve efficiency of work using the head mounted display.
- the superimposition processing unit may further perform a color conversion process for improving visibility of the generated superimposition information on the basis of the image in the visual field direction acquired by the image acquisition unit. According to the head mounted display, the superimposition processing unit performs a color conversion process based on the image in the visual field direction of the user on the superimposition information. For this reason, it is possible to further improve visibility of the virtual image for the user of the head mounted display.
- the superimposition processing unit may withdraw the superimposition information which is displayed as the virtual image to a position which does not hinder the visual field of the user, in response to a request from the user.
- the user of the head mounted display can withdraw the superimposition information which is displayed as the virtual image at a user's own will. For this reason, it is possible to improve convenience to the user of the head mounted display.
- the superimposition processing unit may generate the superimposition information corresponding to an overlap region in which a display region of the virtual image overlaps with an acquisition region of the image acquisition unit in the image acquired by the image acquisition unit. According to the head mounted display, the superimposition processing unit generates the superimposition information corresponding to an overlap region in which a display region of the virtual image overlaps with an acquisition region of the image acquisition unit. For this reason, it is possible to reduce the occurrence of "mismatch between an image which is directly viewed by the user in his/her visual field and a virtual image which is allowed to be visually recognized by the image display unit", which is a problem occurring in the transmission type head mounted display. As a result, when the user visually recognizes a virtual image based on the superimposition information, it is possible to reduce the discomfort which the user feels.
- the procedure management unit may generate the procedure information corresponding to an overlap region in which a display region of the virtual image overlaps with an acquisition region of the image acquisition unit in the image acquired by the image acquisition unit. According to the head mounted display, the procedure management unit generates the procedure information corresponding to an overlap region in which a display region of the virtual image overlaps with an acquisition region of the image acquisition unit. For this reason, it is possible to reduce the occurrence of "mismatch between an image which is directly viewed by the user in his/her visual field and a virtual image which is allowed to be visually recognized by the image display unit", which is a problem occurring in the transmission type head mounted display. As a result, when the user visually recognizes a virtual image based on the procedure information, it is possible to reduce the discomfort which the user feels.
- All of the plurality of constituent elements in the respective aspects of the invention described above are not essential, and some of the plurality of constituent elements may be changed, deleted, exchanged with other new constituent elements, and partially deleted from limited content thereof, as appropriate, in order to solve some or all of the above-described problems or in order to achieve some or all of the effects described in the present specification.
- some or all of the technical features included in one aspect of the invention described above may be combined with some or all of the technical features included in another aspect of the invention described above, and as a result may be treated as an independent aspect of the invention.
- one aspect of the invention may be realized as a device which includes either one or both of the two constituent elements, the superimposition processing unit and the image display unit.
- this device may or may not include the superimposition processing unit.
- the device may or may not include the image display unit.
- This device may be realized as, for example, a head mounted display, but may be realized as devices other than the head mounted display.
- the invention may be realized in various aspects.
- the invention may be realized in aspects such as a head mounted display and a control method for the head mounted display, a medical work supporting device and a control method for the medical work supporting device, a medical work supporting system, a computer program for realizing functions of the methods, the devices, or the system, and a recording medium recording the computer program thereon.
- Fig. 1 is a diagram illustrating a schematic configuration of a head mounted display according to an embodiment of the invention.
- Fig. 2 is a functional block diagram illustrating a configuration of a head mounted display.
- Fig. 3 is a diagram illustrating a state in which image light is emitted by an image light generation portion.
- Fig. 4 is a diagram illustrating an example of a virtual image recognized by a user.
- Fig. 5 is a flowchart illustrating procedures of a superimposition information display process according to a first embodiment.
- Fig. 6 is a diagram illustrating an example of invisible information.
- Fig. 7 is a diagram illustrating an example of appearance information.
- Fig. 8 is a flowchart illustrating procedures of a superimposition information generation process.
- Fig. 1 is a diagram illustrating a schematic configuration of a head mounted display according to an embodiment of the invention.
- Fig. 2 is a functional block diagram illustrating a configuration of a head mounted display.
- FIG. 9 is a diagram illustrating a feature (edge) of a target object extracted from an external scenery image by using a method i.
- Fig. 10 is a diagram illustrating a feature (edge) of the target object extracted from invisible information by using the method i.
- Fig. 11 is a diagram illustrating an example of corrected invisible information.
- Fig. 12A is a diagram illustrating another example of a virtual image recognized by the user.
- Fig. 12B is a diagram illustrating another examples of a virtual image recognized by the user.
- Fig. 13 is a functional block diagram illustrating a configuration of a head mounted display according to a second embodiment.
- Fig. 14 is a diagram illustrating an example of a procedure table.
- Fig. 15 is a flowchart illustrating procedures of a superimposition information display process according to the second embodiment.
- Fig. 16 is a diagram illustrating an example of procedure information.
- Fig. 17 is a diagram illustrating an example of superimposition information to which the procedure information is added.
- Fig. 18 is a diagram illustrating an example of a virtual image visually recognized by a user due to procedures e1 to e9.
- Fig. 19 is a diagram illustrating a schematic configuration of a head mounted display according to a third embodiment.
- Fig. 20 is a functional block diagram illustrating a configuration of a head mounted display according to the third embodiment.
- Fig. 21 is a diagram for explaining parallax information.
- Fig. 16 is a diagram illustrating an example of procedure information.
- Fig. 17 is a diagram illustrating an example of superimposition information to which the procedure information is added.
- Fig. 18 is a diagram illustrating an example of a virtual image visually recognized by a user
- Fig. 22 is a flowchart illustrating procedures of a superimposition information display process according to the third embodiment.
- Fig. 23 is a diagram illustrating an example of invisible information.
- Fig. 24 is a flowchart illustrating procedures of a first superimposition information generation process.
- Fig. 25 is a diagram for explaining step S404 of the first superimposition information generation process.
- Fig. 26 is a flowchart illustrating procedures of a second superimposition information generation process.
- Fig. 27 is a diagram for explaining step S504 of the second superimposition information generation process.
- Fig. 28A is a diagram illustrating an exterior configuration of a head mounted display in a modification example.
- Fig. 28B is a diagram illustrating an exterior configuration of a head mounted displays in a modification example.
- FIG. 1 is a diagram illustrating a schematic configuration of a head mounted display (HMD) according to an embodiment of the invention.
- the head mounted display 100 is a display mounted on the head.
- the head mounted display 100 according to the present embodiment is an optical transmission type head mounted display which allows a user to visually recognize a virtual image and also to directly visually recognize external scenery.
- a type of head mounted display 100 which allows the user to visually recognize an image as a virtual image is exemplified.
- the head mounted display 100 includes an image display section 20 which allows the user to visually recognize a virtual image in a state of being mounted on the head of the user, and a control section (a controller ) 10 which controls the image display section 20.
- the image display section 20 is a mounting body which is mounted on the head of the user, and has a glasses shape in the present embodiment.
- the image display section 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a camera 61.
- the right optical image display unit 26 and the left optical image display unit 28 are respectively disposed so as to be located in front of the right and left eyes of the user when the user wears the image display section 20.
- One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at the position corresponding to the glabella of the user when the user wears the image display section 20.
- the right holding unit 21 is a member which is provided so as to extend over a position corresponding to the temporal region of the user when the user wears the image display section 20, from an end part ER which is the other end of the right optical image display unit 26.
- the left holding unit 23 is a member which is provided so as to extend over a position corresponding to the temporal region of the user when the user wears the image display section 20, from an end part EL which is the other end of the left optical image display unit 28.
- the right holding unit 21 and the left holding unit 23 hold the image display section 20 on the head of the user in the same manner as temples of glasses.
- the right display driving unit 22 is disposed inside the right holding unit 21, that is, on a side opposing the head of the user when the user wears the image display section 20.
- the left display driving unit 24 is disposed inside the left holding unit 23.
- the right holding unit 21 and the left holding unit 23 are also collectively referred to as “holding units”
- the right display driving unit 22 and the left display driving unit 24 are also collectively referred to as “display driving units”
- the right optical image display unit 26 and the left optical image display unit 28 are also collectively referred to as "optical image display units”.
- the display driving units 22 and 24 include liquid crystal displays (hereinafter, referred to as "LCDs") 241 and 242, projection optical systems 251 and 252, and the like (refer to Fig. 2). Details of configurations of the display driving units will be described.
- the optical image display units as optical members include light guiding plates 261 and 262 (refer to Fig. 2) and dimming plates.
- the light guiding plates 261 and 262 are made of a light-transmitting resin material or the like, and guide image light output from the display driving units 22 and 24 to the eyes of the user.
- the dimming plates are thin plate-shaped optical elements, and are disposed so as to cover a surface side of the image display section 20 (a side opposite the eye sides of the user).
- the dimming plates protect the light guiding plates 261 and 262 so as to prevent the light guiding plates 261 and 262 from being damaged, polluted, or the like.
- light transmittance of the dimming plates is adjusted so as to adjust an amount of external light entering the eyes of the user, thereby controlling an extent of visually recognizing a virtual image. Further, the dimming plates may be omitted.
- the camera 61 is disposed at the position corresponding to the glabella of the user when the user wears the image display section 20.
- the camera 61 captures an image of external scenery in the surface side direction of the image display section 20, that is, in a visual field direction of the user when the user wears the head mounted display 100, thereby obtaining an external scenery image.
- the camera 61 is a so-called visible light camera, and an external scenery image acquired by the camera 61 is an image which represents a shape of an object from visible light radiated from the object.
- the camera 61 in the present embodiment is a monocular camera, but may be stereo camera.
- the camera 61 corresponds to an "image acquisition unit" in the appended claims, and the external scenery image acquired by the camera 61 corresponds to an "image in a visual field direction of a user" in the appended claims.
- the image display section 20 further includes a connection unit 40 which connects the image display section 20 to the control section 10.
- the connection unit 40 includes a main body cord 48 connected to the control section 10, a right cord 42, a left cord 44, and a connection member 46.
- the right cord 42 and the left cord 44 are two cords into which the main body cord 48 branches out, and the connection member 46 is provided at the branch point.
- the right cord 42 is inserted into a casing of the right holding unit 21 from an apex AP in the extending direction of the right holding unit 21, and is connected to the right display driving unit 22.
- the left cord 44 is inserted into a casing of the left holding unit 23 from an apex AP in the extending direction of the left holding unit 23, and is connected to the left display driving unit 24.
- the connection member 46 has a jack for connection of an earphone plug 30.
- a right earphone 32 and a left earphone 34 extend from the earphone plug 30.
- the image display section 20 and the control section 10 transmit various signals via the connection unit 40.
- An end part of the main body cord 48 on a side opposite the connection member 46, and the control section 10 are respectively provided with connectors (not illustrated) fitted into each other.
- the connector of the main body cord 48 and the connector of the control section 10 are fitted into or released from each other, and thus the control section 10 is connected to or disconnected from the image display section 20.
- a metal cable or an optical fiber may be used as the right cord 42, the left cord 44, and the main body cord 48.
- the control section 10 is a device used to control the head mounted display 100.
- the control section 10 includes a lighting unit 12, a touch pad 14, a cross key 16, and a power switch 18.
- the lighting unit 12 indicates an operation state (for example, ON and OFF of power, or the like) of the head mounted display 100 by using a light emitting state thereof.
- a light emitting diode (LED) may be used as the lighting unit 12.
- the touch pad 14 detects a contact operation on an operation surface of the touch pad 14 so as to output a signal based on detected content.
- Various touch pads of a capacitance type, a pressure detection type, and an optical type may be employed as the touch pad 14.
- the cross key 16 detects a pushing operation on keys corresponding to vertical and horizontal directions so as to output a signal based on detected content.
- the power switch 18 detects a sliding operation of the switch so as to change a power supply state of the head mounted display 100.
- Fig. 2 is a functional block diagram illustrating a configuration of the head mounted display 100.
- the control section 10 includes an input information acquisition unit 110, a storage unit 120, a power supply 130, a wireless communication unit 132, a CPU 140, an interface 180, transmission units (Tx) 51 and 52, and the above-described constituent elements are connected to each other via a bus (not illustrated).
- the input information acquisition unit 110 acquires a signal based on an operation input which is performed on, for example, the touch pad 14, the cross key 16, the power switch 18, or the like.
- the input information acquisition unit 110 may acquire an operation input from the user by using various methods. For example, not only an operation input using the touch pad 14 or the cross key 16 illustrated in Fig. 2 but also an operation input using a foot switch (a switch operated by the leg of the user) may be acquired.
- a visual line detection unit such as an infrared sensor may be provided in the image display section 20, then a visual line of the user is detected, and an operation input based on a command correlated with a motion of the visual line may be acquired.
- a gesture of the user may be detected using the camera 61, and an operation input based on a command correlated with the gesture may be acquired.
- the fingertip of the user, a ring attached to the hand of the user, or a medical instrument held by the user may be used as a mark for detecting a motion.
- the input information acquisition unit 110 can acquire an operation input from the user even in work (for example, surgery) in which it is difficult for the user to release the user's hand.
- the storage unit 120 is constituted by a ROM, a RAM, a DRAM, a hard disk, or the like.
- the storage unit 120 includes an invisible information storage portion 122 and an appearance information storage portion 124.
- the invisible information storage portion 122 is a storage region which preliminarily stores invisible information of an object which is a target of a superimposition information display process.
- the "invisible information” indicates information which is not shown in an appearance of an object.
- the invisible information in the present embodiment is an image indicating a structure of an object.
- the invisible information is, for example, a computed tomography (CT) image, a magnetic resonance imaging (MRI) image, an X-ray image, a fluoroscopic image, an ultrasonic diagnostic image, an endoscope image, a thermography image, or the like.
- CT computed tomography
- MRI magnetic resonance imaging
- X-ray image X-ray image
- fluoroscopic image fluoroscopic image
- ultrasonic diagnostic image an endoscope image
- thermography image or the like.
- the invisible information is, for example, a design drawing, a circuit diagram, an X-ray image, a fluoroscopic image, an infrared image, a thermography image, or the like.
- the appearance information storage portion 124 is a storage region which preliminarily stores appearance information of an object which is a target of the superimposition information display process.
- the "appearance information" in the present embodiment is an image which indicates a state (appearance) of an object viewed from the outside, and is a picture or the like obtained by a visible light camera.
- the power supply 130 supplies power to the respective units of the head mounted display 100.
- a secondary battery may be used as the power supply 130.
- the wireless communication unit 132 performs wireless communication with other apparatuses in accordance with a predetermined wireless communication standard such as a wireless LAN or Bluetooth.
- the CPU 140 reads and executes the computer programs stored in the storage unit 120 so as to function as an operating system (OS) 150, an image processing unit 160, a sound processing unit 170, a display control unit 190, a superimposition processing unit 142.
- the superimposition processing unit 142 is an application program which executes a superimposition information display process.
- the superimposition information display process is a process of superimposing information which is not shown in an appearance of an object on a real image of the object in a visual field direction of the user for display.
- the image processing unit 160 generates signals on the basis of content (video) which is input via the interface 180 or the wireless communication unit 132. In addition, the image processing unit 160 supplies the generated signals to the image display section 20 via the connection unit 40.
- the signals supplied to the image display section 20 are different in cases of an analog format and a digital format.
- the image processing unit 160 generates and transmits a clock signal PCLK, a vertical synchronization signal VSync, a horizontal synchronization signal HSync, and image data Data.
- the image processing unit 160 acquires an image signal included in the content.
- the acquired image signal is an analog signal which is generally formed by thirty frame images per second, for example, in a case of a moving image.
- the image processing unit 160 separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the acquired image signal, and generates the clock signal PCLK by using a phase locked loop (PLL) circuit or the like (not illustrated) on the basis of a cycle of the separated vertical synchronization signal VSync or horizontal synchronization signal HSync.
- the image processing unit 160 converts an analog image signal from which the synchronization signals are separated into a digital image signal by using an A/D conversion circuit or the like.
- the image processing unit 160 stores the converted digital image signal in a DRAM of the storage unit 120 for each frame as image data Data of RGB data.
- the image processing unit 160 generates and transmits the clock signal PCLK and the image data Data. Specifically, in a case where the content has the digital format, since the clock signal PCLK is output in synchronization with the image signal, the generation of the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the A/D conversion of the analog image signal are not necessary. Further, the image processing unit 160 may perform, on the image data Data stored in the storage unit 120, image processes including a resolution conversion process, various color tone correction processes such as adjustment of luminance and color saturation, a keystone correction process, and the like.
- the image processing unit 160 transmits each of the generated clock signal PCLK, vertical synchronization signal VSync and horizontal synchronization signal HSync, and the image data Data stored in the DRAM of the storage unit 120, via the transmission units 51 and 52.
- the image data Data which is transmitted via the transmission unit 51 is referred to as "right eye image data”
- the image data Data which is transmitted via the transmission unit 52 is referred to as "left eye image data”.
- the transmission units 51 and 52 function as a transceiver for serial transmission between the control section 10 and the image display section 20.
- the display control unit 190 generates control signals for control of the right display driving unit 22 and the left display driving unit 24. Specifically, by using the control signals, the display control unit 190 separately controls a right LCD control portion 211 to turn on and off driving of a right LCD 241, a right backlight control portion 201 to turn on and off driving of a right backlight 221, a left LCD control portion 212 to turn on and off driving of a left LCD 242, and a left backlight control portion 202 to turn on and off driving of a left backlight 222. Accordingly, the display control unit 190 controls each of the right display driving unit 22 and the left display driving unit 24 to generate and emit image light.
- the display control unit 190 causes both of the right display driving unit 22 and the left display driving unit 24 to generate image light, causes either thereof to generate image light, or causes neither thereof to generate image light.
- the display control unit 190 transmits control signals for the right LCD control portion 211 and the left LCD control portion 212 thereto via the transmission units 51 and 52, respectively.
- the display control unit 190 transmits control signals for the right backlight control portion 201 and the left backlight control portion 202 thereto, respectively.
- the sound processing unit 170 acquires an audio signal included in the content so as to amplify the acquired audio signal, and supplies the amplified audio signal to a speaker (not illustrated) of the right earphone 32 connected to the connection member 46 and a speaker (not illustrated) of the left earphone 34 connected thereto.
- the audio signal is processed, and, thus, for example, different sounds of which frequencies are changed are respectively output from the right earphone 32 and the left earphone 34.
- the interface 180 is an interface which connects various external apparatuses OA which are content supply sources to the control section 10.
- the external apparatuses OA include, for example, a personal computer (PC), a mobile phone terminal, a gaming terminal, and the like.
- PC personal computer
- a mobile phone terminal for example, an Apple MacBook Air (CPU), a smart phone terminal, a smart phone terminal, and the like.
- USB interface for example, an USB interface, a micro-USB interface, a memory card interface, and the like may be used as the interface 180.
- the image display section 20 includes the right display driving unit 22, the left display driving unit 24, the right light guiding plate 261 as the right optical image display unit 26, the left light guiding plate 262 as the left optical image display unit 28, and the camera 61.
- the right display driving unit 22 includes a reception portion (Rx) 53, the right backlight (BL) control portion 201 and the right backlight (BL) 221 which function as a light source, the right LCD control portion 211 and the right LCD 241 which function as a display element, and a right projection optical system 251.
- the right backlight control portion 201, the right LCD control portion 211, the right backlight 221, and the right LCD 241 are collectively referred to as an "image light generation portion".
- the reception portion 53 functions as a receiver for serial transmission between the control section 10 and the image display section 20.
- the right backlight control portion 201 drives the right backlight 221 on the basis of an input control signal.
- the right backlight 221 is a light emitting body such as, for example, an LED or an electroluminescent element (EL).
- the right LCD control portion 211 drives the right LCD 241 on the basis of the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right eye image data Data1, which are input via the reception portion 53.
- the right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are disposed in a matrix.
- the right projection optical system 251 is constituted by a collimator lens which converts image light emitted from the right LCD 241 into parallel beams of light flux.
- the right light guiding plate 261 as the right optical image display unit 26 reflects image light output from the right projection optical system 251 along a predetermined light path, so as to guide the image light to the right eye RE of the user.
- the optical image display unit may employ any type as long as a virtual image is formed in front of the eyes of the user by using image light, and, for example, a diffraction grating may be used. In addition, a semi-transmissive reflective film may be used.
- the left display driving unit 24 has the same configuration as the right display driving unit 22.
- the left display driving unit 24 includes a reception portion (Rx) 54, the left backlight (BL) control portion 202 and the left backlight (BL) 222 which function as a light source, the left LCD control portion 212 and the left LCD 242 which function as a display element, and a left projection optical system 252. Detailed description thereof will be omitted.
- Fig. 3 is a diagram illustrating a state in which image light is emitted by the image light generation portion.
- the right LCD 241 drives liquid crystal at respective pixel positions disposed in a matrix so as to change a transmittance of light transmitted through the right LCD 241, thereby modulating illumination light IL applied from the right backlight 221 into effective image light PL representing an image. This is also the same for the left side.
- the backlight type is employed in the present embodiment, but there may be a configuration in which image light is emitted using a front light type or a reflective type.
- Fig. 4 is a diagram illustrating an example of a virtual image recognized by the user.
- Fig. 4 exemplifies a visual field VR of the user.
- the image light which is guided to both eyes of the user of the head mounted display 100 forms an image on the retinas of the user, and thus the user visually recognizes a virtual image VI.
- the virtual image VI is an X-ray image of a patient's hand.
- the user visually recognizes an external scenery SC which is transmitted through the right optical image display unit 26 and the left optical image display unit 28.
- Fig. 4 is a diagram illustrating an example of a virtual image recognized by the user.
- Fig. 4 exemplifies a visual field VR of the user.
- the image light which is guided to both eyes of the user of the head mounted display 100 forms an image on the retinas of the user, and thus the user visually recognizes a virtual image VI.
- the virtual image VI is an X-ray image of a patient's hand.
- the user visually recognizes
- the external scenery SC is a state in which a cutting edge of a scalpel held by a surgeon is in contact with the second joint of the little finger of the patient's right hand.
- the user can visually recognize the external scenery SC on the rear side of the virtual image VI not only in a part which does not overlap with the virtual image VI, but also in a part which overlaps with the virtual image VI.
- the head mounted display according to the present embodiment may allow the user to visually recognize a three-dimensional virtual image by using a side-by-side type.
- a mode hereinafter, also referred to as a "three-dimensional display mode" in which the user is allowed to visually recognize a three-dimensional virtual image
- the LCD is equally divided into the left and right, the right half is used for display for the right eye, and the left half is used for display for the left eye.
- other types such as, for example, a frame packing type and a top-and-bottom type may be used to realize the three-dimensional display mode.
- Superimposition information display process Fig. 5 is a flowchart illustrating procedures of a superimposition information display process according to a first embodiment. Steps S102 and S104 of Fig. 5 indicate procedures of a preprocess of the superimposition information display process. The procedures from step S106 of Fig. 5 indicate procedures of the superimposition information display process.
- the superimposition information display process is a process of superimposing information which is no shown in an appearance of an object on a real image of the object in a visual field direction of the user for display.
- the superimposition information display process includes a method of extracting a feature of an object (hereinafter, referred to as a "target object") which is a target of the superimposition information display process through edge detection, and a method of extracting a feature of the target object through maker detection (details thereof will be described later).
- a target object which is a target of the superimposition information display process through edge detection
- a method of extracting a feature of the target object through maker detection (details thereof will be described later).
- markers are preliminarily attached to the target object in steps S102, S104 and S108.
- markers attached to a target object various types of markers may be used. For example, a tape, a seal, a magic marker, a laser marker, a Magic Tape (registered trademark), and the like may be used. Further, any number of markers attached to a target object may be set.
- the user stores invisible information of an object which is a target of the superimposition information display process in the invisible information storage portion 122.
- the user performs image capturing on an object which is a target of the superimposition information display process by using image capturing apparatuses (a CT apparatus, an MRI apparatus, an X-ray apparatus, an endoscope, a thermography apparatus, and the like), so as to acquire an image (invisible information) indicating a structure of the target object.
- image capturing apparatuses a CT apparatus, an MRI apparatus, an X-ray apparatus, an endoscope, a thermography apparatus, and the like.
- the user stores the acquired image in the invisible information storage portion 122.
- step S102 of the superimposition information display process preferably, an image matching a direction of the target object, expected in the superimposition information display process in and after step S106, is acquired and stored in the invisible information storage portion 122.
- invisible information received from an external device may be stored in the invisible information storage portion 122 instead of capturing an image of the target object.
- Fig. 6 is a diagram illustrating an example of invisible information.
- the invisible information SI exemplified in Fig. 6 is an X-ray image of a person's right hand.
- the target object is a "person's right hand”. It can be seen from the invisible information SI of Fig. 6 that the second joint of the little finger of the person's right hand is deformed due to rheumatoid arthritis.
- step S104 of Fig. 5 the user stores appearance information of the object which is a target of the superimposition information display process in the appearance information storage portion 124. Specifically, the user performs image capturing on the target object by using the camera 61 or a digital camera, thereby acquiring an image of an appearance of the target object. In addition, the user stores the acquired image in the appearance information storage portion 124.
- Fig. 7 is a diagram illustrating an example of the appearance information.
- the appearance information FI exemplified in Fig. 7 is a picture image of a person's right hand.
- the target object is the same as in Fig. 6.
- the superimposition processing unit 142 determines whether or not there is an instruction for starting the superimposition information display process. Specifically, in a case where an instruction for starting the superimposition information display process is received from the input information acquisition unit 110, the superimposition processing unit 142 determines that there is a starting instruction (step S106: YES), and in a case where the instruction is not received, the superimposition processing unit determines that there is no starting instruction (step S106: NO). If there is no starting instruction (step S106: NO), the superimposition processing unit 142 causes the process to transition to step S106 so as to wait for a starting instruction to be received. On the other hand, if there is a starting instruction (step S106: YES), the superimposition processing unit 142 generates superimposition information through a superimposition information generation process described next (step S108).
- FIG. 8 is a flowchart illustrating procedures of the superimposition information generation process.
- the superimposition processing unit 142 acquires an external scenery image (step S202). Specifically, the superimposition processing unit 142 activates the camera 61 so as to instruct the camera 61 to capture an image, and acquires an external scenery image captured by the camera 61.
- the superimposition processing unit 142 After the external scenery image is acquired, the superimposition processing unit 142 performs image recognition on the external scenery image so as to extract a feature of the target object (step S204). Specifically, the superimposition processing unit 142 extracts a feature of the target object included in the external scenery image by using image recognition methods such as the following methods i and ii. In addition, the methods i and ii may be combined with each other.
- the superimposition processing unit 142 detects an edge of the target object. In this case, the detected edge corresponds to a "feature of the target object”.
- Method ii) The superimposition processing unit 142 detects markers attached to the target object. In this case, the detected markers correspond to a "feature of the target object”.
- Fig. 9 illustrates a feature (edge) of the target object extracted from the external scenery image by using the method i.
- the superimposition processing unit 142 performs image recognition on invisible information SI so as to extract a feature of the target object. Specifically, the superimposition processing unit 142 reads invisible information SI corresponding to the target object from the invisible information storage portion 122. The superimposition processing unit 142 extracts a feature of the target object included in the read invisible information SI by using the image recognition methods described in the above methods i and ii. In addition, in order to improve correction accuracy in step S208, image recognition methods used in steps S204 and S206 are preferably unified into one.
- Fig. 10 illustrates a feature (edge) of the target object extracted from the invisible information SI by using the method i.
- the superimposition processing unit 142 corrects the invisible information SI so that the feature of the external scenery image extracted in step S204 and the feature of the invisible information SI extracted in step S206 match each other. Specifically, in a case where the method i is used in steps S204 and S206, the superimposition processing unit 142 performs at least one processing of enlargement, reduction, rotation, inversion, trimming, distortion, and removal of noise, on the image indicating the invisible information SI so that the feature (that is, the edge detected in step S204) of the external scenery image and the feature (that is, the edge detected in step S206) of the invisible information SI match each other in contours or characteristic sites thereof.
- the characteristic sites are the joints, tips of the limbs, the blood vessels, the bones, and the like in a case where the target object is a living body.
- the characteristic sites are protrusions, corners, and the like in a case where the target object is an artifact.
- the superimposition processing unit 142 performs at least one processing of enlargement, reduction, rotation, inversion, trimming, distortion, and removal of noise, on the image indicating the invisible information SI so that positions of the feature (that is, the marker detected in step S204) of the external scenery image and the feature (that is, the marker detected in step S206) of the invisible information SI match each other.
- the superimposition processing unit 142 performs correction on the invisible information SI so that the contours and the characteristic sites (the joints, the fingertips, the blood vessels, and the bones) of the hand match each other in the edge (Fig. 9) detected in step S204 and the edge (Fig. 10) detected in step S206.
- Fig. 11 is a diagram illustrating an example of the corrected invisible information SI.
- the invisible information SI (Fig. 6) is trimmed so as to match a range of the target object (that is, the patient's right hand) included in the external scenery image.
- the invisible information SI (Fig. 6) is reduced so as to match the size of the target object included in the external scenery image, and the invisible information SI (Fig. 6) is rotated and moved so as to match the position of the target object included in the external scenery image.
- the invisible information SI corrected in step S208 corresponds to "superimposition information" in the appended claims.
- the superimposition processing unit 142 can generate the superimposition information by using the invisible information SI of the object stored in the invisible information storage portion 122.
- the external scenery image acquired by the camera 61 (image acquisition unit) in step S202 is an image obtained by capturing a real image of the target object (in other words, an object included in the external scenery) in the visual field direction of the user.
- the superimposition processing unit 142 extracts an edge or markers (a feature of the target object) of the target object included in the external scenery from the external scenery image acquired by the camera 61, and generates superimposition information on the basis of the extracted feature. For this reason, the superimposition processing unit 142 can generate the superimposition information which matches the feature of the target object (the object included in the external scenery) in the visual field direction of the user.
- the superimposition processing unit 142 can extract a contour of the target object included in the external scenery by detecting an edge of the target object through the image recognition. As a result, the superimposition processing unit 142 can generate the superimposition information CP1 by correcting the invisible information SI of the target object on the basis of the extracted contour of the target object.
- the superimposition processing unit 142 can extract a position of the target object included in the external scenery by detecting markers attached to the target object through the image recognition. As a result, the superimposition processing unit 142 can generate the superimposition information CP1 by correcting the invisible information SI of the target object on the basis of the extracted position of the target object.
- the superimposition processing unit 142 displays the superimposition information CP1 in step S110 of Fig. 5. Specifically, the superimposition processing unit 142 transmits the superimposition information CP1 generated due to the superimposition information generation process to the image processing unit 160.
- the image processing unit 160 having received the superimposition information CP1 performs the display process described in Fig. 2 on the superimposition information CP1.
- the superimposition information CP1 is displayed as the virtual image VI in the visual field VR of the user of the head mounted display 100.
- the head mounted display 100 according to the present embodiment is an optical transmission type head mounted display which allows the user to visually recognize a virtual image and also to visually recognize external scenery.
- the user of the head mounted display 100 can visually recognize the real image (a state in which the cutting edge of the scalpel held by the surgeon is in contact with the second joint of the little finger of the patient's right hand) in the external scenery SC which is viewed through the right optical image display unit 26 and the left optical image display unit 28 and the superimposition information CP1 which is viewed as the virtual image VI, in a superimposition state.
- the superimposition processing unit 142 determines whether or not there is an instruction for changing display, and determines what is content of a changing instruction. In a case where there is no display changing instruction, the superimposition processing unit 142 causes the process to proceed to step S108 after waiting for a predetermined time (not illustrated). Accordingly, since capturing of an external scenery image and generation and display of the superimposition information CP1 are repeatedly performed for each predetermined time, it is possible to display the superimposition information CP1 which tracks a motion of the head of the user over time, as the virtual image VI. In addition, the predetermined time may be arbitrarily set.
- the superimposition processing unit 142 When an "image changing" instruction for changing the display of the superimposition information CP1 which is currently displayed is received from the input information acquisition unit 110 (step S112: image changing), the superimposition processing unit 142 causes the process to proceed to step S114.
- the superimposition processing unit 142 generates another superimposition information piece (step S114). Specifically, the superimposition processing unit 142 performs again the superimposition information generation process described in Fig. 8 so as to generate another superimposition information piece.
- the superimposition processing unit 142 uses the "appearance information FI stored in the appearance information storage portion 124" as a processing target instead of the "invisible information SI stored in the invisible information storage portion 122" in the description of Fig. 8.
- another superimposition information piece is information indicating an appearance of the target object.
- the superimposition processing unit 142 displays another superimposition information piece (step S116). Specifically, the superimposition processing unit 142 transmits another superimposition information piece to the image processing unit 160. The image processing unit 160 having received another superimposition information piece performs the display process described in Fig. 2 on another superimposition information piece. As a result, instead of the superimposition information CP1, another superimposition information piece is displayed as the virtual image VI in the visual field VR of the user of the head mounted display 100. Subsequently, the superimposition processing unit 142 causes the process to proceed to step S112.
- the user of the head mounted display 100 can change the display of superimposition information generated on the basis of the invisible information SI indicating a structure of the target object and the display of another superimposition information piece generated on the basis of the appearance information FI indicating an appearance of the target object, at user's own will. For this reason, it is possible to improve convenience to the user of the head mounted display 100.
- step S112 when a "non-display" instruction for not displaying the virtual image which is currently displayed is received from the input information acquisition unit 110 (step S112: non-display), the superimposition processing unit 142 causes the process to proceed to step S118.
- the superimposition processing unit 142 causes the virtual image VI (the superimposition information CP1 or another superimposition information piece) which is currently displayed not to be displayed (step S118). Specifically, the superimposition processing unit 142 stops transmission of the superimposition information CP1 or another superimposition information piece to the image processing unit 160. Accordingly, image data to be displayed is removed, and thus the virtual image VI is not displayed.
- the superimposition processing unit 142 may transmit a request for not displaying an image to the display control unit 190.
- the display control unit 190 may transmit a control signal for causing the LCD control portions to turn off driving of the LCDs or a control signal for causing the backlight control portions to turn off driving of the backlights, thereby not displaying the virtual image VI.
- the user of the head mounted display 100 can cause the virtual image VI which is currently displayed not to be displayed at user's own will. For example, in a case where the user wants to keep an eye on a real image which is viewed through the optical image display units, the user can erase the virtual image VI on purpose. For this reason, it is possible to improve convenience to the user of the head mounted display 100.
- step S112 when an "enlargement" instruction for enlarging and displaying the real image in the visual field direction is received from the input information acquisition unit 110 (step S112: enlargement), the superimposition processing unit 142 causes the process to proceed to step S120.
- the superimposition processing unit 142 acquires an external scenery image (step S120). Specifically, the superimposition processing unit 142 activates the camera 61 so as to instruct the camera 61 to capture an image, and acquires an external scenery image captured by the camera 61.
- the superimposition processing unit 142 After the external scenery image is acquired, the superimposition processing unit 142 performs image recognition on the external scenery image acquired in step S120, and trims a region in a predetermined range centering on a point which is designated along with the enlargement instruction so as to enlarge the trimmed image (step S122). Next, the superimposition processing unit 142 generates enlargement information in which the enlarged image is disposed in an end part, and black dummy data is disposed in other parts. After generating the enlargement information, the superimposition processing unit 142 displays the superimposition information CP1 and the enlargement information. Specifically, the superimposition processing unit 142 transmits the superimposition information CP1 and the enlargement information to the image processing unit 160.
- the image processing unit 160 having received the superimposition information CP1 and the enlargement information generates a composite image which has the superimposition information CP1 and the enlargement information as respective layers, and performs the display process described in Fig. 2 on the generated composite image.
- Figs. 12A and 12B are diagrams illustrating other examples of a virtual image recognized by the user.
- Fig. 12A illustrates a virtual image recognized by the user due to step S122 of Fig. 5.
- the superimposition information CP1 and the enlargement information CP2 are displayed as the virtual image VI in the visual field VR of the user of the head mounted display 100.
- the user of the head mounted display 100 can simultaneously visually recognize the enlargement information CP2 of the real image centering on the designated point, that is, the enlargement information CP2 of at least a part of the object in the visual field direction, in addition to the real image of the target object (in other words, the object included in the external scenery) in the visual field direction and the information (the superimposition information CP1) which is not shown in the appearance of the target object, at user's own will.
- the enlargement information CP2 of the real image centering on the designated point that is, the enlargement information CP2 of at least a part of the object in the visual field direction
- the information the superimposition information CP1 which is not shown in the appearance of the target object, at user's own will.
- the image display section 20 allows the user to visually recognize a virtual image VI (video) which is based on the superimposition information CP1 for superimposing the invisible information SI (in the above-described example, the X-ray image of the person's right hand) of the target object on the target object (in the above-described example, the person's right hand) included in the external scenery SC.
- the head mounted display 100 according to the present embodiment is a head mounted display which allows the virtual image VI (video) and the external scenery SC to be visually recognized together.
- the user of the head mounted display 100 can visually recognize the real image of the target object (in other words, the target object included in the external scenery) in the visual field direction and the information (the superimposition information CP1) which is not shown in the appearance of the target object together, without moving the visual line of the user.
- the invisible information SI stored in the invisible information storage portion 122 is an image indicating a structure of the target object.
- the user of the head mounted display 100 can visually recognize the real image of the target object in the visual field direction and the image indicating the structure of the target object together without moving his/her visual line.
- the head mounted display 100 according to the present embodiment is very useful for minimally invasive surgery.
- the minimally invasive surgery is a surgical method of keeping an incision site caused by a scalpel to a minimum in order to minimize a patient's pain or a scar from surgery due to the surgery.
- the superimposition processing unit 142 further displays auxiliary information CP3 as a virtual image VI in addition to the superimposition information CP1.
- the auxiliary information CP3 may be any information as long as the information is additional information for assisting work of the user of the head mounted display 100.
- the auxiliary information CP3 may be, for example, the following information in the medical head mounted display 100.
- Fig. 12B illustrates an example in which the auxiliary information CP3, which displays a heart rate (HR), a blood pressure (SYS/DIA), and arterial oxygen saturation (SpO2), is displayed as the virtual image VI.
- the auxiliary information CP3 is represented by characters.
- the auxiliary information CP3 a picture, a pattern, a figure, and a combination thereof may be employed in addition to characters.
- the auxiliary information CP3 is preferably disposed, for example, in an end part so as not to overlap with the superimposition information CP1.
- a method of displaying the superimposition information CP1 and the auxiliary information CP3 together is the same as in step S122.
- Fig. 12B illustrates an example of the auxiliary information CP3 in the medical head mounted display 100.
- the auxiliary information CP3 can be displayed as the virtual image VI even in the head mounted display 100 used for other applications.
- the auxiliary information CP3 in the head mounted display 100 used in a construction site may be information which displays information of a building (an age of the building, information regarding the ground on which the building is currently constructed, information regarding a construction method, information regarding materials used in the construction, and the like).
- various pieces of auxiliary information may be employed on the basis of purpose of use of the head mounted display 100.
- the user of the head mounted display 100 can visually recognize the auxiliary information CP3 which is information for assisting work as the virtual image VI (video). For this reason, it is possible to improve efficiency of work using the head mounted display 100.
- a color conversion process is performed on at least one of the superimposition information CP1 generated in step S108, another superimposition information piece generated in step S114, and the enlargement information CP2 generated in step S122.
- the color conversion process is preferably a color conversion process for improving visibility of each piece of information (the superimposition information CP1, another superimposition information piece, and the enlargement information CP2) generated by the superimposition processing unit 142, on the basis of brightness, contrast, color saturation, and a color of the external scenery image captured in step S202.
- the color conversion process may be performed so that a color of each piece of information generated by the superimposition processing unit 142 is converted into a color complementary to a color of the external scenery image captured in step S202.
- the color conversion process may be performed by the superimposition processing unit 142 before each piece of information is transmitted to the image processing unit 160, and may be performed by the image processing unit 160 having received each piece of information.
- the superimposition processing unit 142 withdraws the information (the superimposition information CP1, another superimposition information piece, the enlargement information CP2, and the auxiliary information CP3) which is displayed as the virtual image VI, to a position which does not hinder the visual field of the user.
- the virtual image VI which is currently displayed is the superimposition information CP1 will be described as an example.
- the superimposition processing unit 142 reduces the superimposition information CP1. Then, the superimposition processing unit 142 generates reduction information in which the reduced image is disposed in an end part and black dummy data is disposed in other parts. After generating the reduction information, the superimposition processing unit 142 displays the reduction information.
- the superimposition processing unit 142 transmits the reduction information to the image processing unit 160.
- the image processing unit 160 having received the reduction information performs the display process described in Fig. 2 on the reduction information.
- the reduction information in which the superimposition information CP1 is reduced and disposed in the end part is displayed as the virtual image VI instead of the superimposition information CP1 in the visual field VR of the user of the head mounted display 100.
- the additional process 3 may be performed instead of step S118 of the superimposition information display process (Fig. 5).
- the user of the head mounted display 100 can withdraw the information (the superimposition information CP1, another superimposition information piece, the enlargement information CP2, and the auxiliary information CP3) which is displayed as the virtual image VI (video), to a position which does not hinder the visual field at user's own will.
- the user may withdraw the information which is displayed as the virtual image VI (video) on purpose. For this reason, it is possible to improve convenience to the user of the head mounted display 100.
- the additional process 3 since a variety of reduced information (the superimposition information CP1, another superimposition information piece, the enlargement information CP2, and the auxiliary information CP3) is disposed in the end part, it is possible to allow the user of the head mounted display 100 to pay attention to the presence of a variety of information.
- a variety of reduced information may be used in the same manner as an icon.
- the superimposition information CP1 when the reduced superimposition information CP1 is selected by the user, the superimposition information CP1 may be returned to original position and size thereof. In this way, it is possible to further improve convenience to the user of the head mounted display 100.
- the superimposition processing unit 142 realizes a so-called stabilizer function of suppressing shaking of the virtual image VI due to slight shifting of the head of the user.
- the superimposition processing unit 142 causes the process to proceed to step S108 after waiting for a predetermined time. Accordingly, since capturing of an external scenery image and generation and display of the superimposition information CP1 are repeatedly performed for each predetermined time, it is possible to display the superimposition information CP1 which tracks a motion of the head of the user over time, as the virtual image VI.
- the superimposition processing unit 142 may perform processes indicated in procedures a1 and a2 between steps S202 and S204 of the superimposition information generation process (Fig. 8).
- Procedure a1) A change amount of an RGB histogram is obtained between an external scenery image captured in step S202 and an external scenery image captured in step S202 of the previously performed superimposition information generation process.
- Procedure a2) In a case where the change amount obtained in the procedure a1 is larger than a predetermined threshold value, the processes in and after step S204 are successively performed. In a case where the change amount obtained in the procedure a1 is equal to or smaller than the predetermined threshold value, the superimposition information CP1 generated due to the previously performed superimposition information generation process is set as a processing result of the currently performed superimposition information generation process.
- the superimposition processing unit 142 does not generate new superimposition information CP1, and sets the superimposition information CP1 generated due to the previously performed superimposition information generation process as a processing result of the currently performed superimposition information generation process.
- the superimposition information CP1 it is possible to prevent the superimposition information CP1 from being often changed by tracking a slight motion of the head or breathing of the user of the head mounted display 100. Therefore, it is possible to prevent eyestrain or reduction in concentration of the user.
- the superimposition processing unit 142 changes a size at which the enlargement information CP2 and the auxiliary information CP3 are displayed as the virtual image VI in accordance with a distance between the optical image display units 26 and 28 of the image display section 20 and an object present in the visual field direction of the user.
- the superimposition processing unit 142 may perform processes indicated in procedures b1 to b3 in the superimposition information display process (Fig. 5).
- a range finding sensor is further provided in the head mounted display 100.
- the range finding sensor is a sensor which acquires a distance between an object present in the visual field direction of the user and the image display section 20 by using reflected light, and may be disposed, for example, around the camera 61 of Fig. 1.
- Procedure b1) The superimposition processing unit 142 periodically acquires a measurement value obtained by the range finding sensor.
- Procedure b2) The superimposition processing unit 142 changes an enlargement ratio in accordance with the acquired measurement value of the range finding sensor, during the enlargement process in step S122 of the superimposition information display process (Fig. 5).
- Procedure b3) The superimposition processing unit 142 changes a size of the auxiliary information CP3 in the additional process 1 in accordance with the measurement value of the range finding sensor.
- the superimposition processing unit 142 changes a size at which the enlargement information CP2 and the auxiliary information CP3 are displayed as the virtual image VI (video) in accordance with a distance between an object present in the visual field direction of the user and the image display section 20. For this reason, it is possible to improve convenience to the user of the head mounted display 100.
- the wireless communication unit 132 transmits external scenery images which are acquired in the middle of the above-described process to an external device.
- the external scenery images include, specifically, the external scenery image in step S202 of the superimposition information generation process (Fig. 8) and the external scenery image acquired in step S120 of the superimposition information display process (Fig. 5).
- the external device is, for example, another head mounted display 100, a server, a monitor, or the like.
- the wireless communication unit 132 transmits these external scenery images to the external device through wireless connection.
- the external device which has received the external scenery images may utilize the received external scenery images in various aspects.
- the external scenery images may be displayed on another head mounted display 100 or monitor.
- the external scenery images may be stored in a server so as to be preserved as a log of surgery (work).
- the external device may acquire advice given by a surgeon (worker) who views the displayed images.
- the external device transmits the acquired advice to the head mounted display 100.
- the wireless communication unit 132 of the head mounted display 100 having received the advice, transfers the received advice. Specifically, in a case where the received advice is sound information, the wireless communication unit 132 transfers the sound information to the sound processing unit 170. In a case where the received advice is character information or image information, the wireless communication unit 132 transfers the character information or the image information to the superimposition processing unit 142.
- the sound processing unit 170 having received the sound information amplifies an audio signal included in the sound information so as to supply the amplified audio signal to speakers built in the right earphone 32 and the left earphone 34. Accordingly, the user of the head mounted display 100 can hear the advice with sounds.
- the superimposition processing unit 142 having received the character information or the image information further displays the received character information or image information as the virtual image VI in addition to the superimposition information CP1.
- a method of displaying the character information or the image information is the same as in the auxiliary information CP3 described in the additional process 1. Accordingly, the user of the head mounted display 100 can visually recognize the advice indicated by text or an image as the virtual image VI.
- the external scenery image acquired in the middle of the process can also be viewed and thus can be preserved in the external device.
- the user of the head mounted display 100 can obtain advice from a person who has viewed the external scenery image acquired in the middle of the process. For this reason, it is possible to improve efficiency and accuracy of work using the head mounted display 100.
- the superimposition processing unit 142 performs calibration of a display region of the virtual image VI and an image capturing region of the camera 61.
- the superimposition processing unit 142 performs processes indicated in the following procedures c1 to c7 prior to the superimposition information display process (Fig. 5).
- Procedure c1) Image data of a guide image (hereinafter, referred to as "guide image data") for calibration is stored in the storage unit 120 in advance.
- the guide image is an image for showing an end part of a display region of a virtual image to the user of the head mounted display 100.
- the guide image may be a rectangular image which has the same aspect ratio as the right LCD 241 and the left LCD 242, has a totally white background, and has circular guide marks in four corners thereof and a part located at an intersection of diagonal lines.
- Procedure c2) The superimposition processing unit 142 reads the guide image data stored in the storage unit 120, and transmits the guide image data to the image processing unit 160.
- the process described in Fig. 2 is performed in the image processing unit 160 having received the guide image data.
- the guide image is displayed as the virtual image VI in the visual field VR of the user of the head mounted display 100.
- Procedure c3) The superimposition processing unit 142 guides the user so as to sequentially point at the guide marks of the guide image displayed as the virtual image VI.
- a message using a dialog box may be used, the guiding may be performed using sounds.
- the guiding using sounds is preferable in that the guiding can be performed without obstructing the guide image whose display is in progress.
- Procedure c4) The superimposition processing unit 142 causes the camera 61 to capture an external scenery image.
- Procedure c5) The superimposition processing unit 142 performs image recognition on the external scenery image captured by the camera 61 so as to specify a position of a human fingertip.
- the superimposition processing unit 142 acquires coordinates of the fingertip position in relation to each guide mark so as to preserve the coordinates thereof in the storage unit 120.
- the coordinates of the fingertip position may be defined as a shift amount in the X direction and Y direction when the uppermost left part of the external scenery image is set to (0,0).
- the superimposition processing unit 142 repeatedly performs a series of processes including acquisition of an external scenery image, acquisition of coordinates of a fingertip position, and preservation of the coordinates, in relation to all the guide marks. Subsequently, the superimposition processing unit 142 stops the display of the guide image. In this way, the superimposition processing unit 142 preliminarily acquires a region where an image capturing region of the camera 61 overlaps with a display region of the virtual image VI. In addition, the region where the image capturing region (acquisition region) of the camera 61 overlaps with the display region of the virtual image VI is also referred to as an "overlap region".
- Procedure c7) The superimposition processing unit 142 extracts a feature of an object included in an image of the overlap region from the external scenery image captured by the camera 61 in step S204 of the superimposition information generation process (Fig. 8). In addition, in step S208, superimposition information CP1 corresponding to the overlap region is generated.
- the superimposition processing unit 142 can generate the superimposition information CP1 in relation to the overlap region where the display region of the virtual image VI (video) overlaps with the acquisition region of the camera 61 (image acquisition unit). For this reason, it is possible to reduce the occurrence of "mismatch between an image which is directly viewed by the user in his/her visual field and a virtual image VI (video) which is allowed to be visually recognized by the image display section 20", which is a problem occurring in the transmissive head mounted display 100. As a result, when the user visually recognizes a virtual image based on the superimposition information CP1, it is possible to reduce the discomfort which the user feels.
- Configuration of head mounted display Fig. 13 is a functional block diagram illustrating a configuration of a head mounted display 100a according to the second embodiment.
- a difference from the first embodiment illustrated in Fig. 2 is that a control section 10a is provided instead of the control section 10.
- the control section 10a includes a storage unit 120a instead of the storage unit 120, and a CPU 140a instead of the CPU 140.
- the storage unit 120a includes a procedure table 126 in addition to the respective storage portions illustrated in Fig. 2.
- the CPU 140a includes a superimposition processing unit 142a instead of the superimposition processing unit 142 illustrated in Fig. 2, and further includes a procedure management unit 144 in addition to the respective processing units illustrated in Fig. 2.
- the superimposition processing unit 142a and the procedure management unit 144 perform a superimposition information display process in the second embodiment in cooperation with each other.
- Fig. 14 is a diagram illustrating an example of the procedure table 126.
- the procedure table 126 is a table used in the superimposition information display process in the second embodiment.
- the procedure table 126 is stored in the storage unit 120a in advance.
- the procedure table 126 includes fields of a work number, a procedure number, a procedure name, instruction content, and an image.
- An identifier for uniquely identifying work is stored in the work number.
- an identifier indicating a technique of surgery is stored in the work number.
- the identifier is formed by a character string, and may be formed by, for example, a combination of alphanumeric characters.
- Various information pieces regarding a plurality of procedures included in the work specified by the work number are stored in the respective fields of the procedure number and thereafter.
- An identifier for uniquely identifying a procedure is stored in the procedure number.
- the identifier is formed by a character string, and may be formed by, for example, a combination of alphanumeric characters.
- the name of a corresponding procedure is stored in the procedure name.
- a character string indicating content to be performed by the user in the corresponding procedure is stored in the instruction content.
- the instruction content is represented in a character string with a guide sentence form for the user.
- An image indicating content to be performed by the user is stored in the image field.
- the image is a picture or the like of a scene in which, for example, a third party exemplarily performs a corresponding procedure. Further, in addition to the picture, the image may be drawings (an explanatory diagram, an assembly diagram, a layout drawing, a circuit diagram, and the like), and may be combinations thereof.
- the procedure table 126 is a table which stores procedures of work (that is, surgery) performed by a surgeon in the medical head mounted display 100.
- the work identified by the work number 1 is artificial finger joint replacement surgery.
- the content to be performed by the user in the procedure identified by the procedure number 1 is "preparing for articles necessary in the surgery", and "what necessary articles are”.
- an image indicating content to be performed by the user is not correlated with the procedure identified by the procedure number 1.
- FIG. 15 is a flowchart illustrating procedures of the superimposition information display process according to the second embodiment. A difference from the first embodiment illustrated in Fig. 5 is that steps S302, S304 and S306 are added.
- the procedure management unit 144 estimates a current procedure in step S302. Specifically, the procedure management unit 144 activates the camera 61 so as to instruct the camera 61 to capture an image, and acquires an external scenery image captured by the camera 61. The procedure management unit 144 performs image recognition on the acquired external scenery image so as to estimate a current procedure from a plurality of procedures stored in the procedure table 126. In this way, the procedure management unit 144 can automatically estimate a current procedure on the basis of the image (external scenery image) in the visual field direction of the user, acquired by the camera 61 (image acquisition unit).
- the superimposition processing unit 142a After the current procedure is estimated, the superimposition processing unit 142a performs a superimposition information generation process in step S108. Details thereof are the same as in the first embodiment described in Fig. 8.
- the procedure management unit 144 After the superimposition information generation process is performed, the procedure management unit 144 generates procedure information in step S304. Specifically, the procedure management unit 144 performs processes indicated in the following procedures d1 to d4.
- Procedure d1) The procedure management unit 144 searches the procedure table 126, and extracts entries correlated with the current procedure estimated in step S302.
- Procedure d2) The procedure management unit 144 determines whether or not an image (an image indicating content to be performed by the user) is stored in the image field among the extracted entries.
- the procedure management unit 144 performs the procedure d3 if an image is stored, and performs the procedure d4 if an image is not stored.
- Procedure d3) The procedure management unit 144 generates procedure information in which the image stored in the image field is disposed in an upper end part, the character string stored in the instruction content field is disposed in a lower end part, and black dummy data is disposed in other parts.
- the character string preferably has a color with high visibility in consideration of a color of the external scenery image.
- Procedure d4) The procedure management unit 144 generates procedure information in which the character string stored in the instruction content field is disposed in a lower end part, and black dummy data is disposed in other parts.
- the arrangement of the image and the character string in the procedures d3 and d4 may be arbitrarily changed.
- the image and the character string are preferably disposed around the end part (in other words, around the four sides thereof) of the procedure information CP4.
- Fig. 16 is a diagram illustrating an example of the procedure information.
- a character string MG4 "Insert the artificial finger joint", indicating content to be performed by the user, is disposed in the lower end part.
- an image IM4 indicating a model in which an artificial finger joint is inserted is disposed in the upper right end part.
- the superimposition processing unit 142a and the procedure management unit 144 display the superimposition information CP1 and the procedure information in step S306. Specifically, the superimposition processing unit 142a transmits the superimposition information CP1 to the image processing unit 160, and the procedure management unit 144 transmits the procedure information CP4 to the image processing unit 160.
- the image processing unit 160 which has received the superimposition information CP1 and the procedure information CP4 generates a composite image which has the superimposition information CP1 and the procedure information CP4 as respective layers, and performs the display process described in Fig. 2 on the generated composite image. As a result, the superimposition information CP1 and the procedure information CP4 are displayed as the virtual image VI in the visual field VR of the user of the head mounted display 100a.
- the image display section 20 allows the user to visually recognize, as the virtual image VI (video), the procedure information CP4 indicating content to be performed by the user of the head mounted display 100a in the current procedure. For this reason, in addition to the effects of the first embodiment, it is possible to visually show content to be performed by the user of the head mounted display 100a using the image. Therefore, it is possible to more intelligibly support work performed by the user.
- the virtual image VI video
- the procedure information CP4 indicating content to be performed by the user of the head mounted display 100a in the current procedure.
- an image indicating an incision site of the skin of a patient who actually undergoes surgery may be stored in the image field of the procedure table 126.
- the procedure management unit 144 performs processes indicated in the following procedures e1 to e9 instead of steps S108, S304 and S306 of the superimposition information display process (Fig. 15).
- Procedure e1) The procedure management unit 144 searches the procedure table 126, and extracts entries correlated with the current procedure estimated in step S302.
- Procedure e2) The procedure management unit 144 determines whether or not an image is stored in the image field among the extracted entries.
- the procedure management unit 144 performs the procedures e3 to e7 and e9 if an image is stored, and performs the procedures e8 and e9 if an image is not stored.
- Procedure e3) The superimposition processing unit 142a acquires an external scenery image. Details thereof are the same as in step S202 of the superimposition information generation process (Fig. 8).
- Procedure e4) The superimposition processing unit 142a performs image recognition on the external scenery image so as to extract a feature of the target object. Details thereof are the same as in step S204 of the superimposition information generation process.
- Procedure e5) The superimposition processing unit 142a performs image recognition on the image stored in the image field so as to extract a feature of the target object. Details thereof are the same as in step S206 of the superimposition information generation process. However, instead of the "invisible information" in step S206, the "image stored in the image field" is read.
- Procedure e6 The superimposition processing unit 142a corrects the image stored in the image field so that the feature of the external scenery image extracted in the procedure e4 matches the feature of the image stored in the image field, extracted in the procedure e5, so as to generate superimposition information CP5. Details thereof are the same as in step S208 of the superimposition information generation process. However, instead of the "invisible information" in step S208, the "image stored in the image field" is read.
- Procedure e7) The procedure management unit 144 adds a layer indicating the procedure information to the superimposition information CP5 generated in the procedure e6. Specifically, the procedure management unit 144 disposes the character string stored in the instruction content field of the entry extracted in the procedure e1, in a lower end part of the superimposition information CP5.
- Fig. 17 is a diagram illustrating an example of superimposition information to which the procedure information is added.
- an image IM5 which indicates an incision site of the skin of a patient who actually undergoes surgery, is disposed at an angle, a position and a size matching the feature of the target object (the object included in the external scenery) in the visual field direction of the user.
- a character string MG5 "Incise the skin of the surgical site using a scalpel with a circular edge to expose the finger joint", indicating content to be performed by the user, is disposed in a lower end part.
- Procedure e8) The procedure management unit 144 generates procedure information CP4 in which the character string stored in the instruction content field of the entry extracted in the procedure e1 is disposed in a lower end part, and black dummy data is disposed in other parts.
- Procedure e9) The procedure management unit 144 displays the superimposition information CP5 generated in the procedure e7 or the procedure information CP4 generated in the procedure e8. Specifically, the procedure management unit 144 transmits the generated superimposition information CP5 or procedure information CP4 to the image processing unit 160.
- the image processing unit 160 having received the superimposition information CP5 or the procedure information CP4 performs the display process described in Fig. 2 thereon. As a result, the superimposition information CP5 or the procedure information CP4 is displayed as the virtual image VI in the visual field VR of the user of the head mounted display 100a.
- Fig. 18 is a diagram illustrating an example of a virtual image which is visually recognized by the user due to the procedures e1 to e9. As illustrated in Fig. 18, the superimposition information CP5 is displayed as the virtual image VI in the visual field VR of the user of the head mounted display 100a.
- the user of the head mounted display 100a can visually recognize the real image (a state in which the cutting edge of the scalpel held by the surgeon is in contact with the second joint of the little finger of the patient's right hand) in the external scenery SC which is viewed through the right optical image display unit 26 and the left optical image display unit 28 and the superimposition information CP5 (that is, the image indicating the incision site of the skin of the patient who actually undergoes the surgery, and the character string indicating the content to be performed by the user) which is viewed as the virtual image VI, in a superimposition state.
- the superimposition information CP5 that is, the image indicating the incision site of the skin of the patient who actually undergoes the surgery, and the character string indicating the content to be performed by the user
- FIG. 19 is a diagram illustrating a schematic configuration of a head mounted display 100b according to the third embodiment.
- Fig. 20 is a functional block diagram illustrating a configuration of the head mounted display 100b according to the third embodiment.
- a difference from the first embodiment illustrated in Figs. 1 and 2 is that a control section 10b is provided instead of the control section 10, and an image display section 20b is provided instead of the image display section 20.
- the control section 10b includes a storage unit 120b instead of the storage unit 120, and a CPU 140b instead of the CPU 140.
- the storage unit 120b further includes parallax information 128 in addition to the invisible information storage portion 122 and the appearance information storage portion 124 described in the first embodiment.
- Fig. 21 is a diagram for explaining the parallax information 128.
- the parallax information 128 in the present embodiment is a table in which a plurality of distances Lx between the user and virtual images visually recognized by the user are correlated with parallaxes Px which respectively correspond to the plurality of distances Lx.
- the parallax indicates a disparity (difference) between images formed on the retinas of both eyes.
- a distance L1 between the user and a virtual image VI1 visually recognized by the user is correlated with a parallax of 2 mm.
- a distance L2 between the user and a virtual image VI2 visually recognized by the user is correlated with a parallax of 6 mm
- a distance L3 between the user and a virtual image VI3 visually recognized by the user is correlated with a parallax of 10 mm.
- the human depth interval dramatically decreases as a distance between the user and a virtual image visually recognized by the user becomes larger. For this reason, a larger volume of data may be prepared in the parallax information 128 as the distance Lx becomes smaller.
- This correlation is created in advance by measuring distances between positions of virtual images (that is, image formation positions where a virtual image visually recognized by the right eye and a virtual image visually recognized by the left eye) visually recognized by both eyes of the user, and both eyes of the user, as a result of deviating a display position of a virtual image for the right eye and a display position of a virtual image for the left eye.
- positions of virtual images that is, image formation positions where a virtual image visually recognized by the right eye and a virtual image visually recognized by the left eye
- an image formation position when creating the parallax information 128 may be located on the front side of the reference plane when viewed from the user, and may be located on the rear side of the reference plane when viewed from the user.
- An image formation position may be located both on the front side and on the rear side of the reference plane.
- a virtual image appears to protrude toward the user, and, in the latter case, a virtual image appears to be depressed from the user.
- parallaxes of 2 mm, 6 mm and 10 mm are parallaxes of a central part of a virtual image corresponding to a discriminative visual field and an effective visual field. Therefore, for more accuracy, parallaxes (1.9 mm, 5 mm, and 9 mm) of parts corresponding to a steady-gaze stable field of the user, parallaxes (1.8 mm, 4 mm, and 8 mm) of parts corresponding to an induced visual field of the user, and parallaxes (1.7 mm, 3 mm, and 7 mm) of parts corresponding to an auxiliary visual field of the user, and the like may be differentiated from each other and be stored in the parallax information 128. Furthermore, the specific numerical values in Fig. 21 are only an example.
- the discriminative visual field is a range in which, among the human visual fields, a visual function such as vision or color discrimination is good, and thus high accuracy information can be accepted.
- the discriminative visual field is typically within several degrees from the center.
- the effective visual field is a range in which, among the human visual fields, information is gazed steadily at only with the eye movement, and specific information can be instantaneously accepted.
- the effective visual field is typically within about 15 degrees to both the left and the right, within about 8 degrees upwardly, and within about 12 degrees downwardly.
- the steady-gaze stable field is a range in which, among the human visual fields, a steady gaze is possible comfortably in a state in which the head movement assists the eye movement.
- the steady-gaze stable field is typically within 30 to 45 degrees to both the left and the right, within 20 to 30 degrees upwardly, and within 25 to 40 degrees downwardly.
- the induced visual field is a range in which, among the human visual fields, there is only discrimination capability that is a degree of being capable of determining the presence of presented information, but the human spatial coordinate sense is influenced.
- the induced visual field is typically within 30 to 100 degrees horizontally, and within 20 to 85 degrees vertically.
- the auxiliary visual field is a range in which, among the human visual fields, information acceptance is notably decreased, and an auxiliary function that is a degree of inducing a steady gaze operation due to a strong stimulus or the like is performed.
- the auxiliary visual field is typically within 100 to 200 degrees horizontally, and within 85 to 135 degrees vertically.
- the CPU 140b of Fig. 20 includes a superimposition processing unit 142b instead of the superimposition processing unit 142.
- the superimposition processing unit 142b is different from the first embodiment in content of the performed superimposition information display process. Details thereof will be described later.
- the image display section 20b further includes pupil image capturing cameras 62 and 63, near-infrared-ray receiving and emitting units 64 and 65, a nine-axis sensor 66, and a depth sensor 67, in addition to the respective units described in the first embodiment.
- the pupil image capturing camera 62 is disposed near the end part ER (Fig. 1) of the image display section 20.
- the pupil image capturing camera 62 captures an image of the pupil of the right eye RE of the user when the user wears the head mounted display 100b, in the rear side direction of the image display section 20.
- the pupil image capturing camera 63 is disposed near the EL (Fig. 1) of the image display section 20.
- the pupil image capturing camera 63 captures an image of the pupil of the left eye LE of the user when the user wears the head mounted display 100b.
- the near-infrared-ray receiving and emitting unit 64 is disposed near the end part ER (Fig. 1) of the image display section 20.
- the near-infrared-ray receiving and emitting unit 64 applies infrared rays toward the right eye RE of the user and acquires reflected light in the rear side direction of the image display section 20.
- the near-infrared-ray receiving and emitting unit 65 is disposed near the end part EL (Fig. 1) of the image display section 20.
- the near-infrared-ray receiving and emitting unit 65 applies infrared rays toward the left eye LE of the user and acquires reflected light in the rear side direction of the image display section 20.
- the pupil image capturing cameras 62 and 63 and the near-infrared-ray receiving and emitting units 64 and 65 function as a "visual line detection unit" which detects a visual line of the user in cooperation with each other.
- some of the pupil image capturing cameras 62 and 63 and the near-infrared-ray receiving and emitting units 64 and 65 may be omitted.
- the near-infrared-ray receiving and emitting units 64 and 65 may be omitted, and the pupil image capturing camera 63 and the near-infrared-ray receiving and emitting unit 65 may be omitted.
- the nine-axis sensor 66 is disposed at the position corresponding to the glabella of the user when the user wears the image display section 20.
- the nine-axis sensor 66 is a motion sensor which detects acceleration (in three axes), angular velocity (in three axes), and geomagnetism (in three axes).
- the nine-axis sensor 66 is provided in the image display section 20, and thus functions as a "motion detection unit" which detects a motion of the head of the user when the image display section 20 is mounted on the head of the user.
- the motion of the head includes velocity, acceleration, angular velocity, a direction, and a change in a direction of the head.
- the depth sensor 67 is disposed at the position corresponding to the glabella of the user when the user wears the image display section 20.
- the depth sensor 67 includes an infrared-ray irradiation portion and an infrared-ray sensor, and measures a depth of each point located in front of the depth sensor 67.
- the depth sensor 67 functions as a "position detection unit" which detects a position of the user relative to an object (target object) which is a target of the superimposition information display process.
- Superimposition information display process Fig. 22 is a flowchart illustrating procedures of the superimposition information display process according to the third embodiment. A difference from the first embodiment illustrated in Fig. 5 is that: step S602 is provided instead of step S102; step S604 is provided instead of step S108; step S606 is provided instead of step S110; and step S608 is further provided after step S110.
- the superimposition information display process according to the third embodiment is performed in a three-dimensional display mode.
- the user stores a plurality of invisible information pieces regarding a certain object which is a target of the superimposition information display process in the invisible information storage portion 122.
- the invisible information is an image indicating a structure of a target object acquired by using image capturing apparatuses such as a CT apparatus, an MRI apparatus, an X-ray apparatus, an endoscope, and a thermography apparatus, in the same manner as in step S102 of Fig. 5.
- image capturing apparatuses such as a CT apparatus, an MRI apparatus, an X-ray apparatus, an endoscope, and a thermography apparatus, in the same manner as in step S102 of Fig. 5.
- one invisible information piece is selected from a plurality of invisible information pieces in accordance with a positional relationship between the target object and the user.
- invisible information regarding the target object is acquired from various angles and is stored in the invisible information storage portion 122.
- Fig. 23 is a diagram illustrating an example of the invisible information.
- the invisible information storage portion 122 of the third embodiment stores a plurality of coronary plane CT images SI1 which are any planar images that divide the body of a patient PA into a front side and a rear side; a plurality of transverse cross-sectional CT images SI2 which are any planar images that divide the body of the patient PA into planes perpendicular to the longitudinal direction; and an X-ray image SI3 of a surgical site of the patient PA.
- the invisible information illustrated in Fig. 23 is only an example, and, for example, a sagittal plane CT image may be included, and an MRI image or an endoscope image may be included instead of the CT image.
- the superimposition processing unit 142b In step S604 of Fig. 22, the superimposition processing unit 142b generates superimposition information by performing either of "C-2-1. First superimposition information generation process" and "C-2-2. Second superimposition information generation process" described below.
- a difference between the first superimposition information generation process and the second superimposition information generation process is accuracy with which the superimposition processing unit 142b corrects invisible information. Specifically, correction is performed in the second superimposition information generation process with higher accuracy than in the first superimposition information generation process.
- Which one of the first superimposition information generation process and the second superimposition information generation process is performed may be predefined, and may be designated by the user.
- which one of the two is performed may be automatically determined depending on an amount of a motion of the head of the user detected by the nine-axis sensor, and may be automatically determined depending on the type or a technique of surgery performed using the head mounted display 100b.
- First superimposition information generation process Fig. 24 is a flowchart illustrating procedures of the first superimposition information generation process.
- the superimposition processing unit 142b acquires an external scenery image. Specifically, the superimposition processing unit 142b activates the camera 61 so as to instruct the camera 61 to capture an image, and acquires an external scenery image captured by the camera 61.
- Fig. 25 is a diagram for explaining step S404 of the first superimposition information generation process.
- the superimposition processing unit 142b acquires a visual line and a position of the user in step S404 of Fig. 24. Specifically, the superimposition processing unit 142b acquires a visual line of the user from detection values of the pupil image capturing cameras 62 and 63 and the near-infrared-ray receiving and emitting units 64 and 65. Subsequently, the superimposition processing unit 142b obtains a steady gaze point distance which is a distance between the user and a point PP (in Fig. 25, a steady gaze point PP) at which the user gazes steadily, from the acquired visual line of the user.
- a steady gaze point distance which is a distance between the user and a point PP (in Fig. 25, a steady gaze point PP) at which the user gazes steadily, from the acquired visual line of the user.
- the steady gaze point is a point where the central foveae (locations where vision is most acute in the retinas) of both eyes of the user intersect each other on extension lines thereof.
- the superimposition processing unit 142b acquires positions PO1 to PO3 (Fig. 25) of the user relative to the target object from a detection value of the depth sensor 67.
- the superimposition processing unit 142b may use, for example, a depth DP (Fig. 25) for features M1 to M4 (in Fig. 25, edges or markers) of the target object as a reference for acquiring a position.
- the superimposition processing unit 142b selects one invisible information piece corresponding to the position of the user relative to the target object from the plurality of invisible information pieces stored in the invisible information storage portion 122. Specifically, the superimposition processing unit 142b selects several invisible information pieces which are considered to be optimum on the basis of the positional relationship between the user and the target object, acquired in step S404. For example, in a case where the user is located at a position where the user looks down at the patient PA (target object) illustrated in Fig. 23, the superimposition processing unit 142b selects a plurality of coronary plane CT images SI1. Next, the superimposition processing unit 142b selects one invisible information piece corresponding to the depth acquired in step S404 from the selected several invisible information pieces. In addition, in step S406, the superimposition processing unit 142b may present the selected several invisible information pieces to the user as candidates, and may receive a selection of the invisible information from the user.
- step S408 the superimposition processing unit 142b performs image recognition on the external scenery image so as to extract a feature of the target object. Details thereof are the same as in step S204 of Fig. 8 described in the first embodiment.
- step S410 the superimposition processing unit 142b performs image recognition on the invisible information selected in step S408 so as to extract a feature of the target object. Details thereof are the same as in step S206 of Fig. 8 described in the first embodiment.
- step S412 the superimposition processing unit 142b corrects the invisible information selected in step S408 so that the feature of the external scenery image extracted in step S408 and the feature of the invisible information extracted in step S410 match each other, thereby generating superimposition information. Details thereof are the same as in step S208 of Fig. 8 described in the first embodiment.
- the superimposition processing unit 142b gives a parallax to the superimposition information. Specifically, the superimposition processing unit 142b searches the parallax information 128 by using the steady gaze point distance obtained in step S404 as a keyword, and acquires a parallax stipulated in the parallax information 128. The superimposition processing unit 142b gives the parallax corresponding to the steady gaze point distance acquired from the parallax information 128 to the superimposition information generated in step S412, so as to generate superimposition information for the right eye and superimposition information for the left eye. As described above, according to step S414, the superimposition processing unit 142b can easily generate the superimposition information for the right eye and the superimposition information for the left eye by using the parallax information 128 which is prepared in advance.
- the superimposition processing unit 142b displays the superimposition information in step S606 of Fig. 22. Specifically, the superimposition processing unit 142b transmits the superimposition information for the right eye and the superimposition information for the left eye generated due to the superimposition information generation process to the image processing unit 160.
- the image processing unit 160 which has received the superimposition information for the right eye and the superimposition information for the left eye performs the display process described in Fig. 2 on the respective pieces of the received superimposition information. As a result, a virtual image based on the superimposition information is displayed in the visual field of the user of the head mounted display 100b.
- the parallax corresponding to the steady gaze point distance is given to the superimposition information for the right eye and the superimposition information for the left eye. For this reason, the virtual image visually recognized by the user of the head mounted display 100b is formed around the steady gaze point of the user and is displayed in a three-dimensional manner.
- step S608 of Fig. 22 the superimposition processing unit 142b determines whether or not the visual line and the position of the user are changed.
- the visual line and the position of the user may be acquired in the same method as in step S404 of the first superimposition information generation process (Fig. 25).
- step S608: YES the superimposition processing unit 142b causes the process to proceed to step S604, and repeatedly performs the first or second superimposition information generation process.
- the superimposition processing unit 142b causes the process to proceed to step S112.
- the predetermined threshold value may be arbitrarily defined. Further, a threshold value related to a change amount of a position and a threshold value related to a change amount of a visual line may be different from each other.
- the superimposition processing unit 142b generates the superimposition information for the right eye and the superimposition information for the left eye to which the parallax corresponding to a distance (steady gaze point distance) between the user and the steady gaze point at which the user gazes steadily is given, and the image processing unit 160 of the image display section 20 allows the user to visually recognize the virtual image based on this superimposition information. As a result, the user can visually recognize the superimposition information at a position of the same distance as the steady gaze point.
- the superimposition processing unit 142b can select one invisible information piece corresponding to a position of the user relative to the target object from a plurality of invisible information pieces stored in the invisible information storage portion 122 of the storage unit 120b, so as to correct the selected invisible information. For this reason, the superimposition processing unit 142b can generate superimposition information by using appropriately invisible information on the basis of a positional relationship between the user and the target object.
- Second superimposition information generation process Fig. 26 is a flowchart illustrating procedures of the second superimposition information generation process. A difference from the first superimposition information generation process illustrated in Fig. 24 is that step S502 is provided instead of step S404, and step S504 is provided instead of step S412.
- step S502 the superimposition processing unit 142b acquires a visual line, a position, an angle of the head of the user.
- the visual line and the position of the user are the same as in step S404 of the first superimposition information generation process (Fig. 25).
- the superimposition processing unit 142b acquires the angle of the head of the user from a detection value of the nine-axis sensor 66.
- Fig. 27 is a diagram for explaining step S504 of the second superimposition information generation process.
- the superimposition processing unit 142b corrects the invisible information. Specifically, the superimposition processing unit 142b corrects the invisible information selected in step S408 by using a method described in the following procedures f1 to f3.
- the superimposition processing unit 142b divides an image indicating the invisible information SI into a plurality of regions A1 to An (Fig. 27). As for this division, the superimposition processing unit 142b may measure distances between the user and respective parts of the target object by using triangulation, and may divide the image indicating the invisible information SI into regions on the basis of the measurement result. Specifically, for example, the superimposition processing unit 142b may group the respective parts of the target object for each measured distance from the user, and may collect invisible information pieces SI corresponding to the grouped respective parts as a single region. In addition, as for this grouping, a predetermined allowable range (threshold value) may be set to the distance. In this way, the superimposition processing unit 142b can divide the region of the image indicating the invisible information SI into a plurality of regions efficiently and in light of the actual situation.
- the superimposition processing unit 142b performs at least one processing of enlargement, reduction, trimming, distortion, and removal of noise, on each of the plurality of regions divided in the procedure f1, so as to generate superimposition information. During this processing, the superimposition processing unit 142b performs the processing so that the feature of the external scenery image and the feature of the invisible information SI, detected in steps S408 and S410, match each other.
- the superimposition processing unit 142b may change processing (correction) accuracy on the basis of an information accepting characteristic within the visual field of the user when the processing in the procedure f2 is performed.
- the information accepting characteristic indicates an extent of visual capability.
- the human visual fields may be divided into (1) a discriminative visual field, (2) an effective visual field, (3) a steady-gaze stable field, (4) an induced visual field, and (5) an auxiliary visual field, in an order of a superior information accepting characteristic within the visual field (Fig. 21).
- the superimposition processing unit 142b may increase accuracy of the processing, and, in a case where the processing target region belongs to a visual field (for example, the above (4) and (5)) with an inferior information accepting characteristic, the superimposition processing unit 142b may reduce processing accuracy. This is because, if accuracy of superimposition of the invisible information SI and the target object is increased in the visual field with an inferior information accepting characteristic, there is a concern of uselessness due to a limit of perception capability of the user.
- the superimposition processing unit 142b displays the superimposition information in step S606 of Fig. 22.
- a virtual image based on the superimposition information is displayed in the visual field of the user of the head mounted display 100b.
- the virtual image visually recognized by the user is formed around the steady gaze point of the user and is displayed in a three-dimensional manner.
- the superimposition processing unit 142b divides the image indicating the invisible information SI of the object stored in the invisible information storage portion 122 of the storage unit 120b into a plurality of regions A1 to An. Then, the superimposition processing unit 142b performs at least one processing of enlargement, reduction, trimming, distortion, and removal of noise, on each of the plurality of regions, so as to correct the regions. As above, the superimposition processing unit 142b divides the image indicating the invisible information SI into a plurality of regions A1 to An, and then corrects each region. Therefore, it is possible to improve accuracy of the correction of the image indicating the invisible information SI.
- the superimposition processing unit 142b changes accuracy of processing (correction) on the invisible information SI on the basis of the information accepting characteristic within the visual field of the user. As a result, the superimposition processing unit 142b can efficiently correct the invisible information SI.
- the allocation of the constituent elements to the control section and the image display section are only an example, and may employ various aspects.
- the following aspects may be employed: (i) an aspect in which a processing function such as a CPU and a memory is mounted in the control section, and only a display function is mounted in the image display section; (ii) an aspect in which a processing function such as a CPU and a memory is mounted in both the control section and the image display section; (iii) an aspect in which the control section and the image display section are integrally formed (for example, an aspect in which the image display section includes the control section and functions as a glasses type wearable computer); (iv) an aspect in which a smart phone or a portable game machine is used instead of the control section; and (v) an aspect in which the control section and the image display section are configured to communicate by wireless with each other and to be supplied with power in a wireless manner so as to remove the connection unit (cords).
- control section is provided with the transmission unit
- image display section is provided with the reception unit.
- both of the transmission unit and the reception unit of the above-described embodiments have a bidirectional communication function, and thus can function as a transmission and reception unit.
- control section illustrated in Fig. 2 is connected to the image display section via the wired signal transmission path.
- control section and the image display section may be connected to each other via a wireless signal transmission path such as a wireless LAN, infrared communication, or Bluetooth (registered trademark).
- control section and the image display section illustrated in Fig. 2 may be arbitrarily changed.
- the touch pad may be omitted from the control section, and an operation may be performed using only the cross key.
- other operation interfaces such as an operation stick may be provided in the control section.
- the control section is connectable to a device such as a keyboard or a mouse so as to receive an input from the keyboard or the mouse.
- the head mounted display is a binocular transmission type head mounted display, but may be a monocular head mounted display.
- the head mounted display may be a non-transmissive head mounted display through which external scenery is blocked from being transmitted in a state in which the user wears the head mounted display.
- Figs. 28A and 28B are diagrams illustrating exterior configurations of head mounted displays in a modification example.
- an image display section 20c includes a right optical image display unit 26c instead of the right optical image display unit 26 and a left optical image display unit 28c instead of the left optical image display unit 28.
- the right optical image display unit 26c is formed to be smaller than the optical member of the first embodiment, and is disposed on the obliquely upper side of the right eye of the user when the head mounted display is mounted.
- the left optical image display unit 28c is formed to be smaller than the optical member of the above-described embodiments, and is disposed on the obliquely upper side of the left eye of the user when the head mounted display is mounted.
- an image display section 20d includes a right optical image display unit 26d instead of the right optical image display unit 26 and a left optical image display unit 28d instead of the left optical image display unit 28.
- the right optical image display unit 26d is formed to be smaller than the optical member of the first embodiment, and is disposed on the obliquely lower side of the right eye of the user when the head mounted display is mounted.
- the left optical image display unit 28d is formed to be smaller than the optical member of the first embodiment, and is disposed on the obliquely lower side of the left eye of the user when the head mounted display is mounted.
- the optical image display units have only to be disposed near the eyes of the user. Any size of the optical member forming the optical image display units may be used, and the head mounted display may be implemented in an aspect in which the optical image display units cover only a part of the eyes of the user; in other words, the optical image display units do not completely cover the eyes of the user.
- the function units such as the image processing unit, the display control unit, the superimposition processing unit, the procedure management unit, and the sound processing unit are realized by the CPU developing a computer program stored in the ROM or the hard disk on the RAM and executing the program.
- these function units may be configured using an application specific integrated circuit (ASIC) which is designed for realizing each of the corresponding functions.
- ASIC application specific integrated circuit
- the image display section of the head mounted display is worn as glasses, but the image display section may be a typical flat display device (a liquid crystal display device, a plasma display device, an organic EL display device, or the like).
- the control section and the image display section may be connected to each other via a wired signal transmission path, and may be connected to each other via a wireless signal transmission path. In this way, the control section may be used as a remote control of a typical flat display device.
- an image display section instead of the image display section which is worn as glasses, other types of image display sections such as an image display section which is worn as, for example, a cap, may be employed.
- the earphone may employ an ear-mounted type or a head band type, or may be omitted.
- a head-up display may be configured to be mounted in a vehicle such as an automobile or an airplane.
- the head mounted display may be configured to be built in a body protection tool such as a helmet.
- a secondary battery is used as the power supply, but the power supply is not limited to the secondary battery and may use various batteries.
- a primary battery, a fuel cell, a solar cell, a thermal cell, and the like may be used.
- the image light generation portion is configured using the backlight, the backlight control portion, the LCD, and the LCD control portion.
- the image light generation portion may include a configuration portion for realizing other types along with this configuration portion or instead of this configuration portion.
- the image light generation portion may include an organic electroluminescent (EL) display and an organic EL controller.
- EL organic electroluminescent
- a digital micromirror device may be used as the image light generation portion.
- the invention is applicable to a laser retinal projective head mounted display.
- the head mounted display may include a range finding sensor or a displacement sensor instead of the camera.
- the superimposition processing unit may extract a feature of a target object by using the range finding sensor or the displacement sensor so as to perform the same process as in the above-described embodiments.
- the head mounted display may be provided with an optical filter which suppresses transmission of light with the same color as a color which is frequently used in invisible information.
- the head mounted display may be provided with an optical filter with a color complementary to a color which is frequently used in invisible information. In this way, it is possible to increase visibility of the invisible information.
- any one of the camera, the pupil image capturing cameras, the near-infrared-ray receiving and emitting units, the nine-axis sensor, and the depth sensor may be omitted.
- the pupil image capturing cameras and the near-infrared-ray receiving and emitting units may be omitted, and a camera provided on a ceiling of a surgical room may be substituted, thereby detecting a visual line of the user.
- the camera provided on the ceiling of the surgical room may be used together, thereby improving processing accuracy of invisible information.
- the user stores the invisible information of the object which is a target of the superimposition information display process in the invisible information storage portion.
- the invisible information of the above-described embodiments is assumed to be an image which is captured by an image capturing apparatus (a CT apparatus, an MRI apparatus, an X-ray apparatus, an endoscope, a thermography apparatus, or the like) and is represented in a two-dimensional manner.
- an image capturing apparatus a CT apparatus, an MRI apparatus, an X-ray apparatus, an endoscope, a thermography apparatus, or the like
- invisible information which is represented in a three-dimensional manner may be stored in step S102.
- the invisible information which is represented in a three-dimensional manner is a three-dimensional model which is created from a plurality of images captured by the image capturing apparatus.
- the superimposition processing unit performs rendering of a three-dimensional model on the basis of the external scenery image acquired in step S202, between steps S202 and S204 of the superimposition information generation process (Fig. 8).
- the subsequent processes are continuously performed using the rendered images as "invisible information". In this way, it is possible to visually recognize information which is not shown in the appearance of the target object from various directions.
- the invisible information stored in step S102 is not limited to a still image and may be a moving image.
- steps S114 and S116 another superimposition information piece is generated on the basis of the appearance information.
- various information pieces may be used as a source of another superimposition information piece.
- another invisible information piece may be used which is captured using the image capturing apparatus and is different from the invisible information used when generating the superimposition information CP1.
- step S112 may be omitted.
- the procedure management unit may add information for arousing user's attention to the procedure information as in the following example.
- matters to which the user should pay particular attention are displayed in a pop-up form.
- An image or a character string in the procedure information is displayed in a blinking state.
- a message or an icon indicating the degree of importance, the degree of difficulty, a frequency of accidents, and the like of a corresponding procedure is displayed in the procedure information.
- Modification Example 5 In the second embodiment, a description has been made of the configuration in which the head mounted display is provided with the procedure table and the procedure management unit, and can support work performed by the user without requiring other devices. However, also in a work supporting system including the head mounted display and an externally provided server, the same process may be performed.
- the head mounted display is configured to have the procedure management unit
- the server is configured to have the procedure table.
- the procedure management unit of the head mounted display accesses the procedure table stored in the server so as to refer to data of the procedure table.
- the information of the procedure table stored in a single server can be shared by a plurality of head mounted displays.
- the necessary labor spent updating the procedure table is reduced compared with a configuration in which the procedure table is provided in each head mounted display.
- a plurality of parallax information pieces may be stored in the storage unit.
- the plurality of parallax information pieces may be, for example, parallax information when a dominant eye is the right eye and parallax information when a dominant eye is the left eye, and may be parallax information based on an interocular distance.
- one parallax information piece to be used in the process in step S414 may be selected from a plurality of parallax information pieces stored in the storage unit on the basis of characteristics (for example, the interocular distance or the dominant eye) of the user of the head mounted display. In this way, the superimposition processing unit can select parallax information matching the characteristics (for example, the dominant eye or the interocular distance) of the user, thereby reducing visual discomfort which the user feels.
- an edge emphasis process or a binarization process may be used together, thereby further improving accuracy of correction of invisible information.
- Modification Example 7 The invention is not limited to the above-described embodiments or modification examples, and may be implemented using various configurations within the scope without departing from the spirit thereof.
- the embodiments, the examples, and the modification examples corresponding to technical features of the respective aspects described in Summary of Invention and the technical features in the modification examples may be exchanged or combined as appropriate in order to solve some or all of the above-described problems, or in order to achieve some or all of the above-described effects.
- the technical feature is not described as an essential feature in the present specification, the technical feature may be deleted as appropriate.
- Control Section 12 Lighting unit 14 Touch pad 16 Cross key 18 Power switch 20 Image display section 21 Right holding unit 22 Right display driving unit 23 Left holding unit 24 Left display driving unit 26 Right optical image display unit 28 Left optical image display unit 30 Earphone plug 32 Right earphone 34 Left earphone 40 Connection unit 42 Right cord 44 Left cord 46 Connection member 48 Main body cord 51 Transmission unit 52 Transmission unit 53 Reception portion 54 Reception portion 61 Camera (image acquisition unit) 62 Pupil image capturing camera (visual line detection unit) 63 Pupil image capturing camera (visual line detection unit) 64 Near-infrared-ray receiving and emitting unit (visual line detection unit) 65 Near-infrared-ray receiving and emitting unit (visual line detection unit) 66 Nine-axis sensor (motion detection unit) 67 Depth sensor (position detection unit) 110 Input information acquisition unit 100 Head mounted display 120 Storage unit 122 Invisible information storage portion 124 Appearance information storage portion 126 Procedure table 128 Parallax information 130 Power supply 140 CPU 142 Superimposition processing unit 144 Procedure management unit 160 Image processing unit 1
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Physics & Mathematics (AREA)
- Medical Informatics (AREA)
- Optics & Photonics (AREA)
- General Physics & Mathematics (AREA)
- Heart & Thoracic Surgery (AREA)
- General Health & Medical Sciences (AREA)
- Biophysics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Pathology (AREA)
- Radiology & Medical Imaging (AREA)
- Biomedical Technology (AREA)
- Veterinary Medicine (AREA)
- Molecular Biology (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- High Energy & Nuclear Physics (AREA)
- Public Health (AREA)
- Human Computer Interaction (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Controls And Circuits For Display Device (AREA)
- User Interface Of Digital Computer (AREA)
- Instrument Panels (AREA)
Abstract
Description
A-1. Configuration of head mounted display
Fig. 1 is a diagram illustrating a schematic configuration of a head mounted display (HMD) according to an embodiment of the invention. The head mounted
Fig. 5 is a flowchart illustrating procedures of a superimposition information display process according to a first embodiment. Steps S102 and S104 of Fig. 5 indicate procedures of a preprocess of the superimposition information display process. The procedures from step S106 of Fig. 5 indicate procedures of the superimposition information display process. The superimposition information display process is a process of superimposing information which is no shown in an appearance of an object on a real image of the object in a visual field direction of the user for display.
Fig. 8 is a flowchart illustrating procedures of the superimposition information generation process. The
Method ii) The
In addition, the following additional processes may be further performed in the superimposition information display process (Fig. 5). The additional processes may be added singly or added in combination.
In the
* Information which displays a caution or a warning based on a measurement value of a medical instrument attached to a patient
In the
In the
In the
Procedure a2) In a case where the change amount obtained in the procedure a1 is larger than a predetermined threshold value, the processes in and after step S204 are successively performed. In a case where the change amount obtained in the procedure a1 is equal to or smaller than the predetermined threshold value, the superimposition information CP1 generated due to the previously performed superimposition information generation process is set as a processing result of the currently performed superimposition information generation process.
In the
Procedure b2) The
Procedure b3) The
In the
In the
Procedure c2) The
Procedure c4) The
Procedure c5) The
Procedure c7) The
In the second embodiment of the invention, a description will be made of a configuration in which a head mounted display can further support work performed by a user. Hereinafter, only parts having configurations and operations different from the first embodiment will be described. In addition, in the drawings, the same constituent elements as in the first embodiment are given the same reference numeral as in the first embodiment, and detailed description thereof will be omitted.
Fig. 13 is a functional block diagram illustrating a configuration of a head mounted
Fig. 15 is a flowchart illustrating procedures of the superimposition information display process according to the second embodiment. A difference from the first embodiment illustrated in Fig. 5 is that steps S302, S304 and S306 are added.
Procedure d2) The
Procedure d3) The
Procedure d4) The
Procedure e2) The
Procedure e4) The
Procedure e5) The
Procedure e6) The
Procedure e7) The
Procedure e9) The
In addition, in the superimposition information display process (Fig. 15) according to the second embodiment, the
In the third embodiment of the invention, a description will be made of a configuration capable of further improving accuracy of superimposition of "information (superimposition information) which is not shown in an appearance of an object" on a "real image of the object which is a target of the superimposition information display process" in the superimposition information display process in the first embodiment. Hereinafter, only parts having configurations and operations different from the first embodiment will be described. In addition, in the drawings, the same constituent elements as in the first embodiment are given the same reference numeral as in the first embodiment, and detailed description thereof will be omitted.
Fig. 19 is a diagram illustrating a schematic configuration of a head mounted
Fig. 22 is a flowchart illustrating procedures of the superimposition information display process according to the third embodiment. A difference from the first embodiment illustrated in Fig. 5 is that: step S602 is provided instead of step S102; step S604 is provided instead of step S108; step S606 is provided instead of step S110; and step S608 is further provided after step S110. In addition, the superimposition information display process according to the third embodiment is performed in a three-dimensional display mode.
Fig. 24 is a flowchart illustrating procedures of the first superimposition information generation process. In step S402, the
Fig. 26 is a flowchart illustrating procedures of the second superimposition information generation process. A difference from the first superimposition information generation process illustrated in Fig. 24 is that step S502 is provided instead of step S404, and step S504 is provided instead of step S412.
In addition, in the superimposition information display process (Fig. 22), the
In the above-described embodiments, some of the constituent elements realized in hardware may be realized in software, and, conversely, some of the configurations realized in software may be realized in hardware. In addition, the following modifications may also occur.
In the above-described embodiments, a configuration of the head mounted display has been exemplified. However, any configuration of the head mounted display may be defined within the scope without departing from the spirit of the invention, and, for example, each configuration unit may be added, deleted, changed, or the like.
In the above-described embodiment, an example of the superimposition information display process (Figs. 5 and 22) has been described. However, the procedures of the process illustrated in Figs. 5 and 22 are only an example and may have various modifications. For example, some steps may be omitted, and other steps may be further added. In addition, order of steps to be performed may be changed.
In the second embodiment, an example of the superimposition information display process (Fig. 15) has been described. However, the procedures of the process illustrated in Fig. 15 are only an example and may have various modifications. For example, some steps may be omitted, and other steps may be further added. In addition, order of steps to be performed may be changed.
* In the procedure, matters to which the user should pay particular attention are displayed in a pop-up form.
* An image or a character string in the procedure information is displayed in a blinking state.
* A message or an icon indicating the degree of importance, the degree of difficulty, a frequency of accidents, and the like of a corresponding procedure is displayed in the procedure information.
In the above-described embodiments, an example of the procedure table has been described. However, the details of the procedure table are only an example and may have various modifications. For example, a field may be added, deleted, or changed. In addition, the procedure table may be divided into a plurality of tables so as to be normalized.
In the second embodiment, a description has been made of the configuration in which the head mounted display is provided with the procedure table and the procedure management unit, and can support work performed by the user without requiring other devices. However, also in a work supporting system including the head mounted display and an externally provided server, the same process may be performed.
In the third embodiment, an example of the first and second superimposition information generation processes (Figs. 24 and 26) has been described. However, the procedures of the process illustrated in Figs. 24 and 26 are only an example and may have various modifications. For example, some steps may be omitted, and other steps may be further added. In addition, order of steps to be performed may be changed.
The invention is not limited to the above-described embodiments or modification examples, and may be implemented using various configurations within the scope without departing from the spirit thereof. For example, the embodiments, the examples, and the modification examples corresponding to technical features of the respective aspects described in Summary of Invention and the technical features in the modification examples may be exchanged or combined as appropriate in order to solve some or all of the above-described problems, or in order to achieve some or all of the above-described effects. In addition, if the technical feature is not described as an essential feature in the present specification, the technical feature may be deleted as appropriate.
12 Lighting unit
14 Touch pad
16 Cross key
18 Power switch
20 Image display section
21 Right holding unit
22 Right display driving unit
23 Left holding unit
24 Left display driving unit
26 Right optical image display unit
28 Left optical image display unit
30 Earphone plug
32 Right earphone
34 Left earphone
40 Connection unit
42 Right cord
44 Left cord
46 Connection member
48 Main body cord
51 Transmission unit
52 Transmission unit
53 Reception portion
54 Reception portion
61 Camera (image acquisition unit)
62 Pupil image capturing camera (visual line detection unit)
63 Pupil image capturing camera (visual line detection unit)
64 Near-infrared-ray receiving and emitting unit (visual line detection unit)
65 Near-infrared-ray receiving and emitting unit (visual line detection unit)
66 Nine-axis sensor (motion detection unit)
67 Depth sensor (position detection unit)
110 Input information acquisition unit
100 Head mounted display
120 Storage unit
122 Invisible information storage portion
124 Appearance information storage portion
126 Procedure table
128 Parallax information
130 Power supply
140 CPU
142 Superimposition processing unit
144 Procedure management unit
160 Image processing unit
170 Sound processing unit
180 Interface
190 Display control unit
201 Right backlight control portion
202 Left backlight control portion
211 Right LCD control portion
212 Left LCD control portion
221 Right backlight
222 Left backlight
241 Right LCD
242 Left LCD
251 Right projection optical system
252 Left projection optical system
261 Right light guiding plate
262 Left light guiding plate
PCLK Clock signal
VSync Vertical synchronization signal
HSync Horizontal synchronization signal
Data Image data
Data1 Right eye image data
Data2 Left eye image data
OA External apparatus
PC Personal Computer
SC External scenery
RE Right eye
VI Virtual image
EL End part
IL Illumination light
PL Image light
AP Apex
ER End part
VR Visual field
CP1 Superimposition information
CP2 Enlargement information
CP3 Auxiliary information
CP4 Procedure information
CP5 Superimposition information
DP Depth
Lx Distance
PP Steady gaze point
Px Parallax
SI Invisible information
SI1 Coronary plane CT image
SI2 Transverse cross-sectional CT image
SI3 X-ray image
Claims (24)
- A head mounted display which allows a user to visually recognize a virtual image and external scenery, comprising:
an image display unit that forms the virtual image which is visually recognized by the user; and
a superimposition processing unit that causes the image display unit to form the virtual image based on superimposition information for superimposing invisible information which is not shown in an appearance of an object on the object included in the external scenery. - The head mounted display according to claim 1,
wherein the superimposition processing unit causes the image display unit to repeatedly form the virtual image based on the superimposition information corresponding to the object included in the external scenery every predetermined time. - The head mounted display according to claim 1, further comprising:
a storage unit that stores the invisible information; and
an image acquisition unit that acquires an image of external scenery in a visual field direction of the user when the user wears the head mounted display,
wherein the superimposition processing unit extracts a feature of the object included in the external scenery through image recognition on the image in the visual field direction acquired by the image acquisition unit, and corrects the invisible information of the object stored in the storage unit on the basis of the extracted feature so as to generate the superimposition information. - The head mounted display according to claim 3,
wherein the invisible information is an image indicating a structure of the object. - The head mounted display according to claim 3,
wherein the feature is an edge of the object. - The head mounted display according to claim 3,
wherein the feature is a marker attached to the object. - The head mounted display according to claim 3,
wherein the correction is performed by performing at least one of enlargement, reduction, rotation, inversion, trimming, distortion, and noise removal on an image indicating the invisible information. - The head mounted display according to claim 3,
wherein the correction is performed by dividing an image indicating the invisible information into a plurality of regions and performing at least one of enlargement, reduction, trimming, distortion, and noise removal on each of the plurality of regions. - The head mounted display according to claim 8,
wherein the superimposition processing unit measures a distance between the user and each part of the object by using triangulation, and performs the division on the basis of a result of the measurement. - The head mounted display according to claim 3,
wherein the superimposition processing unit changes accuracy of the correction on the basis of an information accepting characteristic indicating an extent of visual capability within a visual field of the user. - The head mounted display according to claim 3, further comprising:
a visual line detection unit that detects a visual line of the user,
wherein the image display unit forms both the virtual image corresponding to the right eye of the user and the virtual image corresponding to the left eye of the user,
wherein the superimposition processing unit further obtains a steady gaze point distance which is a distance between the user and a steady gaze point at which the user steadily gazes, from the visual line detected by the visual line detection unit, and generates the superimposition information for the right eye and the superimposition information for the left eye to which a parallax corresponding to the obtained steady gaze point distance is given. - The head mounted display according to claim 11, further comprising:
parallax information that correlates a plurality of distances between the user and the virtual image visually recognized by the user with parallaxes which respectively correspond to the plurality of distances,
wherein the superimposition processing unit gives a parallax corresponding to the steady gaze point distance by using the parallax information. - The head mounted display according to claim 12,
wherein the parallax information is provided in a plurality of pieces, and
wherein the superimposition processing unit selects one parallax information piece from the plurality of parallax information pieces in accordance with the user. - The head mounted display according to claim 3, further comprising:
a position detection unit that detects a position of the user relative to the object,
wherein the storage unit stores a plurality of invisible information pieces regarding a single object,
wherein the superimposition processing unit selects one invisible information piece corresponding to the position detected by the position detection unit from the plurality of invisible information pieces stored in the storage unit, and corrects the selected invisible information. - The head mounted display according to claim 3,
wherein the superimposition processing unit further generates enlargement information in which at least a part of the object included in the external scenery is extracted and enlarged, from the image in the visual field direction acquired by the image acquisition unit, and
wherein the image display unit allows the user to visually recognize the virtual image based on the superimposition information and the enlargement information. - The head mounted display according to claim 1,
wherein the superimposition processing unit does not display the virtual image in response to a request from the user. - The head mounted display according to claim 1, further comprising:
a procedure management unit that generates procedure information which is information regarding content which is to be performed by the user in a current procedure,
wherein the image display unit allows the user to visually recognize the virtual image based on the superimposition information and the procedure information. - The head mounted display according to claim 17,
wherein the procedure information includes at least one of an image of a scene in which a third party exemplarily performs the procedure and an image indicating content to be performed by the user when the individual specific procedure is performed. - The head mounted display according to claim 1,
wherein the superimposition processing unit further generates auxiliary information which is additional information for assisting work of the user, and
wherein the image display unit allows the user to visually recognize the virtual image based on the superimposition information and the auxiliary information. - The head mounted display according to claim 3,
wherein the superimposition processing unit further performs a color conversion process for improving visibility of the generated superimposition information on the basis of the image in the visual field direction acquired by the image acquisition unit. - The head mounted display according to claim 1,
wherein the superimposition processing unit withdraws the superimposition information which is displayed as the virtual image to a position which does not hinder the visual field of the user, in response to a request from the user. - The head mounted display according to claim 3,
wherein the superimposition processing unit generates the superimposition information corresponding to an overlap region in which a display region of the virtual image overlaps with an acquisition region of the image acquisition unit in the image acquired by the image acquisition unit. - The head mounted display according to claim 16,
wherein the procedure management unit generates the procedure information corresponding to an overlap region in which a display region of the virtual image overlaps with an acquisition region of the image acquisition unit in the image acquired by the image acquisition unit. - A control method for a head mounted display which allows a user to visually recognize a virtual image and external scenery, the method comprising:
(a) forming the virtual image which is visually recognized by the user; and
(b) forming the virtual image based on superimposition information for superimposing invisible information which is not shown in an appearance of an object on the object included in the external scenery.
Priority Applications (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020157024204A KR101811817B1 (en) | 2013-02-14 | 2014-02-04 | Head mounted display and control method for head mounted display |
US14/764,553 US9916691B2 (en) | 2013-02-14 | 2014-02-04 | Head mounted display and control method for head mounted display |
EP14724164.0A EP2956814A1 (en) | 2013-02-14 | 2014-02-04 | Head mounted display and control method for head mounted display |
BR112015019674A BR112015019674A2 (en) | 2013-02-14 | 2014-02-04 | head mounted display and control method for head mounted display |
RU2015138973A RU2621488C2 (en) | 2013-02-14 | 2014-02-04 | Display fixed on head and method of controlling display fixed on head |
US15/880,023 US10169925B2 (en) | 2013-02-14 | 2018-01-25 | Head mounted display and control method for head mounted display |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2013026222A JP6299067B2 (en) | 2013-02-14 | 2013-02-14 | Head-mounted display device and method for controlling head-mounted display device |
JP2013-026222 | 2013-02-14 | ||
JP2013237820A JP6264855B2 (en) | 2013-11-18 | 2013-11-18 | Head-mounted display device and method for controlling head-mounted display device |
JP2013-237820 | 2013-11-18 |
Related Child Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/764,553 A-371-Of-International US9916691B2 (en) | 2013-02-14 | 2014-02-04 | Head mounted display and control method for head mounted display |
US15/880,023 Continuation US10169925B2 (en) | 2013-02-14 | 2018-01-25 | Head mounted display and control method for head mounted display |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2014125789A1 true WO2014125789A1 (en) | 2014-08-21 |
Family
ID=50729746
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2014/000595 WO2014125789A1 (en) | 2013-02-14 | 2014-02-04 | Head mounted display and control method for head mounted display |
Country Status (8)
Country | Link |
---|---|
US (2) | US9916691B2 (en) |
EP (1) | EP2956814A1 (en) |
KR (1) | KR101811817B1 (en) |
CN (2) | CN109270688B (en) |
BR (1) | BR112015019674A2 (en) |
RU (1) | RU2621488C2 (en) |
TW (1) | TW201432312A (en) |
WO (1) | WO2014125789A1 (en) |
Cited By (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105425951A (en) * | 2015-11-04 | 2016-03-23 | 深圳维爱特科技有限公司 | Application interaction control method and device |
EP3213673A1 (en) * | 2016-03-01 | 2017-09-06 | Shanghai Xiaoyi Technology Co., Ltd. | Smart sports eyewear |
US9928629B2 (en) | 2015-03-24 | 2018-03-27 | Augmedics Ltd. | Combining video-based and optic-based augmented reality in a near eye display |
EP3204816A4 (en) * | 2014-10-07 | 2018-06-20 | Elbit Systems Ltd. | Head-mounted displaying of magnified images locked on an object of interest |
EP3232654A4 (en) * | 2014-12-11 | 2018-11-21 | Sony Corporation | Spectacle-type display device for medical use, information processing device, and information processing method |
EP3258876A4 (en) * | 2015-02-20 | 2018-12-26 | Covidien LP | Operating room and surgical site awareness |
CN109308741A (en) * | 2018-08-08 | 2019-02-05 | 长春理工大学 | A kind of natural interaction craftwork creative design system based on Meta2 |
US10939977B2 (en) | 2018-11-26 | 2021-03-09 | Augmedics Ltd. | Positioning marker |
CN113453639A (en) * | 2019-02-25 | 2021-09-28 | 奥林巴斯株式会社 | Centralized control device |
US11389252B2 (en) | 2020-06-15 | 2022-07-19 | Augmedics Ltd. | Rotating marker for image guided surgery |
US11727649B2 (en) | 2015-09-28 | 2023-08-15 | Montefiore Medical Center | Methods and devices for intraoperative viewing of patient 3D surface images |
US11766296B2 (en) | 2018-11-26 | 2023-09-26 | Augmedics Ltd. | Tracking system for image-guided surgery |
US11801115B2 (en) | 2019-12-22 | 2023-10-31 | Augmedics Ltd. | Mirroring in image guided surgery |
US11896445B2 (en) | 2021-07-07 | 2024-02-13 | Augmedics Ltd. | Iliac pin and adapter |
US11974887B2 (en) | 2018-05-02 | 2024-05-07 | Augmedics Ltd. | Registration marker for an augmented reality system |
US11980506B2 (en) | 2019-07-29 | 2024-05-14 | Augmedics Ltd. | Fiducial marker |
Families Citing this family (69)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10262462B2 (en) | 2014-04-18 | 2019-04-16 | Magic Leap, Inc. | Systems and methods for augmented and virtual reality |
US11103122B2 (en) * | 2014-07-15 | 2021-08-31 | Mentor Acquisition One, Llc | Content presentation in head worn computing |
US20160019715A1 (en) * | 2014-07-15 | 2016-01-21 | Osterhout Group, Inc. | Content presentation in head worn computing |
US9753288B2 (en) | 2014-01-21 | 2017-09-05 | Osterhout Group, Inc. | See-through computer display systems |
JP6264087B2 (en) * | 2014-02-21 | 2018-01-24 | ソニー株式会社 | Display control device, display device, and display control system |
JP6323183B2 (en) * | 2014-06-04 | 2018-05-16 | ソニー株式会社 | Image processing apparatus and image processing method |
US9959591B2 (en) * | 2014-07-31 | 2018-05-01 | Seiko Epson Corporation | Display apparatus, method for controlling display apparatus, and program |
EP2996017B1 (en) | 2014-09-11 | 2022-05-11 | Nokia Technologies Oy | Method, apparatus and computer program for displaying an image of a physical keyboard on a head mountable display |
EP3216023A4 (en) * | 2014-11-07 | 2018-07-18 | Eye Labs, LLC | Visual stabilization system for head-mounted displays |
CN105607253B (en) | 2014-11-17 | 2020-05-12 | 精工爱普生株式会社 | Head-mounted display device, control method, and display system |
WO2016162789A2 (en) * | 2015-04-07 | 2016-10-13 | King Abdullah University Of Science And Technology | Method, apparatus, and system for utilizing augmented reality to improve surgery |
US9724562B2 (en) * | 2015-05-28 | 2017-08-08 | Nike, Inc. | Wearable USB device assembly |
CN106681488A (en) * | 2015-07-20 | 2017-05-17 | 谢培树 | Digital eyewear operated by head |
CN105487660A (en) * | 2015-11-25 | 2016-04-13 | 北京理工大学 | Immersion type stage performance interaction method and system based on virtual reality technology |
JP6644530B2 (en) * | 2015-11-30 | 2020-02-12 | オリンパス株式会社 | Endoscope business support system |
EP3397185A1 (en) * | 2015-12-29 | 2018-11-07 | Koninklijke Philips N.V. | System, controller and method using virtual reality device for robotic surgery |
IL243422B (en) | 2015-12-30 | 2018-04-30 | Elbit Systems Ltd | Managing displayed information according to user gaze directions |
US10850116B2 (en) | 2016-12-30 | 2020-12-01 | Mentor Acquisition One, Llc | Head-worn therapy device |
JP2017215664A (en) * | 2016-05-30 | 2017-12-07 | サン電子株式会社 | Terminal device |
US10037080B2 (en) | 2016-05-31 | 2018-07-31 | Paypal, Inc. | User physical attribute based device and content management system |
US9798385B1 (en) | 2016-05-31 | 2017-10-24 | Paypal, Inc. | User physical attribute based device and content management system |
US10902796B1 (en) * | 2016-09-12 | 2021-01-26 | Apple Inc. | Multi-user display system |
US11839433B2 (en) * | 2016-09-22 | 2023-12-12 | Medtronic Navigation, Inc. | System for guided procedures |
JP6812193B2 (en) * | 2016-10-07 | 2021-01-13 | キヤノン株式会社 | Image display system, image display method, and program |
US10375266B2 (en) * | 2016-10-26 | 2019-08-06 | Orcam Technologies Ltd. | Systems and methods for selecting an action based on a detected person |
US10203505B2 (en) * | 2016-11-16 | 2019-02-12 | Seiko Epson Corporation | Feature balancing |
WO2018097831A1 (en) | 2016-11-24 | 2018-05-31 | Smith Joshua R | Light field capture and rendering for head-mounted displays |
JP6617686B2 (en) | 2016-11-25 | 2019-12-11 | 株式会社デンソー | Wireless power feeding system and power transmission / reception monitoring program |
CN109983532A (en) * | 2016-11-29 | 2019-07-05 | 夏普株式会社 | Display control unit, head-mounted display, the control method of display control unit and control program |
US10235788B2 (en) * | 2017-01-17 | 2019-03-19 | Opentv, Inc. | Overlay contrast control in augmented reality displays |
US10152815B2 (en) * | 2017-01-17 | 2018-12-11 | Opentv, Inc. | Overlay emphasis modification in augmented reality displays |
CN110199357B (en) * | 2017-01-17 | 2023-07-14 | 皇家飞利浦有限公司 | Augmented reality interventional system providing background overlay |
JP6828465B2 (en) * | 2017-01-30 | 2021-02-10 | セイコーエプソン株式会社 | Endoscope operation support system |
US11219422B2 (en) * | 2017-03-14 | 2022-01-11 | Canon Medical Systems Corporation | Image displaying system, image processing apparatus and x-ray imaging system |
CN108664037B (en) * | 2017-03-28 | 2023-04-07 | 精工爱普生株式会社 | Head-mounted display device and method for operating unmanned aerial vehicle |
US10610179B2 (en) * | 2017-06-05 | 2020-04-07 | Biosense Webster (Israel) Ltd. | Augmented reality goggles having X-ray protection |
EP3639083A4 (en) * | 2017-06-14 | 2021-01-27 | Hewlett-Packard Development Company, L.P. | Display adjustments |
JP2019022084A (en) * | 2017-07-18 | 2019-02-07 | オリンパス株式会社 | Wearable device and control method thereof |
CN107275904B (en) | 2017-07-19 | 2024-05-24 | 北京小米移动软件有限公司 | Data connecting wire of virtual reality glasses |
RU176382U1 (en) * | 2017-08-28 | 2018-01-17 | Общество с ограниченной ответственностью "ТрансИнжКом" | INFORMATION GATHERING UNIT FOR A JOINT REALITY DEVICE |
RU2669228C1 (en) * | 2017-09-20 | 2018-10-09 | Галина Александровна Купцова | Spheroperimeter |
WO2019076465A1 (en) * | 2017-10-20 | 2019-04-25 | Huawei Technologies Co., Ltd. | Wearable device and method therein |
JP7036572B2 (en) | 2017-11-02 | 2022-03-15 | マクセル株式会社 | Virtual image display device and head-mounted display using it |
CZ2017777A3 (en) * | 2017-12-05 | 2019-07-03 | Radalytica s.r.o. | A non-destructive method of imaging an internal structure and a device for doing this |
US20190175059A1 (en) | 2017-12-07 | 2019-06-13 | Medtronic Xomed, Inc. | System and Method for Assisting Visualization During a Procedure |
WO2019134757A1 (en) * | 2018-01-08 | 2019-07-11 | Brainlab Ag | Optimizing an atlas |
US10554883B2 (en) * | 2018-03-30 | 2020-02-04 | Ricoh Company, Ltd. | VR system, communication method, and non-transitory computer-readable medium |
DE102018109463C5 (en) * | 2018-04-19 | 2023-10-05 | Voraus Robotik Gmbh | Method for using a multi-unit actuated kinematics, preferably a robot, particularly preferably an articulated robot, by a user using a mobile display device |
WO2019235298A1 (en) | 2018-06-07 | 2019-12-12 | 富士フイルム株式会社 | Image diagnostic assistance device, image diagnostic assistance method, and image diagnostic assistance program |
TWI671707B (en) * | 2018-06-15 | 2019-09-11 | 緯創資通股份有限公司 | Image analysis method, electronic system and non-transitory computer-readable recording medium |
GB2575651B (en) * | 2018-07-17 | 2020-12-16 | Envisics Ltd | Image projector |
WO2020102665A1 (en) * | 2018-11-16 | 2020-05-22 | Lang Philipp K | Augmented reality guidance for surgical procedures with adjustment of scale, convergence and focal plane or focal point of virtual data |
CN109729347B (en) * | 2019-01-23 | 2020-11-24 | 歌尔光学科技有限公司 | Virtual image multiplying power test method and device and virtual display equipment |
CN110222639B (en) * | 2019-06-05 | 2020-03-31 | 清华大学 | Human body stress response testing method and system |
JP2021009270A (en) | 2019-07-03 | 2021-01-28 | セイコーエプソン株式会社 | Display unit and display method |
IT201900013164A1 (en) * | 2019-07-29 | 2021-01-29 | Eye Tech Lab S R L | AUGMENTED REALITY MAGNIFYING GLASSES |
JP2021057737A (en) * | 2019-09-30 | 2021-04-08 | セイコーエプソン株式会社 | Display system, display control method, and program |
US10702224B1 (en) * | 2019-10-24 | 2020-07-07 | Jonathan Ross Vanhooser | Positioning guidance system For X-ray exams |
US11006093B1 (en) | 2020-01-22 | 2021-05-11 | Photonic Medical Inc. | Open view, multi-modal, calibrated digital loupe with depth sensing |
US11623086B2 (en) | 2020-01-24 | 2023-04-11 | Medtronic Xomed, Inc. | System and method for therapy |
US11666755B2 (en) | 2020-01-24 | 2023-06-06 | Medtronic Xomed, Inc. | System and method for therapy |
US11167140B2 (en) | 2020-01-24 | 2021-11-09 | Medtronic Xomed, Inc. | System and method for therapy |
US11167127B2 (en) | 2020-01-24 | 2021-11-09 | Medtronic Xomed, Inc. | System and method for therapy |
CN111563879B (en) * | 2020-03-27 | 2023-09-26 | 北京视博云信息技术有限公司 | Method and device for detecting display quality of application picture |
RU2757991C2 (en) * | 2020-07-06 | 2021-10-25 | Общество с ограниченной ответственностью "Толикети" | Method for automated control of a robotic operational exoscope |
RU201742U1 (en) * | 2020-07-22 | 2020-12-30 | Общество с ограниченной ответственностью "МИКСАР ДЕВЕЛОПМЕНТ" | Augmented reality glasses for use in hazardous production environments |
US11295460B1 (en) | 2021-01-04 | 2022-04-05 | Proprio, Inc. | Methods and systems for registering preoperative image data to intraoperative image data of a scene, such as a surgical scene |
TWI782885B (en) * | 2022-04-25 | 2022-11-01 | 國立臺灣海洋大學 | Posture and food intake correlation detection system for rheumatoid arthritis |
TWI820780B (en) * | 2022-07-01 | 2023-11-01 | 亞東學校財團法人亞東科技大學 | Head-mounted medical aid for bidirectional communication |
Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3599854B2 (en) | 1995-10-09 | 2004-12-08 | オリンパス株式会社 | Medical face-mounted video display |
EP1929956A2 (en) * | 2006-12-08 | 2008-06-11 | Biosense Webster, Inc. | Coloring electroanatomical maps to indicate ultrasound data acquisiton |
US20080246694A1 (en) * | 2007-04-06 | 2008-10-09 | Ronald Fischer | Personal theater display |
US20090034820A1 (en) * | 2007-05-08 | 2009-02-05 | Canon Kabushiki Kaisha | Diagnostic imaging system |
US20090110291A1 (en) * | 2007-10-30 | 2009-04-30 | Canon Kabushiki Kaisha | Image processing apparatus and image processing method |
US20090274271A1 (en) * | 2008-05-02 | 2009-11-05 | Siemens Medical Solutions Usa, Inc. | System and method for selecting a guidance mode for performing a percutaneous procedure |
US20100026714A1 (en) * | 2008-07-31 | 2010-02-04 | Canon Kabushiki Kaisha | Mixed reality presentation system |
US20110052083A1 (en) * | 2009-09-02 | 2011-03-03 | Junichi Rekimoto | Information providing method and apparatus, information display method and mobile terminal, program, and information providing system |
US20120275721A1 (en) * | 2009-12-24 | 2012-11-01 | Bae Systems Plc | Image enhancement |
JP2012223363A (en) | 2011-04-20 | 2012-11-15 | Tokyo Institute Of Technology | Surgical imaging system and surgical robot |
Family Cites Families (35)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH0888815A (en) * | 1994-09-19 | 1996-04-02 | Olympus Optical Co Ltd | Video display system |
JP3625906B2 (en) | 1995-08-23 | 2005-03-02 | オリンパス株式会社 | Surgical microscope equipment |
JPH10221637A (en) | 1997-02-10 | 1998-08-21 | Olympus Optical Co Ltd | Head-mounted image display device |
JPH11309A (en) | 1997-06-12 | 1999-01-06 | Hitachi Ltd | Image processor |
JP2001111991A (en) | 1999-10-08 | 2001-04-20 | Olympus Optical Co Ltd | Face-mounted video display device |
US20040238732A1 (en) * | 2001-10-19 | 2004-12-02 | Andrei State | Methods and systems for dynamic virtual convergence and head mountable display |
US20070015999A1 (en) * | 2005-07-15 | 2007-01-18 | Heldreth Mark A | System and method for providing orthopaedic surgical information to a surgeon |
US20070238981A1 (en) * | 2006-03-13 | 2007-10-11 | Bracco Imaging Spa | Methods and apparatuses for recording and reviewing surgical navigation processes |
JP4961914B2 (en) | 2006-09-08 | 2012-06-27 | ソニー株式会社 | Imaging display device and imaging display method |
US7855743B2 (en) | 2006-09-08 | 2010-12-21 | Sony Corporation | Image capturing and displaying apparatus and image capturing and displaying method |
JP4455609B2 (en) | 2007-03-27 | 2010-04-21 | 株式会社東芝 | Image display device and image display method |
US8165808B2 (en) * | 2007-07-17 | 2012-04-24 | Yahoo! Inc. | Techniques for representing location information |
EP2269533B1 (en) | 2008-03-21 | 2021-05-05 | Atsushi Takahashi | Three-dimensional digital magnifier operation supporting system |
JP2009279193A (en) | 2008-05-22 | 2009-12-03 | Fujifilm Corp | Medical apparatus management system |
US8957835B2 (en) | 2008-09-30 | 2015-02-17 | Apple Inc. | Head-mounted display apparatus for retaining a portable electronic device with display |
JP5476036B2 (en) * | 2009-04-30 | 2014-04-23 | 国立大学法人大阪大学 | Surgical navigation system using retinal projection type head mounted display device and simulation image superimposing method |
PL2430396T3 (en) * | 2009-05-16 | 2020-11-16 | Rapiscan Systems, Inc. | Systems and methods for automated, rapid detection of high-atomic-number materials |
JP5402293B2 (en) * | 2009-06-22 | 2014-01-29 | ソニー株式会社 | Head-mounted display and image display method in head-mounted display |
CN101661163A (en) * | 2009-09-27 | 2010-03-03 | 合肥工业大学 | Three-dimensional helmet display of augmented reality system |
WO2011039681A1 (en) * | 2009-09-29 | 2011-04-07 | Koninklijke Philips Electronics N.V. | Live registration for vessel treatment |
US9162061B2 (en) * | 2009-10-09 | 2015-10-20 | National Ict Australia Limited | Vision enhancement for a vision impaired user |
JP2011082919A (en) * | 2009-10-09 | 2011-04-21 | Sony Corp | Image processing device and method, and program |
JP4679661B1 (en) | 2009-12-15 | 2011-04-27 | 株式会社東芝 | Information presenting apparatus, information presenting method, and program |
US20120200601A1 (en) | 2010-02-28 | 2012-08-09 | Osterhout Group, Inc. | Ar glasses with state triggered eye control interaction with advertising facility |
US20120212484A1 (en) | 2010-02-28 | 2012-08-23 | Osterhout Group, Inc. | System and method for display content placement using distance and location information |
FR2957000B1 (en) * | 2010-03-02 | 2012-03-02 | Commissariat Energie Atomique | METHOD AND SYSTEM FOR HANDLING ROBOTIC MACHINERY IN A CONCEALED ENVIRONMENT |
JP4934228B2 (en) | 2010-06-17 | 2012-05-16 | 新日鉄ソリューションズ株式会社 | Information processing apparatus, information processing method, and program |
WO2011122610A1 (en) | 2010-03-30 | 2011-10-06 | 新日鉄ソリューションズ株式会社 | Information processing apparatus, information processing method, and program |
WO2011123648A2 (en) * | 2010-03-31 | 2011-10-06 | MyLife Global, Inc. | Medical imaging system and related methods |
US9122053B2 (en) * | 2010-10-15 | 2015-09-01 | Microsoft Technology Licensing, Llc | Realistic occlusion for a head mounted augmented reality display |
US8884984B2 (en) * | 2010-10-15 | 2014-11-11 | Microsoft Corporation | Fusing virtual content into real content |
JP5725953B2 (en) * | 2011-04-13 | 2015-05-27 | キヤノン株式会社 | Imaging apparatus, control method therefor, and information processing apparatus |
JP5011455B1 (en) | 2011-06-30 | 2012-08-29 | 医療法人社団神▲崎▼循環器クリニック | X-ray diagnostic equipment |
US8832233B1 (en) * | 2011-07-20 | 2014-09-09 | Google Inc. | Experience sharing for conveying communication status |
CN102609849A (en) | 2012-02-03 | 2012-07-25 | 纽海信息技术(上海)有限公司 | Panoramic exhibiting system and panoramic exhibiting method of commodities |
-
2014
- 2014-02-04 RU RU2015138973A patent/RU2621488C2/en active
- 2014-02-04 WO PCT/JP2014/000595 patent/WO2014125789A1/en active Application Filing
- 2014-02-04 BR BR112015019674A patent/BR112015019674A2/en active Search and Examination
- 2014-02-04 US US14/764,553 patent/US9916691B2/en active Active
- 2014-02-04 EP EP14724164.0A patent/EP2956814A1/en not_active Ceased
- 2014-02-04 KR KR1020157024204A patent/KR101811817B1/en active IP Right Grant
- 2014-02-11 TW TW103104473A patent/TW201432312A/en unknown
- 2014-02-12 CN CN201811214129.7A patent/CN109270688B/en active Active
- 2014-02-12 CN CN201410048928.7A patent/CN103995352B/en active Active
-
2018
- 2018-01-25 US US15/880,023 patent/US10169925B2/en active Active
Patent Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3599854B2 (en) | 1995-10-09 | 2004-12-08 | オリンパス株式会社 | Medical face-mounted video display |
EP1929956A2 (en) * | 2006-12-08 | 2008-06-11 | Biosense Webster, Inc. | Coloring electroanatomical maps to indicate ultrasound data acquisiton |
US20080246694A1 (en) * | 2007-04-06 | 2008-10-09 | Ronald Fischer | Personal theater display |
US20090034820A1 (en) * | 2007-05-08 | 2009-02-05 | Canon Kabushiki Kaisha | Diagnostic imaging system |
US20090110291A1 (en) * | 2007-10-30 | 2009-04-30 | Canon Kabushiki Kaisha | Image processing apparatus and image processing method |
US20090274271A1 (en) * | 2008-05-02 | 2009-11-05 | Siemens Medical Solutions Usa, Inc. | System and method for selecting a guidance mode for performing a percutaneous procedure |
US20100026714A1 (en) * | 2008-07-31 | 2010-02-04 | Canon Kabushiki Kaisha | Mixed reality presentation system |
US20110052083A1 (en) * | 2009-09-02 | 2011-03-03 | Junichi Rekimoto | Information providing method and apparatus, information display method and mobile terminal, program, and information providing system |
US20120275721A1 (en) * | 2009-12-24 | 2012-11-01 | Bae Systems Plc | Image enhancement |
JP2012223363A (en) | 2011-04-20 | 2012-11-15 | Tokyo Institute Of Technology | Surgical imaging system and surgical robot |
Non-Patent Citations (1)
Title |
---|
See also references of EP2956814A1 |
Cited By (22)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP3204816A4 (en) * | 2014-10-07 | 2018-06-20 | Elbit Systems Ltd. | Head-mounted displaying of magnified images locked on an object of interest |
EP3232654A4 (en) * | 2014-12-11 | 2018-11-21 | Sony Corporation | Spectacle-type display device for medical use, information processing device, and information processing method |
US10908681B2 (en) | 2015-02-20 | 2021-02-02 | Covidien Lp | Operating room and surgical site awareness |
EP3258876A4 (en) * | 2015-02-20 | 2018-12-26 | Covidien LP | Operating room and surgical site awareness |
US11750794B2 (en) | 2015-03-24 | 2023-09-05 | Augmedics Ltd. | Combining video-based and optic-based augmented reality in a near eye display |
US9928629B2 (en) | 2015-03-24 | 2018-03-27 | Augmedics Ltd. | Combining video-based and optic-based augmented reality in a near eye display |
US11727649B2 (en) | 2015-09-28 | 2023-08-15 | Montefiore Medical Center | Methods and devices for intraoperative viewing of patient 3D surface images |
CN105425951A (en) * | 2015-11-04 | 2016-03-23 | 深圳维爱特科技有限公司 | Application interaction control method and device |
EP3213673A1 (en) * | 2016-03-01 | 2017-09-06 | Shanghai Xiaoyi Technology Co., Ltd. | Smart sports eyewear |
US11980507B2 (en) | 2018-05-02 | 2024-05-14 | Augmedics Ltd. | Registration of a fiducial marker for an augmented reality system |
US11980508B2 (en) | 2018-05-02 | 2024-05-14 | Augmedics Ltd. | Registration of a fiducial marker for an augmented reality system |
US11974887B2 (en) | 2018-05-02 | 2024-05-07 | Augmedics Ltd. | Registration marker for an augmented reality system |
CN109308741A (en) * | 2018-08-08 | 2019-02-05 | 长春理工大学 | A kind of natural interaction craftwork creative design system based on Meta2 |
US10939977B2 (en) | 2018-11-26 | 2021-03-09 | Augmedics Ltd. | Positioning marker |
US11766296B2 (en) | 2018-11-26 | 2023-09-26 | Augmedics Ltd. | Tracking system for image-guided surgery |
US11980429B2 (en) | 2018-11-26 | 2024-05-14 | Augmedics Ltd. | Tracking methods for image-guided surgery |
CN113453639B (en) * | 2019-02-25 | 2023-11-03 | 奥林巴斯株式会社 | Centralized control device |
CN113453639A (en) * | 2019-02-25 | 2021-09-28 | 奥林巴斯株式会社 | Centralized control device |
US11980506B2 (en) | 2019-07-29 | 2024-05-14 | Augmedics Ltd. | Fiducial marker |
US11801115B2 (en) | 2019-12-22 | 2023-10-31 | Augmedics Ltd. | Mirroring in image guided surgery |
US11389252B2 (en) | 2020-06-15 | 2022-07-19 | Augmedics Ltd. | Rotating marker for image guided surgery |
US11896445B2 (en) | 2021-07-07 | 2024-02-13 | Augmedics Ltd. | Iliac pin and adapter |
Also Published As
Publication number | Publication date |
---|---|
US9916691B2 (en) | 2018-03-13 |
US20150363979A1 (en) | 2015-12-17 |
CN103995352B (en) | 2018-11-16 |
BR112015019674A2 (en) | 2017-07-18 |
US10169925B2 (en) | 2019-01-01 |
KR101811817B1 (en) | 2018-01-25 |
EP2956814A1 (en) | 2015-12-23 |
CN109270688B (en) | 2021-06-15 |
RU2015138973A (en) | 2017-03-17 |
TW201432312A (en) | 2014-08-16 |
RU2621488C2 (en) | 2017-06-06 |
KR20150116892A (en) | 2015-10-16 |
CN109270688A (en) | 2019-01-25 |
CN103995352A (en) | 2014-08-20 |
US20180197344A1 (en) | 2018-07-12 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10169925B2 (en) | Head mounted display and control method for head mounted display | |
JP6299067B2 (en) | Head-mounted display device and method for controlling head-mounted display device | |
US11341726B2 (en) | Information processing apparatus, information processing method, and information processing system | |
TWI617279B (en) | Information processing apparatus, information processing method, and information processing system | |
JP6264855B2 (en) | Head-mounted display device and method for controlling head-mounted display device | |
CN105607255B (en) | Head-mounted display device, method of controlling the same, and computer-readable storage medium | |
JP2015019679A (en) | Information processing device, information processing method, and information processing system | |
JP6369005B2 (en) | Head-mounted display device and method for controlling head-mounted display device | |
JP6311393B2 (en) | Information processing apparatus, information processing method, and information processing system | |
JP2017016599A (en) | Display device, display device control method, and program | |
JP2017134575A (en) | Display device, control method of display device, and program | |
JP6409927B2 (en) | Head-mounted display device and method for controlling head-mounted display device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 14724164 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 14764553 Country of ref document: US |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2014724164 Country of ref document: EP |
|
ENP | Entry into the national phase |
Ref document number: 20157024204 Country of ref document: KR Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 2015138973 Country of ref document: RU Kind code of ref document: A |
|
REG | Reference to national code |
Ref country code: BR Ref legal event code: B01A Ref document number: 112015019674 Country of ref document: BR |
|
ENP | Entry into the national phase |
Ref document number: 112015019674 Country of ref document: BR Kind code of ref document: A2 Effective date: 20150814 |
|
REG | Reference to national code |
Ref country code: BR Ref legal event code: B01E Ref document number: 112015019674 Country of ref document: BR Kind code of ref document: A2 Free format text: REAPRESENTE A PAGINA 55 DO RELATORIO DESCRITIVO (PET860150302913) SEM A MARCACAO DE TEXTO. |
|
ENPW | Started to enter national phase and was withdrawn or failed for other reasons |
Ref document number: 112015019674 Country of ref document: BR Kind code of ref document: A2 Free format text: PEDIDO RETIRADO POR AUSENCIA DE CUMPRIMENTO DA EXIGENCIA PUBLICADA NA RPI NO 2549, DE 12/11/2019. |