JP2016130985A - Head-mounted type display device, method of controlling head-mounted type display device, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a head-mounted type display device that can attract attention of a user to a real object existing in the real world, while ensuring visibility in the real world.SOLUTION: A head-mounted type display device allowing a user to visually recognize a virtual image comprises: an image display part that causes the user to visually recognize the virtual image; an image acquisition part that acquires an outer scenery image in the direction of visual field of the user wearing the head-mounted type display device; and a guide part that performs guide by sound about an object of attention that is at least one real object existing in the real world and included in the acquired outer scenery image, the real object attracting attention of the user.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a head-mounted display device.

  2. Description of the Related Art A head-mounted display device that is used by being mounted on a user's head and that forms a virtual image in the user's visual field region is known. This head-mounted display device is also called a head mounted display (HMD). In recent years, in such an HMD, an augmented reality (AR) technology has been used in order to attract the user's attention to a real object that exists in the real world.

  In order to implement AR in the HMD, the HMD captures an outside scene with, for example, a camera, recognizes an image obtained by the imaging, and generates or acquires a virtual object. In a transmissive HMD in which the user's field of vision is not blocked when the HMD is worn, the user is allowed to visually recognize only a virtual image including a virtual object. The user can experience AR by looking at both real objects in the real world and virtual objects represented by virtual images. Patent Document 1 discloses an image of an arrow that represents a direction in which a user should move next in accordance with the position of a work target that is a real object that exists in the real world and the current position of the user in the HMD. A technique for displaying is described.

JP 2014-71756 A JP 2002-159019 A

  A virtual image of the HMD is formed in the visual field region of the user. For this reason, in the technique described in Patent Document 1, the larger the amount of information displayed as a virtual image (image), the narrower the user's field of view, in other words, the real world visibility decreases. There was a problem. As described above, there is room for improvement in ensuring the visibility of the real world in the technology using AR in order to attract the user's attention to a real object that exists in the real world. In the technique described in Patent Document 2, no consideration is given to prompting the user to pay attention to a real object that exists in the real world.

  For this reason, a head-mounted display device capable of prompting a user's attention to a real object that exists in the real world while ensuring visibility in the real world has been desired.

  SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.

(1) According to one aspect of the present invention, a head-mounted display device in which a user can visually recognize a virtual image is provided. The head-mounted display device includes: an image display unit that allows the user to visually recognize the virtual image; and an image acquisition unit that acquires an outside scene image in the user's visual field direction when the head-mounted display device is mounted. Guidance that uses sound to guide a target object that is real in the real world and that is at least one real object included in the acquired outside scene image, and that is a real object that prompts the user to pay attention A section.
According to the head-mounted display device of this aspect, the guide unit is an actual object that exists in the real world and is at least one real object included in the acquired outside scene image, and that prompts the user to pay attention. Guidance for the object of interest, which is an object, can be performed by sound. Compared with guidance using a virtual image formed in the user's visual field area, the guidance using sound reduces the risk of narrowing the user's visual field, in other words, lowering the visibility of the real world. be able to. As a result, it is possible to provide a head-mounted display device that can prompt the user to pay attention to a real object that actually exists in the real world while ensuring visibility in the real world.

(2) The head-mounted display device according to the above aspect further includes: a line-of-sight detection unit that detects a direction of the user's line of sight when the head-mounted display device is mounted; If it is estimated that the object of interest is not in the detected direction of the line of sight, the guidance is performed by the sound; if the object of interest is estimated to be in the direction of the detected line of sight, the sound In addition, the guidance may be performed by causing the image display unit to form the virtual image including a virtual object to be additionally displayed on the object of interest.
According to the head-mounted display device of this aspect, the guidance unit performs guidance using sound when it is estimated that the object of interest is not in the direction of the user's line of sight. For this reason, the guide unit can suppress the formation of a virtual image in the visual field region of the user for guiding an object of interest that is not in the direction of the line of sight, and can ensure visibility in the real world. . On the other hand, when it is estimated that the target object is in the direction of the user's line of sight, the guide unit forms a virtual image including a virtual object for additionally displaying the target object in addition to the sound. I will guide you. For this reason, the guide unit can perform guidance for the object of interest in the direction of the line of sight using both sound and virtual images. That is, the user can enjoy guidance through visual and auditory senses for the object of interest in the direction of his / her line of sight. As a result, in the head-mounted display device that can attract the user's attention to the real object that exists in the real world while ensuring the visibility in the real world, the convenience of the user is further improved. Can be improved.

(3) In the head-mounted display device of the above aspect; the guidance; guidance for alerting the user to the attention object; guidance of information regarding the attention object; guidance regarding the position of the attention object And at least one of: guidance of the user's work contents for the object of interest based on procedure information prepared in advance.
According to the head-mounted display device of this form, guidance by the guide unit can be realized in various modes. For example, when performing guidance for alerting the user to the attention object, the guidance unit can perform guidance for avoiding danger to the user. When guiding information related to the object of interest, the guiding unit can perform guidance for providing information to the user. When performing guidance regarding the position of the object of interest, the guidance unit can provide guidance for assisting the operation of the user. When guiding the user's work content for the object of interest, the guide unit can provide guidance for work support to the user.

(4) In the head-mounted display device according to the above aspect; the image acquisition unit repeatedly acquires the outside scene image; the guide unit further recognizes each of the acquired plurality of outside scene images; A real object included in a plurality of outside scene images and having a large change with time may be used as the object of interest.
According to the head-mounted display device of this form, the guide unit can automatically identify the object of interest from the acquired outside scene image. In addition, since the guidance unit sets a real object having a large change over time in a plurality of repeatedly acquired outside scene images as a target object, for example, a real object having a large movement amount or a high movement speed is selected as the target object. It can be. The attention object identified in this way is particularly suitable for guidance for avoiding danger.

(5) In the head-mounted display device according to the above aspect; the guide unit further recognizes the acquired outside scene image; an actual object included in the outside scene image and has a predetermined feature An object may be the attention object.
According to the head-mounted display device of this form, the guide unit can automatically identify the object of interest from the acquired outside scene image. In addition, in order to make a real object having a predetermined characteristic an attention object in an outside scene image, for example, by preparing a “predetermined characteristic” in advance, various kinds of real objects are set as attention objects. Can do. The target object identified in this way is suitable for all types of guidance such as danger avoidance, information provision, motion assistance, and work support.

(6) The head-mounted display device according to the above aspect further includes: a line-of-sight detection unit that detects a direction of the user's line of sight when the head-mounted display device is mounted; The outside scene image is repeatedly acquired; the line-of-sight detection unit repeatedly detects the direction of the line of sight; the guide unit is further: a plurality of the acquired outside scene images and the direction of the line of sight detected multiple times And a real object that is included in the plurality of outside scene images and that is estimated to have a large proportion at the end of the line of sight may be used as the attention object.
According to the head-mounted display device of this form, the guide unit can automatically identify the object of interest from the acquired outside scene image and the detected direction of the line of sight. In addition, since the guide unit uses a real object that is estimated to have a large proportion of the point of the line of sight in a plurality of repeatedly acquired outside scene images and the direction of the line of sight as an attention object, for example, An object can be a target object. The object of interest identified in this way is particularly suitable for types of guidance such as information provision, movement assistance, and work assistance.

(7) The head-mounted display device of the above aspect further includes: an input information acquisition unit that acquires an operation input from the user; and the guide unit is further acquired with the acquired outside scene image. The operation input may be collated; a real object included in the outside scene image and designated by the user may be the attention object.
According to this form of the head-mounted display device, the guide unit can automatically identify the object of interest from the acquired outside scene image and the acquired operation input. In addition, since the guidance unit uses the real object specified by the user as the target object, the target object can be specified reflecting the intention of the user. The target object identified in this way is suitable for all types of guidance such as danger avoidance, information provision, motion assistance, and work support.

(8) In the head mounted display device according to the above aspect; the image acquisition unit repeatedly acquires the outside scene image; the guide unit; the time-lapse of the object of interest in the plurality of acquired outside scene images The guidance may be repeatedly performed according to the change.
According to this form of the head-mounted display device, the guidance unit repeatedly performs guidance according to the temporal change of the object of interest in the plurality of repeatedly acquired outside scene images. As a result, the guide unit can perform guidance corresponding to the latest state of the target object at any time. In other words, the guide unit can perform guidance while tracking the target object.

  A plurality of constituent elements of each embodiment of the present invention described above are not essential, and some or all of the effects described in the present specification are to be solved to solve part or all of the above-described problems. In order to achieve the above, it is possible to appropriately change, delete, replace with a new component, and partially delete the limited contents of some of the plurality of components. In order to solve part or all of the above-described problems or to achieve part or all of the effects described in this specification, technical features included in one embodiment of the present invention described above. A part or all of the technical features included in the other aspects of the present invention described above may be combined to form an independent form of the present invention.

  For example, one embodiment of the present invention can be realized as an apparatus including some or all of the three elements of the image display unit, the image acquisition unit, and the guide unit. That is, this apparatus may or may not have an image display unit. In addition, this apparatus may or may not have an image acquisition unit. Moreover, this apparatus may have a guide part and does not need to have it. Such a device can be realized, for example, as a head-mounted display device, but can also be realized as a device other than the head-mounted display device. Any or all of the technical features of each form of the head-mounted display device described above can be applied to this device. For example, an apparatus according to an embodiment of the present invention has an object of urging a user to pay attention to a real object that exists in the real world while ensuring visibility in the real world. However, there are other demands for this device such as downsizing of the device, improvement in convenience, cost reduction during device manufacture, resource saving, and ease of manufacture.

  The present invention can be realized in various modes. For example, a head-mounted display device, a head-mounted display device control method, a system including a head-mounted display device, and these methods The present invention can be realized in the form of an apparatus, a computer program for realizing the functions of the system, a server apparatus for distributing the computer program, a storage medium storing the computer program, and the like.

It is explanatory drawing which shows schematic structure of the head mounted display apparatus in one Embodiment of this invention. 2 is a block diagram functionally showing the configuration of the HMD 100. FIG. It is explanatory drawing which shows an example of the virtual image visually recognized by the user. It is a flowchart which shows the procedure of a guidance process. It is a figure which shows an example of the information 122 for guidance. It is a figure explaining the 1st specific example of a guidance process. It is a figure explaining the 1st specific example of a guidance process. It is a figure explaining the 1st specific example of a guidance process. It is a figure explaining the 2nd specific example of guidance processing. It is a figure explaining the 2nd specific example of guidance processing. It is a figure explaining the 2nd specific example of guidance processing. It is explanatory drawing which shows the structure of the external appearance of HMD in a modification.

A. Embodiment:
A-1. Configuration of head mounted display device:
FIG. 1 is an explanatory diagram showing a schematic configuration of a head-mounted display device according to an embodiment of the present invention. The head-mounted display device 100 of the present embodiment is a display device that is mounted on the head, and is also called a head mounted display (HMD). The HMD 100 is an optically transmissive head mounted display that allows a user to visually recognize a virtual image and at the same time also visually recognize an outside scene.

  The HMD 100 according to the present embodiment performs guidance for an object of interest by “voice only” or “a combination of voice and augmented reality (AR) processing” by performing a guidance process described later. Specifically, the HMD 100 detects the direction of the user's line of sight, and when it is estimated that the object of interest is not in the direction of the line of sight, the HMD 100 performs guidance only by voice and estimates that the object of attention is in the direction of the line of sight. If so, guidance is provided by a combination of voice and augmented reality processing. Details of the guidance process will be described later. In this embodiment, “voice” functions as “sound”.

  The “attention object” means at least one real object identified from real objects that are objects that exist in the real world, and means a real object that prompts the user to pay attention. The real object means an arbitrary person, an arbitrary animal or plant, an arbitrary object (including an artificial object, a natural object, or the like). A real object is an “real object of interest” that is an object that the user is interested in (for example, the line of sight), and is outside the user's interest (for example, the line of sight that is in the field of view). It can include both “real background objects” that are objects. That is, in the guidance process, the entire real object including both the real interest object and the real background object can be processed. A method for specifying the object of interest will be described later.

“Augmented reality processing” is processing for adding information to a real object by the HMD 100. In the augmented reality processing, information displayed in addition to the real object is referred to as a “virtual object”. “Additional” means to include all of the following. The real object and the virtual object additionally displayed on the real object may or may not be related.
Add information to real objects using virtual objects.
Emphasize real objects using virtual objects.
-Delete and attenuate information (color, shape, etc.) held by a real object using a virtual object.
Any combination of two or more of addition, emphasis, deletion, and attenuation described above.

  The HMD 100 includes an image display unit 20 that allows a user to visually recognize a virtual image while being mounted on the user's head, and a control unit (controller) 10 that controls the image display unit 20. In the following description, a virtual image visually recognized by the user with the HMD 100 is also referred to as a “display image” for convenience. In addition, the HMD 100 emitting the image light generated based on the image data is also referred to as “displaying an image”.

A-1-1. Configuration of image display:
FIG. 2 is a block diagram functionally showing the configuration of the HMD 100. The image display unit 20 is a wearing body that is worn on the user's head, and has a glasses shape in the present embodiment (FIG. 1). The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a camera 61. And a line-of-sight detection unit 62, a microphone 63, and a nine-axis sensor 66. Hereinafter, the positional relationship and function of each unit of the image display unit 20 in a state where the user wears the image display unit 20 will be described.

  As shown in FIG. 1, the right optical image display unit 26 and the left optical image display unit 28 are disposed so as to be positioned in front of the user's right eye and in front of the left eye, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected at a position corresponding to the eyebrow of the user. As shown in FIG. 2, the right optical image display unit 26 includes a right light guide plate 261 and a light control plate (not shown). The right light guide plate 261 is formed of a light transmissive resin material or the like, and guides the image light output from the right display driving unit 22 to the right eye RE of the user while reflecting the image light along a predetermined optical path. The light control plate is a thin plate-like optical element, and is disposed so as to cover the front side of the image display unit 20 (the side opposite to the user's eye side). The light control plate protects the light guide plate 261 and suppresses damage, dirt adhesion, and the like. Further, by adjusting the light transmittance of the light control plate, it is possible to adjust the external light quantity entering the user's eyes and adjust the ease of visual recognition of the virtual image. The dimming plate can be omitted.

  The left optical image display unit 28 includes a left light guide plate 262 and a light control plate (not shown). These details are the same as those of the right optical image display unit 26. The right optical image display unit 26 and the left optical image display unit 28 are collectively referred to simply as “optical image display unit”. The optical image display unit can use any method as long as it forms a virtual image in front of the user's eyes using image light. For example, the optical image display unit may be realized by using a diffraction grating, or a transflective film may be used. It may be realized using.

  As shown in FIG. 1, the right holding unit 21 extends from the other end ER of the right optical image display unit 26 to a position corresponding to the user's temporal region. The left holding unit 23 is provided to extend from the other end EL of the left optical image display unit 28 to a position corresponding to the user's temporal region. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses. The right holding unit 21 and the left holding unit 23 are collectively referred to simply as a “holding unit”.

  As shown in FIG. 1, the right display driving unit 22 is disposed inside the right holding unit 21 (the side facing the user's head). The left display driving unit 24 is disposed inside the left holding unit 23. As shown in FIG. 2, the right display driving unit 22 functions as a receiving unit (Rx) 53, a right backlight (BL) control unit 201 and a right backlight (BL) 221 that function as a light source, and a display element. A right LCD (Liquid Crystal Display) control unit 211, a right LCD 241, and a right projection optical system 251 are provided. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are collectively referred to as “image light generation unit”.

  The receiving unit 53 functions as a receiver for serial transmission between the control unit 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED (Light Emitting Diode) or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data Data for the right eye input via the reception unit 53. . The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix. The right projection optical system 251 is a collimating lens that converts the image light emitted from the right LCD 241 to light beams in a parallel state.

  The left display driving unit 24 includes a receiving unit (Rx) 54, a left backlight (BL) control unit 202 and a left backlight (BL) 222 that function as a light source, a left LCD control unit 212 and a left that function as a display element. An LCD 242 and a left projection optical system 252 are provided. These details are the same as those of the right display drive unit 22. The right display drive unit 22 and the left display drive unit 24 are collectively referred to simply as “display drive unit”.

  As shown in FIG. 1, the camera 61 is a stereo camera disposed at a position corresponding to the upper left and right corners of the user. The left and right cameras 61 each capture an outside scene (external scenery) in the front side direction of the image display unit 20, in other words, the user's viewing direction when the HMD 100 is worn, and two outside scene images corresponding to the left and right sides. To get. The camera 61 is a so-called visible light camera, and the outside scene image acquired by the camera 61 is an image representing the shape of the object from the visible light emitted from the object. The camera 61 in the present embodiment is a stereo camera, but may be a monocular camera. The camera 61 functions as an “image acquisition unit”.

  As shown in FIG. 1, the line-of-sight detection unit 62 is disposed at a position corresponding to the lower part of the left and right corners of the user. The left and right line-of-sight detection units 62 each include an infrared light emitting unit and an infrared light receiving unit (not shown). The right gaze detection unit 62 receives the infrared light emitted from the infrared light emitting unit and reflected by the right eye of the user. The CPU 140 of the control unit 10 acquires the movement of the line of sight of the user's right eye based on the intensity of infrared light received by the right line of sight detection unit 62. Similarly, the left line-of-sight detection unit 62 receives the infrared light emitted from the infrared light emitting unit and reflected by the left eye of the user. The CPU 140 detects the line of sight of the user's left eye based on the intensity of infrared light received by the left line-of-sight detection unit 62. The reflectance of infrared rays is different depending on whether the infrared ray hits the iris (black eye), the infrared ray hits the eyelid, or the infrared ray hits the white eye. Therefore, the CPU 140 can acquire the movement of the user's line of sight based on the intensity of infrared rays received by the line-of-sight detection unit 62. In this case, the line-of-sight detection unit 62 and the CPU 140 function as a “line-of-sight detection unit” that detects the direction of the user's line of sight. In the present embodiment, the line-of-sight detection unit 62 is provided on each of the left and right sides, but may be either the left or right side.

  As shown in FIG. 1, the microphone 63 is a microphone disposed at a position corresponding to the upper side of the user's nose. The microphone 62 acquires external sound. “Voice” means not only a human voice but also a broad sense of sound including mechanical sounds.

  As shown in FIG. 1, the 9-axis sensor 66 is disposed at a position corresponding to the temple on the right side of the user. The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). Since the 9-axis sensor 66 is provided in the image display unit 20, when the image display unit 20 is mounted on the user's head, the movement of detecting the movement of the user's head on the head mounted display 100. Functions as a detection unit. Here, the movement of the head includes changes in the speed, acceleration, angular velocity, direction, and direction of the head.

  As shown in FIG. 1, the image display unit 20 includes a connection unit 40 for connecting the image display unit 20 and the control unit 10. The connection unit 40 includes a main body cord 48 connected to the control unit 10, a right cord 42 and a left cord 44 branched from the main body cord 48, and a connecting member 46 provided at the branch point. The connecting member 46 is provided with a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30. The image display unit 20 and the control unit 10 transmit various signals via the connection unit 40. For example, a metal cable or an optical fiber can be employed for each cord of the connection unit 40.

A-1-2. Configuration of control unit:
The control unit 10 is a device for controlling the HMD 100. As shown in FIG. 1, the control unit 10 includes a determination key 11, a lighting unit 12, a display switching key 13, a track pad 14, a luminance switching key 15, a direction key 16, a menu key 17, and a power source. And a switch 18. The determination key 11 detects a pressing operation and outputs a signal for determining the content operated in the control unit 10. The lighting unit 12 is realized by, for example, an LED, and notifies the operation state of the HMD 100 (for example, ON / OFF of the power source) by the light emission state. The display switching key 13 detects a pressing operation, and outputs a signal for switching the display mode of the content moving image between 3D and 2D, for example.

  The track pad 14 detects the operation of the user's finger on the operation surface of the track pad 14 and outputs a signal corresponding to the detected content. As the track pad 14, various methods such as an electrostatic method, a pressure detection method, and an optical method can be adopted. The luminance switching key 15 detects a pressing operation and outputs a signal for increasing or decreasing the luminance of the image display unit 20. The direction key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The power switch 18 switches the power-on state of the HMD 100 by detecting a slide operation of the switch.

  2, the control unit 10 includes an input information acquisition unit 110, a storage unit 120, a power supply 130, a wireless communication unit 132, a GPS module 134, a CPU 140, an interface 180, and a transmission unit (Tx ) 51 and 52, and the respective parts are connected to each other by a bus (not shown).

  The input information acquisition unit 110 acquires signals corresponding to operation inputs to the enter key 11, the display switching key 13, the track pad 14, the luminance switching key 15, the direction key 16, the menu key 17, and the power switch 18. The input information acquisition unit 110 can acquire operation inputs using various methods other than those described above. For example, the user's gesture may be detected using the camera 61, and an operation input by a command associated with the gesture may be acquired. When detecting a gesture, a user's fingertip, a ring attached to the user's hand, a medical instrument held by the user's hand, or the like can be used as a mark for motion detection. For example, an operation input by a command associated with the user's line of sight detected by the line-of-sight detection unit 62 or the movement of the eyes may be acquired. This command may be set to be addable by the user. For example, an operation input by recognizing voice acquired by the microphone 63 may be acquired. For example, an operation input by a foot switch (not shown) (a switch operated by a user's foot) may be acquired. If the operation input by these methods can be acquired, the input information acquisition unit 110 can acquire the operation input from the user even in the work where it is difficult for the user to release his / her hand.

  The storage unit 120 includes a ROM, a RAM, a DRAM, a hard disk, and the like. The storage unit 120 stores various computer programs including an operating system (OS). The storage unit 120 stores guidance information 122 and a guidance mode 124. The guidance information 122 is information used in the guidance process. Details will be described later.

  The guidance mode 124 is information for designating a guidance method in guidance processing. Guidance methods in the guidance process include a “with tracking mode” and a “without tracking mode”. In the no-tracking mode, the guide unit 144 performs guidance for the object of interest only once. In the tracking mode, the guide unit 144 repeatedly performs guidance on the target object as the target object changes with time. In the guidance mode 124, information representing either one of the no tracking mode or the tracking mode is stored. The user can change the content of the guidance mode 124 by an operation input by the input information acquisition unit 110.

  The power supply 130 supplies power to each part of the HMD 100. As the power supply 130, for example, a secondary battery can be used. The wireless communication unit 132 performs wireless communication with an external device in accordance with a predetermined wireless communication standard. The predetermined wireless communication standard is, for example, infrared, short-range wireless communication exemplified by Bluetooth (registered trademark), wireless LAN exemplified by IEEE 802.11, or the like. The GPS module 134 detects the current position of the user of the HMD 100 by receiving a signal from a GPS satellite, and generates current position information representing the current position information of the user. The current position information can be realized by coordinates representing latitude and longitude, for example.

  The CPU 140 reads out and executes a computer program stored in the storage unit 120, thereby executing an augmented reality processing unit 142, a guide unit 144, an operating system (OS) 150, an image processing unit 160, an audio processing unit 170, and a display. It functions as the control unit 190.

  The augmented reality processing unit 142 executes augmented reality processing when performing guidance by “a combination of voice and augmented reality processing” in the guidance processing. In the augmented reality processing, the augmented reality processing unit 142 adds a virtual object to the object of interest (real object) and displays it. Details will be described later. The guide unit 144 executes guidance processing to implement guidance for the object of interest by “voice only” or “combination of voice and augmented reality processing”. Details will be described later.

  The image processing unit 160 performs signal processing for image display. Specifically, when content (video) is input via the interface 180 or the wireless communication unit 132, the image processing unit 160 generates image data Data based on the content. When the image processing unit 160 receives image data from another functional unit of the HMD 100, the image processing unit 160 sets the received data as image data Data. The image processing unit 160 may execute image processing such as resolution conversion processing, various tone correction processing such as adjustment of luminance and saturation, and keystone correction processing on the image data Data. The image processing unit 160 transmits the image data Data, the clock signal PCLK, the vertical synchronization signal VSync, and the horizontal synchronization signal HSync to the image display unit 20 via the transmission units 51 and 52. The image data Data transmitted via the transmission unit 51 is also referred to as “right eye image data Data1”, and the image data Data transmitted via the transmission unit 52 is also referred to as “left eye image data Data2”.

  The display control unit 190 generates control signals for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 190 turns on and off the left and right LCDs 241 and 242 by the left and right LCD control units 211 and 212 and the left and right backlights by the left and right backlight control units 201 and 202 according to the control signal. The generation and emission of image light by the right display drive unit 22 and the left display drive unit 24 are controlled by individually controlling the driving ON / OFF of the 221 and 222, respectively. The display control unit 190 transmits these control signals to the image display unit 20 via the transmission units 51 and 52.

  The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and supplies the acquired audio signal to a speaker (not shown) of the right earphone 32 and a speaker (not shown) of the left earphone 34.

  The interface 180 communicates with the external device OA in accordance with a predetermined wired communication standard. The predetermined wired communication standards include, for example, MicroUSB (Universal Serial Bus), USB, HDMI (High Definition Multimedia Interface, HDMI is a registered trademark), DVI (Digital Visual Interface), VGA (Video Graphics Array), Composite, RS- The wired LAN is exemplified by 232C (Recommended Standard 232) and IEEE802.3. As the external device OA, for example, a personal computer PC, a mobile phone terminal, a game terminal, or the like can be used.

  FIG. 3 is an explanatory diagram illustrating an example of a virtual image visually recognized by the user. FIG. 3A illustrates the user's visual field VR when the augmented reality processing is not executed. As described above, the image light guided to both eyes of the user of the HMD 100 forms an image on the retina of the user, so that the user visually recognizes the virtual image VI. In the example of FIG. 3A, the virtual image VI is a standby screen of the OS 150 of the HMD 100. The user views the outside scene SC through the right optical image display unit 26 and the left optical image display unit 28. As described above, the user of the HMD 100 of the present embodiment can see the virtual image VI and the outside scene SC behind the virtual image VI for the portion of the visual field VR where the virtual image VI is displayed. In addition, the portion of the visual field VR where the virtual image VI is not displayed can be seen through the optical scene display portion and directly through the outside scene SC.

  FIG. 3B illustrates the visual field VR of the user when the augmented reality process is executed. By executing the augmented reality process, the user visually recognizes the virtual image VI including the virtual object VO. The virtual object VO is an image of an apple arranged so as to be superimposed on the foot of the mountain of the outside scene SC. In this way, the user can experience augmented reality by seeing both the virtual object VO included in the virtual image VI and the real object in the outside scene SC that can be seen behind the virtual image VI. it can.

A-2. Guidance process:
FIG. 4 is a flowchart showing the procedure of the guidance process. The guidance process is a process for performing guidance for the object of interest by “voice only” or “a combination of voice and augmented reality processing”. The start trigger of the guidance process can be arbitrarily determined. For example, a power ON of the HMD 100 may be used as a start trigger, and a process start request from the OS 150 of the HMD 100 or a specific application may be used as a start trigger.

  In step S102, the guide unit 144 causes the camera 61 to acquire an outside scene image.

  In step S104, the guide unit 144 identifies at least one real object from among the real objects included in the outside scene image acquired in step S102, as a target object that should attract attention from the user. Specifically, the guide unit 144 specifies the object of attention using any one of the following methods a1 to a4. Methods a1 to a4 may be used alone or in combination.

(A1) A method of using the change over time of a real object:
The guide unit 144 recognizes the outside scene image acquired in step S102, and models various objects (for example, glasses, bottles, human bodies, cars, trees, etc.) stored in the storage unit 120 or the like in advance. Then, all real objects included in the outside scene image are recognized by performing comparison by pattern matching or statistical identification method (procedure a11). In step a11, instead of recognizing all real objects, only main real objects may be recognized. In step a11, the outside scene image may be divided into a plurality of areas for image recognition. The plurality of areas may be, for example, four areas obtained by dividing the outside scene image into two equal parts in the horizontal direction and the vertical direction, or nine areas obtained by dividing the outside scene image into three parts in the horizontal direction and the vertical direction, respectively. There may be.
The guiding unit 144 repeats the acquisition of the outside scene image (step S102) and the recognition of the real object with respect to the acquired latest outside scene image (procedure a11) a plurality of times (procedure a12).
From the result of step a12, the guide unit 144 identifies a real object having a large change over time in a plurality of repeatedly acquired outside scene images as a target object (step a13). The large change over time includes both a case where the amount of movement over time is large and a case where the movement speed over time is high. The number of objects of interest specified here may be one or more.

  According to the method a1, the guide unit 144 can automatically identify the object of interest from the acquired outside scene image. In addition, in the plurality of repeatedly acquired outside scene images, the guide unit 144 sets a real object having a large change over time as a target object. For example, the guide unit 144 pays attention to a real object having a large movement amount or a high movement speed. It can be an object. The attention object identified in this way is particularly suitable for guidance for avoiding danger.

(A2) Method of using features of real object:
The guide unit 144 recognizes the outside scene image acquired in step S102, and features (for example, color features, shape features, size features, etc.) stored in the storage unit 120 or the like in advance. Functions as a “predetermined feature”) and performs comparison by pattern matching or statistical identification (step a21).
The guide unit 144 identifies a real object that matches the comparison result of the procedure a21 in the outside scene image as a target object (procedure a22). The number of objects of interest specified here may be one or more.

  According to the method a2, the guide unit 144 can automatically identify the object of interest from the acquired outside scene image. In addition, since the guide unit 144 uses a real object having a predetermined feature in the outside scene image as a target object, for example, by preparing the “predetermined feature” in advance as described above, various real objects can be displayed. It can be an object of interest. The target object identified in this way is suitable for all types of guidance such as danger avoidance, information provision, motion assistance, and work support.

(A3) Method of using the user's line of sight:
The guide unit 144 recognizes the outside scene image acquired in step S102 and recognizes all (or main) real objects included in the outside scene image (procedure a31). Details are the same as in step a11.
The guide unit 144 causes the line-of-sight detection unit 62 to detect the direction of the user's line of sight. The guide unit 144 collates the real object recognized in the procedure a31 with the detected gaze direction, and extracts a real object estimated to be ahead of the gaze direction (procedure a32).
The guide unit 144 acquires an outside scene image (step S102), recognizes a real object with respect to the latest acquired outside scene image (procedure a31), and extracts a real object estimated to be ahead of the direction of the line of sight ( Procedure a32) is repeated a plurality of times (procedure a33).
The guide unit 144 identifies, as a target object, a real object that is estimated to have a higher proportion of the direction of the line of sight in the plurality of repeatedly acquired outside scene images and the direction of the line of sight from the result of the procedure a33. (Procedure a34). The number of objects of interest specified here may be one or more.

  According to the method a3, the guide unit 144 can automatically identify the object of interest from the acquired outside scene image and the detected line-of-sight direction. In addition, the guide unit 144 uses a real object that is estimated to have a large proportion of the point of view in the plurality of repeatedly acquired outside scene images and the direction of the line of sight as an attention object. A real object can be a target object. The object of interest identified in this way is particularly suitable for types of guidance such as information provision, movement assistance, and work assistance.

(A4) Method of using user designation:
The guide unit 144 recognizes the outside scene image acquired in step S102 and recognizes all (or main) real objects included in the outside scene image (procedure a41). Details are the same as in step a11.
The guide unit 144 acquires an operation input from the user acquired by the input information acquisition unit 110. The guiding unit 144 collates the real object recognized in the procedure a41 with the acquired operation input, and specifies the real object designated by the user as the attention object (procedure a42). The number of objects of interest specified here may be one or more. As described above, the operation input from the user via the input information acquisition unit 110 may be, for example, an input using a gesture, an input using a movement of eyes, or a voice. The input may be an input using a foot switch.

  According to the method a4, the guide unit 144 can automatically identify the object of interest from the acquired outside scene image and the acquired operation input. In addition, since the guide unit 144 uses the real object specified by the user as the attention object, the guidance object 144 can identify the attention object reflecting the intention of the user. The target object identified in this way is suitable for all types of guidance such as danger avoidance, information provision, motion assistance, and work support.

  In step S106 in FIG. 4, the guide unit 144 acquires content to be guided to the user (hereinafter also referred to as “guidance content”) for the object of interest specified in step S104. Specifically, the guidance unit 144 acquires the guidance content by searching the guidance information 122 using the name, feature, model, and the like of the object of interest.

  FIG. 5 is a diagram illustrating an example of the guidance information 122. The guidance information 122 includes object 1, object 2, and guidance content.

  In “Object 1” and “Object 2”, information for collation using the name, feature, model, etc. of the object of interest is stored. In the example of FIG. 5, the names of the objects of interest are stored in the objects 1 and 2. Therefore, when the guidance information 122 illustrated in FIG. 5 is used, a search using the name of the object of interest is performed in step S106 (FIG. 4) of the guidance process. When a search using the feature of the target object is performed in step S106 of the guidance process, information representing the feature of the target object is stored in object 1 and object 2. When a search using the model of the object of interest is performed in step S106 of the guidance process, the object 1 and the object 2 store information representing the model of the object of attention.

  In “Guidance contents”, information representing contents to be guided to the user is stored. In the example of FIG. 5, the guidance content stores one or two character strings representing the content to be guided to the user. In FIG. 5, a character string without “(*)” notation at the end of the character string represents guidance information used in both the guidance in the mode without tracking and the guidance in the mode with tracking. What has the notation “(*)” at the end of the character string represents guidance information used only for guidance in the tracking mode. The guidance content may be stored as a character string as illustrated, may be stored as a graphic or an image, or may be stored as program data executed in the guide unit 144.

The “guidance content” may include the following information b1 to b4.
(B1) Guidance for alerting the user to the object of interest: According to the information b1, the guidance unit 144 can implement guidance for avoiding danger to the user.
(B2) Information guidance on the object of interest: According to the information b2, the guidance unit 144 can carry out guidance for providing information to the user.
(B3) Guidance on the position of the object of interest: According to the information b3, the guiding unit 144 can perform guidance for assisting the operation of the user.
(B4) User's work content guidance for the object of interest: According to the information b4, the guide unit 144 can implement guidance for work support for the user.

  Specifically, in the example of the entry E1 in FIG. 5, when the object 1 is “bottle” and the object 2 is “glass”, the “bottle mouth glass” is set in both the non-tracking mode and the tracking mode. “Guide to the center of the screen (corresponding to information b3)” and “Guiding the amount of liquid in the glass (corresponding to information b2)” in the tracking mode. Further, in the example of entry E5, when the object 1 is “car”, “notice guidance to the car (corresponding to information b1)” in both the non-tracking mode and the tracking mode and tracking In the mode, “Guide distance with car (corresponding to information b2)” is stored. Thus, the object 2 may not store data.

  In step S108 in FIG. 4, the guide unit 144 causes the line-of-sight detection unit 62 to detect the direction of the user's line of sight.

  In step S110, the guide unit 144 collates the position of the target object specified in step S104 in the outside scene image with the detected line-of-sight direction, and estimates whether the target object is in the line-of-sight direction. .

  When it is estimated that the object of interest is not in the line of sight (step S110: no line of sight), in step S112, the guide unit 144 performs guidance based on the guidance content acquired in step S106 only by voice. Specifically, the guidance unit 144 generates or acquires voice data for guidance, and transmits the voice data to the voice processing unit 170. The sound processing unit 170 outputs sound based on the received sound data from the speaker in the right earphone 32 and the speaker in the left earphone 34.

  When it is estimated that the object of interest is in the direction of the line of sight (step S110: in the direction of the line of sight), in step S114, the guide unit 144 performs guidance based on the guidance content acquired in step S106 with voice and augmented reality processing. It carries out by the combination. A specific method of voice guidance is the same as in step S112. A specific method of guidance by augmented reality processing is as shown in the following procedures c1 to c5.

(C1) The guide unit 144 performs augmented reality processing on information (name, feature, model, etc. of the target object) for specifying the target object specified in step S104 and the guide content acquired in step S106. To the unit 142.

(C2) The augmented reality processing unit 142 causes the camera 61 to acquire an outside scene image. The augmented reality processing unit 142 acquires the position of the target object in the acquired outside scene image using the information for specifying the target object acquired in step c1. The “position” includes the distance from the user and the horizontal and vertical positions in the user's field of view. At this time, the augmented reality processing unit 142 may calculate the position of the object of interest using two or more outside scene images acquired by the camera 61 that is a stereo camera. In addition, the augmented reality processing unit 142 uses one or more outside scene images acquired by the camera 61 and various sensors (not shown) (for example, a depth sensor, a distance measuring sensor, etc.) in combination to determine the position of the target object. May be calculated.

(C3) The augmented reality processing unit 142 displays a virtual object to be additionally displayed on the object of interest according to the content of the guidance content acquired in step c1 that does not have “(*)” at the end. Get or generate images, characters, graphic symbols, etc. to represent. Examples of images, characters, graphic symbols and the like representing virtual objects are, for example, arrows, lines, scales, scales, pointers, shortcuts, menus, radio buttons, selection buttons, soft keyboards, and the like. The augmented reality processing unit 142 may acquire a virtual object stored in advance in the storage unit 120 or may acquire a virtual object stored in advance in another device connected to the HMD 100 via a network.

(C4) The augmented reality processing unit 142 arranges the virtual object acquired or generated in step c3 in accordance with the position of the target object acquired in step c2, and generates additional image data in which black is arranged in other portions. To do. At the time of alignment, the augmented reality processing unit 142 may use a characteristic part (such as an edge) of the target object, or may use a marker such as a marker attached to the target object. At the time of alignment, the augmented reality processing unit 142 may use image recognition using a model (or image) of the target object stored in the storage unit 120 in advance. At the time of alignment, the augmented reality processing unit 142 may be based on an instruction from the user. The augmented reality processing unit 142 may perform image processing such as enlargement, reduction, rotation, and color conversion on the virtual object when arranging the virtual object.

(C5) The augmented reality processing unit 142 transmits an image based on the generated additional image data to the image processing unit 160 and causes the optical image display units 26 and 28 of the image display unit 20 to display the images.

  After step S112 or step S114 in FIG. 4 is completed, the guide unit 144 acquires the content stored in the guide mode 124. When the content of the guidance mode 124 is “no tracking mode”, the guidance unit 144 ends the guidance process. When the content of the guidance mode 124 is “tracking mode”, the guide unit 144 waits for an arbitrary predetermined time, and then repeats the process of step S102 and the processes of steps S108 to S114. In the second and subsequent steps S108 to S114, the guide unit 144 includes “(*) at the end of the attention object identified in the first step S104 and the guidance content acquired in the first step S106. ”And the content of what has the notation. That is, step S104 and step S106 are not executed after the second time.

A-3. Specific examples of guidance processing:
A-3-1. First specific example:
6-8 is a figure explaining the 1st specific example of a guidance process. In the first specific example, a description will be given of guidance processing in a scene where a user is pouring liquid from a bottle to a glass. The first specific example is an example of operation assistance and information provision. In step S102 of the guidance process (FIG. 4), the guidance unit 144 acquires an outside scene image. FIG. 6 is an example of an outside scene image acquired in step S102. The outside scene image IM1 includes OB11 to OB13 which are real objects.

  In step S104 of the guidance process, the guidance unit 144 identifies an object of interest from any of the real objects OB11 to OB13 using any of the methods a1 to a4 described above. For example, it is assumed that the glass OB12 and the bottle OB11 are specified as the target object. In step S106 of the guidance process, the guidance unit 144 searches the guidance information 122 using the glass OB12 and the bottle OB11, and the guidance contents “entry the bottle mouth to the center of the glass” and “ “Guid the amount of liquid (*)”.

  In step S108 of the guidance process, the guidance unit 144 detects the direction of the user's line of sight. When both the glass OB12 and the bottle OB11 are not in the line of sight (step S110: no line of sight), in step S112, the guidance unit 144 obtains the guidance content “bottle mouth in the middle of the glass in step S106. Guidance based on “Guidance on” is performed only by voice. As a result, only the guidance voice SO11 (FIG. 7) “Please move the bottle mouth to the left” is reproduced from the speakers in the left and right earphones 32, 34, and the virtual image VI including the virtual object VO11 is not displayed.

  When at least one of the glass OB12 and the bottle OB11 is in the direction of the line of sight (step S110: in the direction of the line of sight), the guide unit 144 in step S114 displays the guidance content “bottle mouth” acquired in step S106. Guidance based on “Guided in the center of the glass” is implemented by a combination of voice and augmented reality processing. As a result, as shown in FIG. 7, the guidance sound SO11 “Please move the bottle mouth to the left” is reproduced from the speakers in the left and right earphones 32 and 34, and the center line of the glass OB12 is represented. A virtual image VI including the virtual object VO11 is displayed.

  When the content stored in the guidance mode 124 is the “no tracking mode”, the guidance unit 144 ends the process as described above. When the content stored in the guidance mode 124 is the “tracking mode”, the guidance unit 144 waits for a predetermined time. In step S102 of the second guidance process, the guidance unit 144 acquires an outside scene image.

  In step S108 of the second guidance process, the guidance unit 144 detects the direction of the user's line of sight. When both the glass OB12 and the bottle OB11 are not in the line of sight (step S110: no line of sight), the guide 144 in the second step S112 displays the guidance content “liquid in the glass” acquired in step S106. Guidance based on “Guidance of amount (*)” is performed only by voice. As a result, only the guidance voice SO12 (FIG. 8) “It will be full soon” is reproduced from the speakers in the left and right earphones 32, 34, and the virtual image VI including the virtual object VO12 is not formed.

  When at least one of the glass OB12 and the bottle OB11 is in the direction of the line of sight (step S110: in the direction of the line of sight), in the second step S114, the guide unit 144 displays the guidance content “glass” acquired in step S106. Guidance based on the guidance of the amount of liquid in (*) "is performed by a combination of voice and augmented reality processing. As a result, as shown in FIG. 8, the guidance voice SO12 “It is almost full” is reproduced from the speakers in the left and right earphones 32 and 34, and the virtual object VO11 representing the center line of the glass OB12 and the current A virtual image VI including a virtual object VO12 in which the amount of liquid is indicated by an arrow is displayed.

A-3-2. Second specific example:
FIGS. 9-11 is a figure explaining the 2nd specific example of a guidance process. In the second specific example, guidance processing in a scene where the user is driving a car will be described. The second specific example is an example of danger avoidance and information provision. In step S102 of the guidance process (FIG. 4), the guidance unit 144 acquires an outside scene image. FIG. 9 is an example of an outside scene image acquired in step S102. The outside scene image IM2 includes OB21 to OB26 that are real objects.

  In step S104 of the guidance process, the guidance unit 144 specifies the object of interest from any of the real objects OB21 to OB26 using any of the methods a1 to a4 described above. For example, it is assumed that the signboard OB21 and the car OB23 are specified as the attention object. In step S106 of the guidance process, the guidance unit 144 searches the guidance information 122 using the signboard OB21 and the car OB23, and guides the entry contents “Estimation guidance for the signboard” and “Guide the contents of the signboard”. Then, the guidance contents of entry E5 “Guidance for alerting the car” and “Guide distance to car (*)” are acquired.

  In step S108 of the guidance process, the guidance unit 144 detects the direction of the user's line of sight. When the signboard OB21 is not in the direction of the line of sight (step S110: no line of sight), in step S112, the guide unit 144 obtains the guidance contents “guidance for calling attention to the signboard” and “the contents of the signboard” acquired in step S106. Guidance based on “guidance” is performed only by voice. As a result, only the guidance voice SO21 (FIG. 10) “Please pay attention to the rear-end collision” is reproduced from the speakers in the left and right earphones 32 and 34, and the virtual image VI including the virtual object VO21 is not displayed. When the vehicle OB23 is not in the line of sight (step S110: no line of sight), in step S112, the guide unit 144 provides voice guidance for guidance based on the guidance content “guidance for alerting the vehicle” acquired in step S106. Implemented only by. As a result, only the guidance voice SO22 (FIG. 10) “Please pay attention to the car ahead” is reproduced from the speakers in the left and right earphones 32, 34, and the virtual image VI including the virtual object VO22 is not displayed.

  When the signboard OB21 is in the direction of the line of sight (step S110: in the direction of the line of sight), in step S114, the guide unit 144 obtains the guidance contents “guidance for alerting the signboard” and “the contents of the signboard” acquired in step S106. Guidance based on “guidance” is performed by a combination of voice and augmented reality processing. As a result, as shown in FIG. 10, the guidance sound SO21 “Please pay attention to the rear-end collision” is reproduced from the speakers in the left and right earphones 32 and 34, and the virtual object VO21 for emphasizing the signboard OB21 is included. A virtual image VI is displayed. When the vehicle OB23 is in the direction of the line of sight (step S110: in the direction of the line of sight), in step S114, the guide unit 144 provides guidance based on the guidance content “guidance for alerting the vehicle” acquired in step S106. And a combination of augmented reality processing. As a result, as shown in FIG. 10, the guidance sound SO22 “Please pay attention to the car ahead” is reproduced from the speakers in the left and right earphones 32, 34, and the virtual object VO22 for emphasizing the car OB23. A virtual image VI including is displayed.

  When the content stored in the guidance mode 124 is the “no tracking mode”, the guidance unit 144 ends the process as described above. When the content stored in the guidance mode 124 is the “tracking mode”, the guidance unit 144 waits for a predetermined time. In step S102 of the second guidance process, the guidance unit 144 acquires an outside scene image. Since the acquired outside scene image does not include the signboard OB, the guide unit 144 excludes the signboard OB21 from the object of interest in the second process.

  In step S108 of the second guidance process, the guidance unit 144 detects the direction of the user's line of sight. When the vehicle OB23 is not in the line of sight (step S110: no line of sight), in the second step S112, the guidance unit 144 obtains the guidance content “Guide distance from car (*)” acquired in step S106. Guidance based on is performed only by voice. As a result, only the guidance voice SO23 (FIG. 10) “5 m ahead vehicle is approaching” is reproduced from the speakers in the left and right earphones 32, 34, and the virtual image VI including the virtual object VO21 is not displayed.

  When the vehicle OB23 is in the direction of the line of sight (step S110: in the direction of the line of sight), in the second step S114, the guide unit 144 obtains the guidance content “Guide distance from the car (*)” acquired in step S106. Based on the combination of voice and augmented reality processing. As a result, as shown in FIG. 10, the guidance sound SO23 “5 m ahead vehicle is approaching” is reproduced from the speakers in the left and right earphones 32 and 34, and the virtual object VO22 for emphasizing the car OB23. And a virtual image VI including the virtual object VO23 indicating the distance between the user and the car OB23.

  As described above, according to the guidance process of the above embodiment, the guide unit 144 is at least one real object that exists in the real world and is included in the outside scene image acquired in step S102. On the other hand, guidance for the attention object, which is a real object that prompts attention, can be implemented by at least the voices SO11 to SO23 (steps S112 and S114). Compared with guidance using the virtual image VI formed in the user's visual field area, guidance using the voices SO11 to SO23 may reduce the visual field of the user, in other words, visibility in the real world is reduced. The fear can be reduced. As a result, it is possible to provide a head-mounted display device (HMD 100) that can prompt the user's attention to a real object that exists in the real world while ensuring visibility in the real world.

  Furthermore, according to the guidance processing of the above embodiment, when it is estimated that the object of interest is not in the direction of the user's line of sight (step S110: none in the direction of the line of sight), only the voices SO11 to SO23. The guidance is implemented by (Step S112). For this reason, the guide unit 144 can suppress the formation of the virtual image VI in the user's visual field region for guiding an object of interest that is not in the direction of the line of sight, and ensure visibility in the real world. Can do. On the other hand, when it is estimated that the target object is in the direction of the user's line of sight (step S110: in the direction of the line of sight), the guide unit 144 further adds to the target object in addition to the voices SO11 to SO23. Guidance is implemented by forming a virtual image VI including the virtual objects VO11 to VO23 to be displayed (step S114). For this reason, the guide part 144 can perform guidance with respect to the object of interest in the direction of the line of sight using both the sound SO11 to SO23 and the virtual image VI. That is, the user can enjoy guidance through visual and auditory senses for the object of interest in the direction of his / her line of sight. As a result, in the head-mounted display device (HMD100) capable of prompting the user's attention to a real object that exists in the real world while ensuring visibility in the real world, the user can further Convenience can be improved.

  Furthermore, according to the guidance process of the above embodiment, when the guidance mode 124 is the “tracking mode”, the guidance unit 144 changes the temporal change of the object of interest in the plurality of outside scene images repeatedly acquired in step S102. Accordingly, the guidance in step S112 or step S114 is repeatedly performed. As a result, the guide unit 144 can perform guidance corresponding to the latest state of the target object at any time. In other words, the guide unit 144 can perform guidance while tracking the target object.

B. Variations:
In the above embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. Good. In addition, the following modifications are possible.

・ Modification 1:
In the said embodiment, it illustrated about the structure of HMD. However, the configuration of the HMD can be arbitrarily determined without departing from the gist of the present invention. For example, each component can be added, deleted, converted, and the like.

The allocation of components to the control unit and the image display unit in the above embodiment is merely an example, and various aspects can be employed. For example, the following aspects may be adopted.
(I) A mode in which processing functions such as a CPU and a memory are mounted on the control unit, and only a display function is mounted on the image display unit. (Ii) Processing functions such as a CPU and a memory are provided in both the control unit and the image display unit. Aspect to be mounted (iii) A mode in which the control unit and the image display unit are integrated (for example, a mode in which the control unit is included in the image display unit and functions as a glasses-type wearable computer)
(Iv) A mode in which a smartphone or a portable game machine is used instead of the control unit. (V) The control unit and the image display unit are connected by a connection via a wireless signal transmission path such as a wireless LAN, infrared communication, or Bluetooth. The aspect which abolished the connection part (code). In this case, power supply to the control unit or the image display unit may be performed wirelessly.

  For example, the configuration of the control unit exemplified in the above embodiment can be arbitrarily changed. Specifically, for example, both the transmission unit (Tx) of the control unit and the reception unit (Rx) of the image display unit have a function capable of bidirectional communication, and may function as a transmission / reception unit. For example, a part of an operation interface (such as various keys and a track pad) provided in the control unit may be omitted. For example, the control unit may be provided with another operation interface such as an operation stick. For example, the control unit may be configured such that a device such as a keyboard or a mouse can be connected, and an input may be received from the keyboard or the mouse. For example, each processing unit (for example, an image processing unit, a display control unit, an augmented reality processing unit, or the like) included in the control unit is an ASIC (Application Specific Integrated Circuit) designed to realize the function. (Integrated circuit). For example, the control unit is mounted with a secondary battery as a power source, but the power source is not limited to a secondary battery, and various batteries can be used. For example, a primary battery, a fuel cell, a solar cell, a thermal cell, or the like may be used.

  For example, the configuration of the image display unit exemplified in the above embodiment can be arbitrarily changed. Specifically, for example, the image light generation unit of the image display unit may be combined with the above-described configuration unit (backlight, backlight control unit, LCD, LCD control unit) or in place of the above-described configuration unit. You may provide the structure part for implement | achieving a system. For example, the image light generation unit may include an organic EL (Organic Electro-Luminescence) display and an organic EL control unit. For example, the image generation unit may include a digital micromirror device or the like instead of the LCD. For example, the present invention can be applied to a laser retinal projection type head-mounted display device. For example, a bone conduction speaker may be provided instead of the earphone or together with the earphone. According to sound reproduction using a bone conduction speaker, sound can be clearly transmitted to the user even in a noisy environment (for example, in a factory).

  FIG. 12 is an explanatory diagram showing an external configuration of the HMD in the modification. The image display unit 20x in FIG. 10A includes a right optical image display unit 26x and a left optical image display unit 28x. These are formed smaller than the optical member of the above-described embodiment, and are respectively disposed obliquely above the left and right eyes of the user when the HMD is mounted. The image display unit 20y of FIG. 10B includes a right optical image display unit 26y and a left optical image display unit 28y. These are formed smaller than the optical member of the above-described embodiment, and are respectively disposed obliquely below the left and right eyes of the user when the HMD is mounted. Thus, it is sufficient that the optical image display unit is disposed in the vicinity of the user's eyes. The size of the optical member forming the optical image display unit is arbitrary, and the optical image display unit covers only a part of the user's eyes, in other words, the optical image display unit completely covers the user's eyes. There may be no aspect.

  For example, the HMD is a binocular transmissive HMD, but may be a monocular HMD. For example, it may be configured as a non-transparent HMD that blocks the transmission of outside scenes when the user wears the HMD, or may be configured as a video see-through in which a camera is mounted on the non-transparent HMD. For example, the earphone may be an ear-hook type or a headband type, or may be omitted. For example, instead of the image display unit worn like glasses, a normal flat display device (liquid crystal display device, plasma display device, organic EL display device, etc.) may be employed. Also in this case, the connection between the control unit and the image display unit may be either wired or wireless. In this way, the control unit can be used as a remote controller for a normal flat display device. For example, instead of an image display unit worn like glasses, an image display unit of another aspect such as an image display unit worn like a hat or an image display unit built in a body protective device such as a helmet is used. It may be adopted. For example, you may comprise as a head-up display (HUD, Head-Up Display) mounted in vehicles, such as a motor vehicle and an airplane, or another transportation means.

Modification 2
In the above embodiment, an example of the guidance process has been described. However, the processing procedure shown in the above embodiment is merely an example, and various modifications are possible. For example, some steps may be omitted, and other steps may be added. The order of the steps performed may be changed.

  For example, the guide unit may omit the detection of the line-of-sight direction (step S108). In this case, the guidance unit may perform guidance for the object of interest only by voice regardless of the direction of the user's line of sight (step S112).

  For example, the guidance unit obtains information provided by DSSS (Driving Safety Support Systems) instead of obtaining the guidance content (step S106) or together with obtaining the guidance content, and performs guidance in step S112 or step S114. May be implemented. In addition, the guide unit may acquire information provided by “support system for vehicles, pedestrians, etc.” other than DSSS, and may perform the guidance in step S112 or step S114.

  For example, the guidance unit may perform only one of guidance in the tracking mode and guidance in the non-tracking mode in the guidance processing (steps S112 and S114). In this case, the guidance mode of the storage unit may be omitted. In addition, for example, the guide unit may perform guidance in the mode without tracking in principle, and may perform guidance in the mode with tracking only when a predetermined condition is satisfied. The predetermined condition is, for example, whether or not the change of the object of interest with time is greater than or equal to a predetermined amount (the detailed determination method is the same as the method a1 described above), and whether or not the feature of the object of interest matches the predetermined feature. (The detailed determination method is the same as the method a2 described above), whether the user's line of sight is directed to the object of interest (the detailed determination method is the same as the method a3 described above), and whether the user is specified for the object of interest. (The detailed determination method is the same as the method a4 described above).

  For example, the guidance unit may shorten the content of the guidance voice when performing guidance using a predetermined number or more of guidance voices in the guidance processing (steps S112 and S114). Specifically, in the example of FIG. 10, the guidance unit may shorten the guidance voice “Please pay attention to the rear-end collision” like “Warning rear-end collision”. The guidance voice may be shortened as “forward attention”. The predetermined number may be two or more, may be three or more, and may be settable by the user. In this way, the time required for guidance using a plurality of guidance voices can be shortened.

  For example, when performing guidance using a plurality of guidance voices in the guidance processing (steps S112 and S114), the guidance unit changes the order in which guidance is performed according to the importance of the object of interest corresponding to the guidance voice. You may decide. In determining the importance of the attention object, the above-described methods a1 to a4 are used. For example, the attention object identified by the method a4 has the highest priority, and the attention object identified by the method a1 has the priority 2. Number, etc. Also, the user may be able to change the correspondence between the method of identifying the object of interest and the priority order. In this way, it is possible to first guide the guidance voice for the attention object with a high priority, so that convenience can be improved.

  For example, in the guidance process (steps S112 and S114), the guidance unit determines whether the speaker in the right earphone or the speaker in the left earphone is in accordance with the positional relationship between the object of interest and the user. A guidance voice may be output. Specifically, for example, when the attention object is located on the right side of the user, the guidance unit outputs guidance voice from the speaker in the right earphone, and when the attention object is located on the left side of the user. The guidance voice may be output from a speaker in the left earphone. For example, when the head-mounted display device includes a speaker having a wide frequency response characteristic and directivity, an audio processing system such as Dolby (registered trademark) can be used. In this case, the guidance unit can generate voice data that reproduces the guidance voice from the position where the target object exists, and can supply the voice data to the voice processing system for output. In this way, the user who has heard the guidance voice can intuitively know the positional relationship between the object and the object of interest.

  For example, in the guidance process (steps S112 and S114), the guidance unit may reproduce a warning sound such as a buzzer, a warning sound such as an alarm, a melody, or the like instead of the guidance voice.

  For example, in the guidance process (steps S112 and S114), the guidance unit may perform guidance for the object of interest by vibration instead of sound (sound wave) or together with sound. In this case, for example, the head-mounted display device includes a vibration unit with a built-in vibration motor or the like, and the guide unit can drive the vibration motor instead of reproducing the guidance voice or together with the reproduction of the voice. Good. In this case as well, the guide unit may select one of, for example, a vibration motor built in the right holding unit and a vibration motor built in the left holding unit, depending on the positional relationship between the object of interest and the user. Either of them may be driven.

  For example, the guidance unit may add parallax to the virtual object displayed in the guidance process (step S114).

  For example, the following guidance processing may be performed together with the guidance processing described above. The head-mounted display device includes a distance acquisition unit that acquires a distance between the user and the object of interest. The guidance unit reproduces the guidance voice when the distance between the user and the object of interest is within a predetermined distance (for example, 20 m). The guidance voice may be a warning sound, a warning sound, or a melody. The guide unit may change the predetermined distance according to the moving speed of the user (for example, 20 m when the speed is 50 km / h or more). The distance acquisition unit may acquire the distance by a stereo camera method using the stereo camera shown in FIG. 2, for example, or acquire the distance by a monocular camera method using the movement distance between the monocular camera and the monocular camera. May be. The distance acquisition unit may acquire the distance by a pattern irradiation method using an infrared (or ultrasonic) light receiving / emitting unit, or by a ToF (Time of Flight) method using an infrared (or ultrasonic) light receiving / emitting unit. The distance may be acquired.

  For example, the following adjustment process may be performed as a pre-process of the above-described guidance process or as a pre-process of step S114 of the guidance process. The head-mounted display device includes an IMU (Inertial Measurement Unit). Using the IMU, the CPU associates an area in the user's visual field direction (for example, an imaging area of a camera) with a virtual image display area in the image display unit. In this way, the augmented reality processing unit can improve the accuracy of alignment between the virtual object displayed in step S114 of the guidance process and the real object (target object). In addition, it is possible to suppress the display blur of the virtual object due to the fine shake of the user's head. Also, when performing mask processing of unnecessary images in augmented reality processing, the speed of mask processing can be improved.

  For example, in an environment where there are a plurality of head-mounted display device users, the guidance process may be performed using the relative positional relationship of each user. Specifically, for example, in specifying the object of interest (step S104), the guide unit of the HMD 100x also uses other HMDs 100y, 100z, etc., having a predetermined relative positional relationship with the user of the HMD 100x, as the object of attention. Good. For example, in the guidance (steps S112 and S114), the guidance unit of the HMD 100x is connected to the speaker in the right earphone and the left earphone in accordance with the relative positional relationship between the user of the HMD 100x and other users of the HMDs 100y and 100z. The guidance voice may be output from any one of the speakers, and the position where the virtual object is displayed may be adjusted. In this way, guidance based on the relative positional relationship of each user can be realized in an environment where there are users of a plurality of head-mounted display devices (for example, in factories, schools, etc.). .

・ Modification 3:
In the above embodiment, an example of guidance information has been shown. However, the configuration and content of the guidance information shown in the above embodiment is merely an example, and various modifications can be made. For example, a part of the configuration may be omitted or another configuration may be added. The data stored in the guide information may be changed.

  For example, there may be one “object” field, which is information for collation using the name of the object of interest, or may be three or more.

  For example, procedure information may be stored instead of or together with the guidance content. “Procedure information” is information used to teach a series of operation procedures to an HMD user. The operation procedures widely include various categories of operation procedures. For example, the procedure of the producer's operation in cooking and sewing, the procedure of the worker's operation in factory work, the procedure of the athlete's operation in sports such as tennis are included. Further, the operation procedure includes a procedure of thinking operation that does not involve the movement of the user's body, such as a procedure of thinking such as a thought process and a procedure of thinking in the learning process. The procedure of these operations may be continuous or discontinuous at regular intervals.

-Modification 4:
In the above embodiment, a specific example of the guidance process has been shown. However, the specific examples shown in the above embodiment are merely examples, and various modifications are possible. For example, the voice guidance mode exemplified and the virtual object mode in augmented reality processing can be arbitrarily changed.

  For example, in the first and second specific examples, the real object is described as one “thing” unit. However, the guide unit may perform the guide process on the assumption that there are a plurality of real objects in one “thing”. Specifically, the guidance process may be performed on the assumption that there are a plurality of real objects such as a body, a bumper, a tire, and a tail lamp in one “car”.

  For example, in the first specific example, the guide unit may display a virtual object representing a scale for guiding the liquid amount to the user in the vicinity of the glass. Similarly, the guiding unit may display a virtual object representing a mark for guiding a position where the liquid injection should be stopped at an arbitrary point on the glass. Similarly, the guide unit may display a scale for guiding weight to the user in the vicinity (preferably the lower part) of the glass. The guide unit may change the value of the scale according to the liquid amount estimated by the image recognition.

  For example, in the second specific example, the guide unit may change the aspect of the virtual object for emphasizing the car according to the distance from the car ahead. Specifically, the guide unit can make the virtual object more conspicuous (larger and darker) as the distance from the vehicle ahead is shorter.

Modification 5:
The present invention is not limited to the above-described embodiments, examples, and modifications, and can be realized with various configurations without departing from the spirit thereof. For example, the technical features in the embodiments, examples, and modifications corresponding to the technical features in each embodiment described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the above-described effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

DESCRIPTION OF SYMBOLS 10 ... Control part 11 ... Decision key 12 ... Illumination part 13 ... Display switching key 14 ... Trackpad 15 ... Luminance switching key 16 ... Direction key 17 ... Menu key 18 ... Power switch 20 ... Image display part 21 ... Right holding part 22 ... Right display drive unit 23 ... Left holding unit 24 ... Left display drive unit 26 ... Right optical image display unit 28 ... Left optical image display unit 30 ... Earphone plug 32 ... Right earphone 34 ... Left earphone 40 ... Connection unit 42 ... Right cord 44 ... Left code 46 ... Connecting member 48 ... Body code 51 ... Transmitting unit 53 ... Receiving unit 61 ... Camera 62 ... Gaze detection unit 63 ... Mic 66 ... 9-axis sensor 100 ... Head mounted display (head-mounted display device)
DESCRIPTION OF SYMBOLS 110 ... Input information acquisition part 120 ... Memory | storage part 122 ... Information for guidance 124 ... Guidance mode 130 ... Power supply 132 ... Wireless communication part 134 ... GPS module 140 ... CPU
142 ... Augmented reality processing unit 144 ... Guide unit 150 ... OS
DESCRIPTION OF SYMBOLS 160 ... Image processing part 170 ... Audio | voice processing part 180 ... Interface 190 ... Display control part 201 ... Right backlight control part 202 ... Left backlight control part 211 ... Right LCD control part 212 ... Left LCD control part 221 ... Right backlight 222 ... Left backlight 241 ... Right LCD
242 ... Left LCD
251 ... Right projection optical system 252 ... Left projection optical system 261 ... Right light guide plate 262 ... Left light guide plate VSync ... Vertical synchronization signal HSync ... Horizontal synchronization signal Data ... Image data OA ... External device PC ... Personal computer VI ... Virtual image SC ... Outside view VR ... Visual field RE ... Right eye LE ... Left eye IM1, IM2 ... Outside scene image OB11-OB26 ... Real object VO11-VO23 ... Virtual object SO11-SO23 ... Audio

Claims (10)

A head-mounted display device that allows a user to visually recognize a virtual image,
An image display unit for allowing the user to visually recognize the virtual image;
An image acquisition unit for acquiring an outside scene image in the visual field direction of the user in a state in which the head-mounted display device is mounted;
A guidance unit that uses sound to guide a target object that is present in the real world and that is at least one real object included in the acquired outside scene image and is a real object that prompts the user to pay attention; ,
A head-mounted display device comprising: The head-mounted display device according to claim 1, further comprising:
A line-of-sight detection unit that detects the direction of the line of sight of the user in a state in which the head-mounted display device is mounted;
The guide part is
When it is estimated that the object of interest is not in the direction of the detected line of sight, the guidance is performed with the sound,
When the object of interest is estimated to be in the direction of the detected line of sight, in addition to the sound, the image display unit further includes the virtual image including a virtual object for additionally displaying the object of interest. A head-mounted display device that performs the guidance by forming the head-mounted display device. The head-mounted display device according to claim 1 or 2,
The guidance is
Guidance for alerting the user to the object of interest;
Information guidance on the object of interest;
Guidance on the position of the object of interest;
Based on procedure information prepared in advance, guidance of the user's work content for the object of interest;
A head-mounted display device including at least one of the above. The head-mounted display device according to any one of claims 1 to 3,
The image acquisition unit repeatedly acquires the outside scene image,
The guide unit further includes:
Recognizing each of the plurality of acquired outside scene images,
A head-mounted display device that uses a real object that is included in the plurality of outside scene images and has a large change over time as the object of interest. The head-mounted display device according to any one of claims 1 to 3,
The guide unit further includes:
Image recognition of the acquired outside scene image,
A head-mounted display device that uses a real object included in the outside scene image and having a predetermined characteristic as the attention object. The head-mounted display device according to any one of claims 1 to 3, further comprising:
A line-of-sight detection unit that detects the direction of the line of sight of the user in a state in which the head-mounted display device is mounted;
The image acquisition unit repeatedly acquires the outside scene image,
The line-of-sight detection unit repeatedly detects the direction of the line of sight,
The guide unit further includes:
The plurality of acquired outside scene images and the direction of the line of sight detected multiple times are collated,
A head-mounted display device in which a real object that is included in the plurality of outside scene images and is estimated to have a high proportion of the line of sight is the attention object. The head-mounted display device according to any one of claims 1 to 3, further comprising:
An input information acquisition unit for acquiring an operation input from the user;
The guide unit further includes:
The acquired outside scene image is collated with the acquired operation input,
A head-mounted display device in which a real object included in the outside scene image and designated by the user is the attention object. The head-mounted display device according to any one of claims 1 to 7,
The image acquisition unit repeatedly acquires the outside scene image,
The guide part is
A head-mounted display device that repeatedly performs the guidance according to a change with time of the object of interest in the plurality of acquired outside scene images. A method for controlling a head-mounted display device,
(A) acquiring an outside scene image in the user's field of view in a state where the head-mounted display device is mounted;
(B) Guidance is given by sound to at least one real object that exists in the real world and is an actual object that is included in the acquired outside scene image and that prompts the user to pay attention. Process,
A method comprising: A computer program,
A function of acquiring an outside scene image in a user's field of view in a state in which the head-mounted display device is mounted;
A function for performing sound guidance on an object of interest that is an actual object that exists in the real world and that is at least one real object included in the acquired outside scene image and that attracts attention to the user;
A computer program that enables a computer to realize

Images