JP2016224086A - Display device, control method of display device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent a problem caused from inconsistency between a range in which display images can be seen and a range in which an outside scene can be seen on a display device when a user watches images with the display device put on and watches a scene outside of the display device.SOLUTION: A head-mount type display device 100 has an image display unit 20 which transmits an outside scene to allow a user to view images displayed together with the outside scene. The head-mount type display device 100 also includes a control unit 140 that acquires an image of outside scene which includes an outside scene viewed through the image display unit 20. Based on the acquired image of outside scene, the control unit 140 recognizes an object which can be viewed through the image display unit 20 and displays a piece of information on the object on the image display unit 20.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a display device, a display device control method, and a program.

  2. Description of the Related Art Conventionally, a display device called a head mounted display (HMD) that is used by being worn on a head by a user is known (for example, see Patent Document 1). This type of display device is used, for example, for the purpose of performing display related to the real space, as described in Patent Document 1, but the wearer has a positional relationship between the content to be displayed and the real space. There was a problem of making a discrepancy. In response to this problem, Patent Document 1 describes a configuration in which a display is corrected by detecting a change in the movement of the user's head.

JP 2007-213407 A

In the configuration described in Patent Document 1, while the HMD is being worn, the user sees the display image but does not see the outside world. For this reason, in the structure of patent document 1, a user's vestibular reflex motor sense and the display discrepancy are prevented.
By the way, when considering a configuration in which an outside scene outside the HMD can be viewed while the user visually recognizes the HMD image, there is a problem that the display image of the HMD and the appearance of the outside scene are different. That is, for the user wearing the HMD, the range in which the display image is visually recognized is unlikely to coincide with the range in which the outside scene can be seen. For example, a situation in which the display image can be seen only in a smaller range than the range in which the outside scene can be seen may occur. A method for performing effective display with such a configuration has been desired.
The present invention has been made in view of the above circumstances, and when a user wears a display device and sees an image and an outside scene of the display device, a range where the display image of the display device can be seen and a range where the outside scene can be seen The purpose is to prevent problems caused by inconsistencies.

In order to achieve the above object, a display device of the present invention includes an external scene image that includes an external scene that transmits an external scene and displays an image that can be viewed together with the external scene, and an external scene that is viewed through the display section. And a control for recognizing an object viewed through the display unit and displaying information on the object on the display unit based on the outside scene image acquired by the outside scene image acquisition unit And a section.
According to the present invention, since an object viewed through the display unit is recognized based on the outside scene image, information on the object is displayed in accordance with the position and size of the object in the outside scene visually recognized by the user. it can. For this reason, inconsistency between the outside scene visually recognized by the user and the display image can be prevented, and effective display can be performed.

In the display device, the control unit may guide the line of sight to the target by the display unit when the target is outside a range where the information is displayed on the display unit. It is characterized by performing display.
According to the present invention, it is possible to guide the line of sight so that the position of the target falls within the range in which information is displayed.

Further, according to the present invention, in the display device, the control unit displays an image of an outside scene in a range that is visible through the display unit based on the outside scene image acquired by the outside scene image acquiring unit. It is characterized by displaying.
ADVANTAGE OF THE INVENTION According to this invention, the appearance of the image of the outside scene visually recognized by the user can be changed.

In the display device according to the present invention, the display unit is a head-mounted display unit mounted on a user's head, and the first display unit displays the image and the first display unit. A second display unit that displays an image that is visible in a wider range than the first display unit on the outside scene side of the unit, and the control unit guides the line of sight to the target by the second display unit It is characterized by performing guidance display.
ADVANTAGE OF THE INVENTION According to this invention, when a head-mounted display apparatus displays on two display parts, inconsistency with the external scene visually recognized by a user and a display image can be prevented. In addition, the user's line of sight can be guided out of the display image in a state where the user is viewing the display image.

In the display device according to the present invention, the display unit is a head-mounted display unit that is mounted on a user's head, and includes a line-of-sight detection unit that detects the line of sight of the user, The control unit changes the display of the display unit in accordance with the direction of the user's line of sight detected by the line-of-sight detection unit.
ADVANTAGE OF THE INVENTION According to this invention, a display image can be made to respond | correspond to a user's eyes | visual_axis, and inconsistency with the external scene visually recognized by a user and a display image can be prevented more effectively.

Further, according to the present invention, in the display device, the control unit displays a partial area of the outside scene image acquired by the outside scene image acquisition unit on the display unit, and the user's line of sight detected by the line of sight detection unit. The position of the partial area displayed on the display unit in the outside scene image is changed in accordance with the direction of.
According to the present invention, since the outside scene image is displayed in correspondence with the user's line of sight, inconsistency between the outside scene visually recognized by the user and the display image can be prevented, and the outside scene image can be visually recognized by the user.

The display device may further include an imaging unit that captures a range including an outside scene that is visible through the display unit, and the outside scene image acquisition unit converts the captured image of the imaging unit into the outside scene image. It is characterized by acquiring as.
According to the present invention, an outside scene image corresponding to the outside scene visually recognized by the user can be acquired and displayed without preparing an outside scene image in advance.

Further, the present invention is characterized in that, in the display device, the outside scene image acquisition unit generates the outside scene image by acquiring and synthesizing a plurality of captured images captured by the imaging unit.
According to the present invention, an outside scene image having a preferable size and angle of view can be obtained without being limited to the angle of view of the imaging unit.

Further, the present invention provides the display device including a plurality of the imaging units, wherein the outside scene image acquisition unit acquires and combines the plurality of captured images captured by the plurality of imaging units, respectively. Generating.
According to the present invention, an outside scene image having a preferable size and angle of view can be obtained using a plurality of captured images.

The display device may further include a connection unit connected to an external device, and the outside scene image acquisition unit acquires the outside scene image from the external device connected to the connection unit. .
According to the present invention, an outside scene image can be acquired without storing the outside scene image in advance or providing a means for imaging.

The display device may further include a position detection unit that detects a current position, and the outside scene image acquisition unit may detect the current position detected by the position detection unit from the external device connected to the connection unit. The corresponding outside scene image is acquired.
According to the present invention, an outside scene image corresponding to the position of the display device can be acquired.

In the display device, the control unit may recognize the target by extracting the target image from the outside scene image acquired by the outside scene image acquisition unit.
According to the present invention, it is possible to quickly detect the position of an object using an outside scene image.
Here, the control unit may recognize the target by detecting a preset target image from the outside scene image and specifying the position of the detected image in the outside scene image. In addition, a target to be recognized may be set in advance in the control unit, and data regarding this target may be stored in the storage unit in advance. The data stored here is data for detecting and recognizing a target, for example, a feature amount of the target image. For example, when the target is an object, the setting data includes a feature amount indicating the color, shape, and other characteristics of the captured image when the object is captured. In this case, the control unit performs a process of extracting the object image from the image data of the outside scene image, calculates the feature amount of the extracted object image, and calculates the feature amount and the feature included in the data stored in advance. Compare with quantity. When the feature amounts are close or the same, it can be determined that the object of the image extracted from the outside scene image matches the target, and this object is recognized as the target. In addition, when data of a plurality of feature amounts is stored for the target, the control unit may recognize the target from the outside scene image based on the plurality of feature amounts. In addition, the control unit may detect the distance to the target using a distance detection unit that detects the distance to the target, and recognize the target based on the detected distance. As a method for recognizing a target from an outside scene image, not only the outside scene image is acquired and the processing based on the feature amount is performed, but also an object included in the outside scene image is instructed by a person (user) using the display device. The target may be selected from the above. In this case, the user's instruction may be an instruction by voice, or may be an operation of a finger or the like on an operation unit provided in the display device. Further, when the display device includes a mechanism for recognizing the gesture operation, the user may specify the target by the gesture operation and recognize the specified target from the outside scene.

In order to achieve the above object, a display device according to the present invention is a display device that is worn on a user's head, and displays an image so that it can be seen through the outside scene and visible together with the outside scene. And a control unit for recognizing an object located outside the range that can be seen through the display area and outputting information related to the object.
According to the present invention, even if a target that attracts the user's attention is not in a position where it can be visually recognized by overlapping the display area, information regarding this target can be provided to the user.

In the display device according to the aspect of the invention, the control unit guides the user to a state where the target can be visually recognized when the target is outside a range in which the target is visible through the display unit. Outputting the information.
ADVANTAGE OF THE INVENTION According to this invention, when the object which makes a user pay attention is not in the position which permeate | transmits a display part and a user can visually recognize, the information regarding an object can be output. Therefore, for example, the user's line of sight can be guided toward the target.

In the display device according to the aspect of the invention, the control unit transmits the user through the display area when the target is visually recognized through the outside of the display area in the display unit. The information for guiding the target to a visually recognizable state is output.
According to the present invention, the user's line of sight is guided when the target to be noticed by the user is at a position that can be seen by the user through the display unit and is not visible at the position overlapping the display area. The target can be visually recognized.

  In the present invention, the control unit may display characters and / or images constituting the information in a display area of the display unit. Moreover, in this invention, the sound output part which outputs a sound is provided, The said control part may output the sound which comprises the said information by the said sound output part.

Further, according to the present invention, in the display device, the control unit acquires position information indicating the position of the target, and based on the acquired position information, the user sees through the display area and visually recognizes the position information. The relative position between the range and the object is obtained and the information is output.
According to the present invention, regardless of the position of the target, information can be output corresponding to the position of the target, and for example, the user can be guided to face the direction of the target. For this reason, for example, even if the position of the target is not a position where the target can be imaged or visually recognized from the display device, information can be output.

Further, in the display device according to the present invention, the control unit acquires an outside scene image obtained by capturing the outside scene, and a range in which the user visually recognizes the display area based on the acquired outside scene image. And calculating the relative position of the object and outputting the information.
ADVANTAGE OF THE INVENTION According to this invention, the information regarding an object can be displayed corresponding to the position of the object in the outside scene visually recognized by the user. For this reason, inconsistency between the outside scene visually recognized by the user and the display image can be prevented, and effective display can be performed.

In order to achieve the above object, the display device control method according to the present invention controls a display device that includes a display unit that displays an image so that the image can be visually recognized together with the outside scene. An outside scene image including an outside scene that is visible through the screen is acquired, a target that is viewed through the display section is recognized based on the acquired outside scene image, and information related to the target is displayed on the display section. It is characterized by doing.
According to the present invention, since an object viewed through the display unit is recognized based on the outside scene image, information on the object is displayed in accordance with the position and size of the object in the outside scene visually recognized by the user. it can. For this reason, an effective display corresponding to the outside scene visually recognized by the user can be performed.

In order to achieve the above object, a program of the present invention transmits a computer that controls a display device including a display unit that displays an image so that the program can be seen through the outside scene and the outside scene. An external scene image acquisition unit that acquires an external scene image including an external scene to be visually recognized, and an object that is viewed through the display unit based on the external scene image acquired by the external scene image acquisition unit, and information on the target Is made to function as a control unit for displaying on the display unit.
According to the present invention, since the display device is controlled to recognize the object that is viewed through the display unit based on the outside scene image, it corresponds to the position and size of the object in the outside scene that the user visually recognizes. Information about the subject can be displayed. For this reason, an effective display corresponding to the outside scene visually recognized by the user can be performed.

In order to achieve the above object, a program according to the present invention provides a computer that controls a display device having a display unit that includes a display area that transmits an outside scene and displays an image so as to be visible together with the outside scene. The control unit recognizes an object located outside the range that can be visually recognized and outputs information related to the object.
According to the present invention, even if a target that attracts the user's attention is not in a position where it can be visually recognized by overlapping the display area, information regarding this target can be provided to the user.

Explanatory drawing which shows the external appearance structure of the head mounted display apparatus of 1st Embodiment. The figure which shows the structure of the optical system of an image display part. The figure which shows the principal part structure of an image display part. The functional block diagram of each part which comprises a head mounting | wearing type display apparatus. The flowchart which shows operation | movement of the head mounted display apparatus of 1st Embodiment. The flowchart which shows the 1st aspect of an outside scene image acquisition process. The flowchart which shows the imaging process which concerns on the 2nd aspect of an outside scene image acquisition process. The flowchart which shows the 2nd aspect of an outside scene image acquisition process. The flowchart which shows the 3rd aspect of an outside scene image acquisition process. It is a figure which shows the example of a display of a head mounted type display apparatus, (A) shows a user's visual field, (B) shows the example of an outside scene image, (C) shows the example of a display of a guidance image. It is a figure which shows the example of a display of a head mounting type display apparatus, (A) shows the example of a reduction display, (B) shows the example which detects a gaze direction, (C) is the example of a display which expanded the gaze direction Indicates. The flowchart which shows operation | movement of the head mounted display apparatus of 2nd Embodiment. It is a figure which shows the specific example of guidance operation | movement of a head mounted display apparatus, (A) shows the example of guidance operation by an image, (B) shows the example of guidance operation by an image and a text. The figure which shows the specific example of guidance operation | movement of a head mounting | wearing type display apparatus. It is a figure which shows the specific example of guidance operation | movement of a head mounting type display apparatus, (A) shows the example which makes a marker the operation object, (B) shows the operation example with respect to a marker, (C) shows the operation with respect to a marker Another example of The flowchart which shows operation | movement of the head mounted display apparatus of 3rd Embodiment. It is a figure which shows the example of a display of a head-mounted display apparatus, (A) shows a user's visual field, (B) shows the example of an outside scene image, (C) shows the example of a display of an outside scene image. The functional block diagram of each part which comprises the head mounted display apparatus of 4th Embodiment. The figure which shows the structure of the optical system of the image display part of 4th Embodiment. The flowchart which shows operation | movement of a head mounting | wearing type display apparatus. It is a figure which shows the example of a display of a head-mounted display apparatus, (A) shows the example of a display of right transparent LCD and left transparent LCD, (B) shows the example of a display by a half mirror, (C) is Indicates that the display overlaps.

[First Embodiment]
FIG. 1 is an explanatory diagram showing an external configuration of a head-mounted display device 100 (display device) according to an embodiment to which the present invention is applied.
The head-mounted display device 100 includes an image display unit 20 that allows the user to visually recognize a virtual image while being mounted on the user's head, and a control device 10 that controls the image display unit 20. The control device 10 also functions as a controller for the user to operate the head mounted display device 100.

  The image display unit 20 is a wearing body that is worn on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a right camera. 61 (imaging part), left camera 62 (imaging part), microphone 63, and distance sensor 64 are provided. The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user when the user wears the image display unit 20.

  The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses.

  The right display drive unit 22 and the left display drive unit 24 are disposed on the side facing the user's head when the user wears the image display unit 20. The right display drive unit 22 and the left display drive unit 24 are collectively referred to simply as “display drive unit”, and the right optical image display unit 26 and the left optical image display unit 28 are simply referred to as “optical image display unit”. Also called.

The display driving units 22 and 24 include liquid crystal displays 241 and 242 (hereinafter referred to as “LCDs 241 and 242”), projection optical systems 251 and 252 described later with reference to FIGS.
The right optical image display unit 26 and the left optical image display unit 28 include light guide plates 261 and 262 (FIG. 2) and a light control plate 20A. The light guide plates 261 and 262 are formed of a light transmissive resin or the like, and guide the image light output from the display driving units 22 and 24 to the user's eyes. The light control plate 20A is a thin plate-like optical element, and is disposed so as to cover the front side of the image display unit 20 which is the side opposite to the user's eye side. As the light control plate 20A, various materials such as a material that has almost no light transmission, a material that is nearly transparent, a material that attenuates the amount of light and transmits light, and a material that attenuates or reflects light of a specific wavelength may be used. it can. By appropriately selecting the optical characteristics (light transmittance, etc.) of the light control plate 20A, the external light amount incident on the right optical image display unit 26 and the left optical image display unit 28 from the outside is adjusted to visually recognize the virtual image. You can adjust the ease. In the present embodiment, a case will be described in which at least a light control plate 20A having such a light transmittance that a user wearing the head-mounted display device 100 can visually recognize an outside scene is used. The light control plate 20A protects the right light guide plate 261 and the left light guide plate 262, and suppresses damage to the right light guide plate 261 and the left light guide plate 262, adhesion of dirt, and the like.
The dimming plate 20A may be detachable from the right optical image display unit 26 and the left optical image display unit 28, or may be mounted by replacing a plurality of types of dimming plates 20A, or may be omitted. .

The right camera 61 is disposed at the end on the right holding unit 21 side on the front surface of the head-mounted display device 100. The left camera 62 is disposed at the end on the left holding unit 23 side on the front surface of the head-mounted display device 100. The right camera 61 and the left camera 62 are digital cameras including an imaging element such as a CCD or a CMOS and an imaging lens, and may be a monocular camera or a stereo camera.
The right camera 61 and the left camera 62 image at least a part of the outside scene in the front side direction of the head-mounted display device 100, in other words, the user's view direction when the head-mounted display device 100 is mounted. . The width of the angle of view of the right camera 61 and the left camera 62 can be set as appropriate, but in the present embodiment, as will be described later, the user visually recognizes through the right optical image display unit 26 and the left optical image display unit 28. An angle of view including the outside world. Furthermore, it is more preferable that the imaging ranges of the right camera 61 and the left camera 62 are set so that the entire field of view of the user through the light control plate 20A can be imaged.
The right camera 61 and the left camera 62 execute imaging in accordance with the control of the image acquisition unit 161 (FIG. 3) included in the control unit 140 and output captured image data to the image acquisition unit 161.

The distance sensor 64 is disposed at the boundary between the right optical image display unit 26 and the left optical image display unit 28. When the user wears the image display unit 20, the position of the distance sensor 64 is substantially in the middle between both eyes of the user in the horizontal direction and above the eyes of the user in the vertical direction.
The distance sensor 64 detects the distance to the measurement object located in the preset measurement direction. The distance sensor 64 includes, for example, a light source such as an LED or a laser diode, and a light receiving unit that receives reflected light that is reflected from the light to be measured. In this case, the distance sensor 64 executes a triangular distance measurement process or a distance measurement process based on a time difference in accordance with the control of the position detection unit 162 described later. The distance sensor 64 may include a sound source that emits ultrasonic waves and a detection unit that receives ultrasonic waves reflected by the measurement object. In this case, the distance sensor 64 performs a distance measurement process based on a time difference until the reflection of the ultrasonic wave according to control of the position detection unit 162 described later. The distance sensor 64 may include a light source, a light receiving unit, or a sound source and a detection unit, and the position detection unit 162 may perform distance measurement processing.
The measurement direction of the distance sensor 64 of the present embodiment is the front side direction of the head-mounted display device 100 and overlaps with the imaging directions of the right camera 61 and the left camera 62.

FIG. 2 is a principal plan view showing the configuration of the optical system provided in the image display unit 20. FIG. 2 shows the user's left eye LE and right eye RE for explanation.
The left display driving unit 24 includes a left backlight 222 having a light source such as an LED and a diffusion plate, a transmissive left LCD 242 arranged on an optical path of light emitted from the diffusion plate of the left backlight 222, and a left LCD 242. A left projection optical system 252 including a lens group that guides the image light L that has passed through the lens. The left LCD 242 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

The left projection optical system 252 includes a collimator lens that converts the image light L emitted from the left LCD 242 into a light beam in a parallel state. The image light L converted into a parallel light beam by the collimator lens is incident on the left light guide plate 262. The left light guide plate 262 is a prism formed with a plurality of reflecting surfaces that reflect the image light L, and the image light L is guided to the left eye LE side through a plurality of reflections inside the left light guide plate 262. The left light guide plate 262 is formed with a half mirror 262A (reflection surface) positioned in front of the left eye LE.
The image light L reflected by the half mirror 262A is emitted from the left optical image display unit 28 toward the left eye LE, and this image light L forms an image on the retina of the left eye LE so that the user can visually recognize the image.

  The right display driving unit 22 is configured to be symmetrical with the left display driving unit 24. The right display driving unit 22 includes a right backlight 221 having a light source such as an LED and a diffusion plate, a transmissive right LCD 241 disposed on an optical path of light emitted from the diffusion plate of the right backlight 221, and a right LCD 241. A right projection optical system 251 including a lens group that guides the image light L that has passed through the lens. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

The right projection optical system 251 includes a collimator lens that converts the image light L emitted from the right LCD 241 into a light beam in a parallel state. The image light L converted into a parallel light beam by the collimator lens is incident on the right light guide plate 261. The right light guide plate 261 is a prism formed with a plurality of reflecting surfaces that reflect the image light L, and the image light L is guided to the right eye RE side through a plurality of reflections inside the right light guide plate 261. The right light guide plate 261 is formed with a half mirror 261A (reflection surface) located in front of the right eye RE.
The image light L reflected by the half mirror 261A is emitted from the right optical image display unit 26 toward the right eye RE, and this image light L forms an image on the retina of the right eye RE to make the user visually recognize the image.

  The image light L reflected by the half mirror 261A and the external light OL transmitted through the light control plate 20A are incident on the right eye RE of the user. The image light L reflected by the half mirror 262A and the external light OL transmitted through the light control plate 20A are incident on the left eye LE. As described above, the head-mounted display device 100 causes the image light L and the external light OL of the internally processed image to overlap each other and enter the user's eyes, and for the user, seeing through the light control plate 20A. An outside scene can be seen, and an image by the image light L is visually recognized over the outside scene. Thus, the head-mounted display device 100 functions as a see-through display device.

  The left projection optical system 252 and the left light guide plate 262 are collectively referred to as “left light guide unit”, and the right projection optical system 251 and the right light guide plate 261 are collectively referred to as “right light guide unit”. The configuration of the right light guide and the left light guide is not limited to the above example, and any method can be used as long as a virtual image is formed in front of the user's eyes using image light. It may be used, or a semi-transmissive reflective film may be used.

  The image display unit 20 is connected to the control device 10 via the connection unit 40. The connection unit 40 includes a main body cord 48, a right cord 42, a left cord 44, and a connecting member 46 that are connected to the control device 10. The right cord 42 and the left cord 44 are codes in which the main body cord 48 is branched into two. The right cord 42 is inserted into the housing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and is connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24.

  The connecting member 46 is provided at a branch point between the main body cord 48, the right cord 42 and the left cord 44, and has a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30. A microphone 63 is provided in the vicinity of the earphone plug 30. The earphone plug 30 to the microphone 63 are combined into one cord, and the cord branches from the microphone 63 and is connected to each of the right earphone 32 and the left earphone 34.

  For example, as shown in FIG. 1, the microphone 63 is arranged so that the sound collecting unit of the microphone 63 faces the user's line of sight, collects sound, and sends the sound signal to the sound processing unit 187 (FIG. 4). Output. For example, the microphone 63 may be a monaural microphone or a stereo microphone, may be a directional microphone, or may be an omnidirectional microphone.

  The right cord 42, the left cord 44, and the main body cord 48 only need to be capable of transmitting digital data, and can be constituted by, for example, a metal cable or an optical fiber. Further, the right cord 42 and the left cord 44 may be combined into a single cord.

  The image display unit 20 and the control device 10 transmit various signals via the connection unit 40. The end of the main body cord 48 opposite to the connecting member 46 and the control device 10 are provided with connectors (not shown) that fit together. By fitting the connector of the main body cord 48 and the connector of the control device 10 or removing this fitting, the control device 10 and the image display unit 20 can be contacted and separated.

  The control device 10 controls the head-mounted display device 100. The control device 10 includes switches including an enter key 11, a lighting unit 12, a display switching key 13, a luminance switching key 15, a direction key 16, a menu key 17, and a power switch 18. In addition, the control device 10 includes a track pad 14 that is operated by a user with fingers.

  The determination key 11 detects a pressing operation and outputs a signal for determining the content operated by the control device 10. The lighting unit 12 includes a light source such as an LED (Light Emitting Diode), and notifies the operating state (for example, power ON / OFF) of the head-mounted display device 100 according to the lighting state of the light source. The display switching key 13 outputs, for example, a signal instructing switching of the image display mode in response to the pressing operation.

  The track pad 14 has an operation surface for detecting a contact operation, and outputs an operation signal according to an operation on the operation surface. The detection method on the operation surface is not limited, and an electrostatic method, a pressure detection method, an optical method, or the like can be adopted. The luminance switching key 15 outputs a signal instructing increase / decrease of the luminance of the image display unit 20 in response to the pressing operation. The direction key 16 outputs an operation signal in response to a pressing operation on a key corresponding to the up / down / left / right direction. The power switch 18 is a switch for switching power on / off of the head-mounted display device 100.

  3A and 3B are diagrams showing the configuration of the main part of the image display unit 20, where FIG. 3A is a main part perspective view of the image display unit 20 viewed from the user's head side, and FIG. 3B is the right camera 61 and the left side. It is explanatory drawing of the angle of view of the camera 62. FIG. In FIG. 3A, illustration of the right cord 42 and the left cord 44 connected to the image display unit 20 is omitted.

FIG. 3A shows the side of the image display unit 20 in contact with the user's head, in other words, the side visible to the user's right eye RE and left eye LE. In other words, the back sides of the right optical image display unit 26 and the left optical image display unit 28 are visible.
In FIG. 3A, the half mirror 261A that irradiates the user's right eye RE with the image light and the half mirror 262A that irradiates the left eye LE with the image light appear as substantially rectangular areas. Further, the entire right optical image display unit 26 and left optical image display unit 28 including the half mirrors 261A and 262A transmit external light as described above. For this reason, the user sees the outside scene through the entire right optical image display unit 26 and left optical image display unit 28, and visually recognizes a rectangular display image at the positions of the half mirrors 261A and 262A.

As described above, the right camera 61 is arranged at the end on the right holding unit 21 side so as to face the front of the image display unit 20, and the left camera 62 is arranged at the end on the left holding unit 23 side. Further, the distance sensor 64 is disposed facing forward in the center of the right optical image display unit 26 and the left optical image display unit 28.
FIG. 3B is a diagram schematically showing the positions of the right camera 61, the left camera 62, and the distance sensor 64 together with the right eye RE and the left eye LE of the user in plan view. The angle of view (imaging range) of the right camera 61 is indicated by CR, and the angle of view (imaging range) of the left camera 62 is indicated by CL. Note that FIG. 3B shows horizontal field angles CR and CL, but the actual field angles of the right camera 61 and the left camera 62 expand in the vertical direction as in a general digital camera.

The angle of view CR and the angle of view CL are substantially bilaterally symmetric with respect to the center position of the image display unit 20, and both the field angle CR and the angle of view CL include the frontal direction of the center position of the image display unit 20. For this reason, the angles of view CR and CL overlap in front of the central position of the image display unit 20.
For example, as shown in FIG. 3B, when there is a target OB in the front direction of the image display unit 20, the target OB is included in both the view angle CR and the view angle CL. For this reason, the target OB is captured in both the captured image of the right camera 61 and the captured image of the left camera 62. Here, when the user gazes at the target OB, the user's line of sight is directed toward the target OB as indicated by reference signs RD and LD in the figure. In general, the viewing angle of a human is about 200 degrees in the horizontal direction and about 125 degrees in the vertical direction. Among them, the effective field of view with excellent information receiving ability is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction. Furthermore, the stable gaze field in which the gaze point that the human gazes at appears to be quickly and stably is about 60 to 90 degrees in the horizontal direction and about 45 to 70 degrees in the vertical direction.
Therefore, when the gazing point is the target OB, the effective visual field is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction around the lines of sight RD and LD, and 60 to 90 degrees in the horizontal direction and 45 degrees to the vertical direction. A stable visual field is about 70 degrees, and a viewing angle is about 200 degrees in the horizontal direction and about 125 degrees in the vertical direction.

  In addition, an actual field of view that a user wearing the head-mounted display device 100 sees through the image display unit 20 and through the right optical image display unit 26 and the left optical image display unit 28 is a real field ( This is called FOV (Field Of View). In the configuration of the present embodiment shown in FIGS. 1 and 2, the real field of view corresponds to an actual field of view that the user sees through the right optical image display unit 26 and the left optical image display unit 28. The real field of view is narrower than the viewing angle and stable focus field described with reference to FIG.

  The right camera 61 and the left camera 62 are preferably capable of imaging a wider range than the user's field of view. Specifically, the entire view angles CR and CL are at least wider than the effective field of view of the user. Is preferred. Further, it is more preferable that the entire angles of view CR and CL are wider than the actual field of view of the user. More preferably, the entire angle of view CR, CL is wider than the stable viewing field of the user, and most preferably, the entire angle of view CR, CL is wider than the viewing angle of the user.

  For this reason, the right camera 61 and the left camera 62 are arranged such that the field angle CR and the field angle CL overlap each other in front of the image display unit 20 as illustrated in FIG. Further, the right camera 61 and the left camera 62 may be configured by wide angle cameras. That is, the right camera 61 and the left camera 62 may include a so-called wide-angle lens as an imaging lens and can capture a wide angle of view. The wide-angle lens may include a lens called a super-wide-angle lens or a quasi-wide-angle lens, or may be a single-focus lens or a zoom lens. The lens group composed of a plurality of lenses includes the right camera 61 and the left camera. 62 may be provided. Further, the angle of view CR of the right camera 61 and the angle of view CL of the left camera 62 may not be the same angle. Further, the imaging direction of the right camera 61 and the imaging direction of the left camera 62 do not have to be completely parallel. When the captured image of the right camera 61 and the captured image of the left camera 62 are superimposed, it is only necessary to capture a range wider than the visual field of the user.

  In FIG. 3B, the detection direction of the distance sensor 64 is indicated by reference numeral 64A. In the present embodiment, the distance sensor 64 is configured to be able to detect the distance from the center position of the image display unit 20 to the object located in the front direction, and detects the distance to the target OB, for example. Since the user wearing the head-mounted display device 100 turns his head in the direction of gazing, it can be considered that the object to be gazed is in front of the image display unit 20. For this reason, if the distance sensor 64 arrange | positioned in the center of the image display part 20 makes the front of the image display part 20 the detection direction 64A, it can detect the distance to the object which a user gazes at.

  Further, as shown in FIG. 3A, a line-of-sight sensor 68 is disposed on the user side of the image display unit 20. A pair of line-of-sight sensors 68 are provided at the center positions of the right optical image display unit 26 and the left optical image display unit 28 so as to correspond to the right eye RE and the left eye LE of the user. The line-of-sight sensor 68 includes, for example, a pair of cameras that respectively capture the right eye RE and the left eye LE of the user. The line-of-sight sensor 68 captures images in accordance with the control of the control unit 140 (FIG. 4), and the control unit 140 detects reflected light and pupil images on the eyeball surfaces of the right eye RE and the left eye LE from the captured image data. Identify the direction.

FIG. 4 is a functional block diagram of each part constituting the head-mounted display device 100.
The head-mounted display device 100 includes an interface 125 for connecting various external devices OA that are content supply sources. As the interface 125, for example, an interface corresponding to wired connection such as a USB interface, a micro USB interface, a memory card interface, or the like may be used, and the interface 125 may be configured by a wireless communication interface. The external device OA is an image supply device that supplies an image to the head-mounted display device 100, and a personal computer (PC), a mobile phone terminal, a portable game machine, or the like is used.

The control device 10 includes a control unit 140, an input information acquisition unit 110, a storage unit 120, a transmission unit (Tx) 51, and a transmission unit (Tx) 52.
The input information acquisition unit 110 is connected to the operation unit 135. The operation unit 135 includes the track pad 14, the direction key 16, the power switch 18, and the like. The input information acquisition unit 110 acquires input contents based on a signal input from the operation unit 135. The control device 10 includes a power supply unit (not shown), and supplies power to each unit of the control device 10 and the image display unit 20.

The storage unit 120 is a nonvolatile storage device, and stores various computer programs and data related to these programs. The storage unit 120 may store still image data or moving image data to be displayed on the image display unit 20.
The storage unit 120 stores setting data 121. The setting data 121 includes various setting values used by the control unit 140. The setting value included in the setting data 121 may be a value input in advance by operating the operation unit 135, or a setting value from an external device OA or another device (not shown) via the communication unit 117 or the interface 125. May be received and stored.

A three-axis sensor 113, a GPS 115, a communication unit 117, and a voice recognition unit 114 are connected to the control unit 140. The triaxial sensor 113 is a triaxial acceleration sensor, and the control unit 140 acquires the detection value of the triaxial sensor 113. With the three-axis sensor 113, the control unit 140 can detect the movement of the control device 10, and can detect an operation such as shaking the control device 10, for example. The 3-axis sensor 113 may be replaced with a 9-axis sensor. In this case, the control unit 140 can acquire detection values of the three-axis acceleration sensor, the three-axis angular velocity sensor, and the three-axis geomagnetic sensor, and can detect the attitude, orientation, and movement of the control device 10, for example.
The GPS 115 includes an antenna (not shown), receives a GPS (Global Positioning System) signal, and calculates the current position of the control device 10. The GPS 115 outputs the current position and the current time obtained based on the GPS signal to the control unit 140. The GPS 115 may have a function of acquiring the current time based on information included in the GPS signal and correcting the time counted by the control unit 140.

The communication unit 117 executes wireless data communication conforming to a standard such as a wireless LAN (WiFi (registered trademark)), Miracast (registered trademark), or Bluetooth (registered trademark).
When the external device OA is wirelessly connected to the communication unit 117, the control unit 140 acquires content data from the communication unit 117 and causes the image display unit 20 to display an image. On the other hand, when the external device OA is wired to the interface 125, the control unit 140 acquires content data from the interface 125 and causes the image display unit 20 to display an image. The communication unit 117 and the interface 125 function as a data acquisition unit DA that acquires content data from the external device OA.

  The control unit 140 stores a CPU (not shown) for executing a program, a RAM (not shown) for temporarily storing a program and data executed by the CPU, and a basic control program and data executed by the CPU in a nonvolatile manner. ROM (not shown). The control unit 140 reads and executes the computer program stored in the storage unit 120, and executes an operating system (OS) 150, an image processing unit 160, an image acquisition unit 161, a position detection unit 162, an image selection unit 163, and an operation processing unit 164. , Function as a display position control unit 165, a sound processing unit 187, and a display control unit 190.

The image processing unit 160 acquires an image signal included in the content. The image processing unit 160 separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the acquired image signal. Further, the image processing unit 160 generates a clock signal PCLK using a PLL (Phase Locked Loop) circuit or the like (not shown) according to the period of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like (not shown). The image processing unit 160 stores the converted digital image signal in the RAM of the control unit 140 for each frame as image data of the target image (Data in the figure). This image data is, for example, RGB data.
Note that the image processing unit 160 may perform a resolution conversion process for converting the resolution of the image data into a resolution suitable for the right display driving unit 22 and the left display driving unit 24 as necessary. The image processing unit 160 also performs image adjustment processing for adjusting the brightness and saturation of image data, 2D image data is created from 3D image data, or 2D / 3D conversion processing is performed to generate 3D image data from 2D image data. May be executed.

  The image processing unit 160 transmits the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data Data stored in the RAM via the transmission units 51 and 52, respectively. The transmission units 51 and 52 function as a transceiver and execute serial transmission between the control device 10 and the image display unit 20. The image data Data transmitted via the transmission unit 51 is referred to as “right eye image data”, and the image data Data transmitted via the transmission unit 52 is referred to as “left eye image data”.

  The display control unit 190 generates a control signal for controlling the right display drive unit 22 and the left display drive unit 24, and generates and generates image light by each of the right display drive unit 22 and the left display drive unit 24 based on the control signal. Control injection. Specifically, the right LCD 241 is controlled to be turned on / off by the right LCD control unit 211, and the right backlight 221 is turned on / off by the right backlight control unit 201. Further, the display control unit 190 controls the driving of the left LCD 242 by the left LCD control unit 212 and the driving of the left backlight 222 by the left backlight control unit 202.

  The audio processing unit 187 acquires an audio signal included in the content, amplifies the acquired audio signal, and outputs the amplified audio signal to the right earphone 32 and the left earphone 34. In addition, the sound processing unit 187 acquires sound collected by the microphone 63 and converts it into digital sound data. The voice processing unit 187 may perform a preset process on the digital voice data.

  The image display unit 20 includes the right camera 61, the left camera 62, and the distance sensor 64 described above. The image display unit 20 includes an interface 25, a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, and a left guide as a left optical image display unit 28. An optical plate 262, a nine-axis sensor 66, and a line-of-sight sensor 68 are provided.

  The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). When the image display unit 20 is mounted on the user's head, the control unit 140 can detect the movement of the user's head based on the detection value of the 9-axis sensor 66. For example, the control unit 140 can estimate the magnitude of the tilt and the tilt direction of the image display unit 20 based on the detection value of the 9-axis sensor 66.

The interface 25 includes a connector to which the right cord 42 and the left cord 44 are connected. The interface 25 outputs the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data Data transmitted from the transmission unit 51 to the corresponding reception units (Rx) 53 and 54. Further, the interface 25 outputs a control signal transmitted from the display control unit 190 to the corresponding receiving units 53 and 54, the right backlight control unit 201, or the left backlight control unit 202.
The interface 25 is an interface for connecting the right camera 61, the left camera 62, the distance sensor 64, the 9-axis sensor 66, and the line-of-sight sensor 68. Imaging data of right camera 61 and left camera 62, detection result of distance sensor 64, detection result of acceleration (three axes), angular velocity (three axes), geomagnetism (three axes) by nine-axis sensor 66, and gaze sensor 68 The detection result is sent to the control unit 140 via the interface 25.

  The right display drive unit 22 includes the right backlight 221, the right LCD 241, and the right projection optical system 251 described above. The right display driving unit 22 includes a receiving unit 53, a right backlight (BL) control unit 201 that controls the right backlight (BL) 221, and a right LCD control unit 211 that drives the right LCD 241.

  The reception unit 53 operates as a receiver corresponding to the transmission unit 51 and executes serial transmission between the control device 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right eye image data Data input via the reception unit 53. .

  The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display driving unit 24 includes the left backlight 222, the left LCD 242 and the left projection optical system 252 described above. The left display driving unit 24 includes a receiving unit 54, a left backlight control unit 202 that drives the left backlight 222, and a left LCD control unit 212 that drives the left LCD 242.

The reception unit 54 operates as a receiver corresponding to the transmission unit 52 and executes serial transmission between the control device 10 and the image display unit 20. The left backlight control unit 202 drives the left backlight 222 based on the input control signal. The left LCD control unit 212 drives the left LCD 242 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right-eye image data Data input via the reception unit 54. .
The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are collectively referred to as a right “image light generation unit”. Similarly, the left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are collectively referred to as a left “image light generation unit”.

  As shown in FIG. 3A, the range in which the user wearing the head-mounted display device 100 visually recognizes the image light emitted from the half mirrors 261A and 262A is smaller than the actual visual field. For example, when the head-mounted display device 100 displays an image that exhibits an AR (Augmented Reality) effect (hereinafter referred to as an AR image), the AR image is displayed so as to overlap the target OB that the user gazes at. In this case, the AR effect can be obtained by making the AR image appear to overlap the target OB, but the area where the AR image is visually recognized is restricted by the size of the half mirrors 261A and 262A. For this reason, even if the target OB is included in the real field of view, it may be difficult to display the AR image at a position overlapping the target OB. In addition, when displaying an image other than the AR image, there is a possibility that the area in which the head-mounted display device 100 can display an image is smaller than the actual visual field, which may be a functional limitation.

  Therefore, the head-mounted display device 100 uses the images of the outside scene in a wider range than the actual field of view of the user, such as images captured by the right camera 61 and the left camera 62 and images prepared in advance. Display corresponding to the outside scene, such as displaying the AR content corresponding to the target OB in the outside scene that is being watched.

  The image acquisition unit 161 acquires an outside scene image corresponding to the current position of the head-mounted display device 100. The outside scene image may be a captured image of the right camera 61 and the left camera 62 or an image acquired from an external device that can communicate via the communication unit 117. The process in which the image acquisition unit 161 acquires an outside scene image will be described later.

  The position detection unit 162 performs a function for the control unit 140 to detect (recognize) an object (object) from the outside scene image. In this function, the position detection unit 162 analyzes an outside scene image acquired by the image acquisition unit 161, for example. The position detection unit 162 detects an object image from the outside scene image and specifies the position of the detected image in the outside scene image, thereby detecting the position of the object that the user gazes at. This process will be described later. Further, the position detection unit 162 may detect the position of the object by controlling the distance sensor 64 and measuring the distance to the detected object. You may detect the position of a target object using distance together.

The image selection unit 163 selects the AR content to be displayed from among the AR content input from the external device OA to the interface 125.
The operation processing unit 164 detects the movement of the image display unit 20 based on the detection value of the 9-axis sensor 66 and performs processing corresponding to the movement of the image display unit 20. The movement of the image display unit 20 can be regarded as the movement of the head of the user wearing the head-mounted display device 100. The operation processing unit 164 controls the line-of-sight sensor 68 to detect the user's line-of-sight direction.
The display position control unit 165 corresponds to the position of the object detected by the position detection unit 162, the movement of the image display unit 20 detected by the motion processing unit 164, and / or the right display driving unit 22 corresponding to the user's line-of-sight direction. And the control which concerns on the display position of AR image of the left display drive part 24 is performed.

FIG. 5 is a flowchart showing the operation of the head-mounted display device 100.
When the user wears the head-mounted display device 100 and the display of the AR content is instructed by the operation of the operation unit 135, the control unit 140 starts an operation related to the display (step S11).
The controller 140 causes the image acquisition unit 161 to execute an outside scene image acquisition process (step S12). There are three specific modes of outside scene image acquisition processing.

FIG. 6 is a flowchart showing a first aspect of the outside scene image acquisition processing in step S12. In the first mode, the image acquisition unit 161 controls the right camera 61 and the left camera 62 to execute imaging (step SA1), and acquires captured image data (step SA2).
The image acquisition unit 161 determines whether there are a plurality of acquired captured image data (step SA3). In the head-mounted display device 100 of this embodiment, each of the two right cameras 61 and the left camera 62 outputs captured image data, so the image acquisition unit 161 acquires a plurality of captured image data (step SA3; YES) ) To obtain outside scene image data by synthesizing the acquired captured image data (step SA4). When the number of captured image data acquired in step SA2 is one (step SA3; NO), the image acquisition unit 161 uses the acquired captured image data as outside scene image data. The case where the number of captured image data is one is, for example, the case where one captured image data within the range captured by the right camera 61 and the left camera 62 is input to the control unit 140, or the case where the right camera 61 or the left camera 62 When it is set to use only one of them, there is a case where the configuration is provided with one camera.

7 and 8 show a second aspect of the outside scene image acquisition process, FIG. 7 is a flowchart showing the imaging process, and FIG. 8 is a flowchart showing the outside scene image acquisition process.
In the second mode, during operation of the head-mounted display device 100, the image acquisition unit 161 executes imaging with the right camera 61 and the left camera 62 at regular intervals. That is, the image acquisition unit 161 starts an imaging operation triggered by the operation start of the head-mounted display device 100 or the AR content display start (step SB1), and executes imaging by the right camera 61 and the left camera 62 ( Step SB2).
The image acquisition unit 161 acquires captured image data captured by the right camera 61 and the left camera 62 (step SB3). Subsequently, the image acquisition unit 161 acquires the coordinates of the current position of the head-mounted display device 100 and / or the detection value of the 9-axis sensor 66 by the GPS 115 (step SB4). For example, the image acquisition unit 161 can acquire the current position of the GPS 115 and the detection values of the geomagnetic sensor, the angular velocity sensor, and the acceleration sensor of the nine-axis sensor 66, and can specify the current position and posture of the head-mounted display device 100. .

The image acquisition unit 161 stores the captured image data acquired in step SB3 in the storage unit 120 in association with the current position and detection value acquired in step SB4 (step SB5).
Thereafter, the image acquisition unit 161 determines whether or not a condition for ending imaging is satisfied (step SB6). If the condition for ending imaging is not satisfied (step SB6; NO), the process returns to step SB2. . The condition for the end of imaging is that when the end of imaging is instructed by operating the operation unit 135, the operation mode of the head-mounted display device 100 is the operation mode for displaying AR content, the right camera 61, and the left camera 62. For example, the mode may be switched to an operation mode in which imaging is not performed.

For example, the head-mounted display device 100 may repeatedly execute the operations of Steps SB2 to SB6 every predetermined time in a state where the right camera 61 and the left camera 62 can perform imaging. In this case, regardless of whether or not the head-mounted display device 100 performs AR display, the storage unit 120 stores a captured image in association with the current position of the head-mounted display device 100 and the detection value of the sensor. Data is stored. If the storage capacity of the storage unit 120 is limited, the image acquisition unit 161 transmits the captured image data, the current position, and the detection value of the sensor in association with each other to an external device connected via the communication unit 117. May be. Examples of the external device in this case include a storage device connected to a LAN (Local Area Network), a storage device called online storage, a cloud storage server, and the like.
If the condition for ending imaging is satisfied (step SB6; YES), this process ends.

The outside scene image acquisition process of FIG. 8 uses the captured image data stored in the operation of FIG. 7, but does not need to be executed in conjunction with or in cooperation with the operation of FIG. That is, the execution timing of the operation of FIG. 7 and the operation of FIG.
The image acquisition unit 161 detects and acquires the current position of the head-mounted display device 100 using the GPS 115, and acquires the detection value of the 9-axis sensor 66 (step SB11). Subsequently, the image acquisition unit 161 acquires captured image data stored in association with the current position acquired in step SB11 and the detection value of the sensor from an external device connected by the storage unit 120 or the communication unit 117 ( Step SB12). In step SB12, the image acquisition unit 161 searches the captured image data using the current position and sensor detection value acquired in step SB11 as search conditions. The image acquisition unit 161 may acquire captured image data that meets all of the search conditions, or may acquire captured image data that meets only some of the search conditions.

  The image acquisition unit 161 determines whether there are a plurality of captured image data acquired in step SB12 (step SB13). When there are a plurality of captured image data (step SB13; YES), the image acquisition unit 161 combines the acquired captured image data into one captured image data (step SB14), and proceeds to step S13 in FIG. To do. On the other hand, when the acquired captured image data is one (step SB13; NO), the image acquisition unit 161 proceeds to step S13 in FIG.

FIG. 9 shows a third mode of outside scene image acquisition processing.
In the third aspect, the image acquisition unit 161 acquires captured image data from an external device. The external device in this case is a server device or the like that is communicably connected to the head-mounted display device 100 via a communication line such as a LAN laid in a limited range or a wide area network such as the Internet. is there. The external device stores image data corresponding to the position of the head-mounted display device 100, the detection value of the sensor, and the like.

The image acquisition unit 161 detects and acquires the current position of the head-mounted display device 100 with the GPS 115, and acquires the detection value of the 9-axis sensor 66 (step SC1). Subsequently, the image acquisition unit 161 transmits the current position and sensor detection value acquired in step SB11 to an external device connected via the communication unit 117, and requests captured image data (step SC3).
Thereafter, the image acquisition unit 161 receives the image data transmitted by the external device as a response to the request transmitted in step SC3 (step SC3).

  The image acquisition unit 161 determines whether there are a plurality of captured image data received in step SC3 (step SC4). When there are a plurality of captured image data (step SC4; YES), the image acquisition unit 161 combines the received plurality of captured image data into one captured image data (step SC5), and proceeds to step S13 in FIG. To do. On the other hand, when the received captured image data is one (step SC4; NO), the image acquisition unit 161 proceeds to step S13 in FIG.

The outside scene image acquisition process of the first, second, or third aspect is executed, and the image acquisition unit 161 obtains one outside scene image.
Thereafter, the position detection unit 162 recognizes an AR display target from the acquired outside scene image (step S13). The target of AR display is an object that the user of the head-mounted display device 100 visually recognizes in the outside scene, and may be any one of humans, animals, and inanimate objects. Including things that cannot be moved. The object is viewed through the right optical image display unit 26 and the left optical image display unit 28 when the user faces the target. The head-mounted display device 100 performs AR display by displaying information related to a preset target on the image display unit 20 so that the information can be seen simultaneously with the target.

  A target to be detected is set in the position detection unit 162 in advance. The target can be set for each AR display content, and is included in, for example, data of AR display content (hereinafter referred to as AR content). Further, data related to the target to be detected may be stored as setting data 121. In step S <b> 13, the position detection unit 162 acquires data for detecting a target from the AR content or setting data 121, and recognizes a target (target) included in the outside scene image using this data. The data for detecting the object is data used for processing for detecting the image of the object from the captured image, and is a feature amount of the image of the object. For example, when the target object is an object, the setting data includes a feature amount indicating the color, shape, and other characteristics of the captured image when the object is imaged. In this case, the position detection unit 162 performs a process of extracting the object image from the image data of the outside scene image, calculates the feature amount of the extracted object image, and calculates the calculated feature amount and the feature included in the setting data 121. Compare with quantity. When the feature values are close or the same, the object of the image extracted from the outside scene image can be recognized as the target. Further, when the setting data 121 includes a plurality of feature amounts for the object, the position detection unit 162 can detect and recognize the object from the outside scene image based on the plurality of feature amounts. If the target cannot be recognized in the outside scene image, the position detection unit 162 waits until the image acquisition unit 161 acquires a new outside scene image in step S12, and performs processing for recognizing the target for the new outside scene image.

In step S13, the method of recognizing the object by the position detection unit 162 is not limited to the method of recognizing the image of the object based on the feature amount of the image as described above. For example, the target may be selected from an object included in the outside scene image according to a user instruction. In this case, the user's instruction may be an instruction by voice, and the voice processing unit 187 converts the voice collected by the microphone 63 into text, so that the position detection unit 162 recognizes and identifies the target. Get information for. For example, when the sound that specifies the feature of the object in the captured image, such as the color or the shape of the target, is converted into text, the position detection unit 162 detects an image corresponding to the specified feature from the captured image, recognize.
The method for inputting information related to the object may be an operation on the track pad 14 or the control unit 140 may detect a gesture operation with a finger or the like. In this case, the user may perform a gesture operation by moving a finger or the like within the imaging range of the right camera 61 and the left camera 62, and the control unit 140 may detect this gesture operation. Specifically, a method of designating the target object itself by pointing the target object at the user may be employed. In addition, a gesture may be used such that the user points the direction of the object with a hand or a finger, and the range in which the user visually recognizes the object is surrounded with the hand or the finger.

  Subsequently, the position detection unit 162 calculates the position of the target recognized in step S13, or the position and distance (step S14). The position detection unit 162 calculates a position and distance based on the head-mounted display device 100. More specifically, the relative position of the object with respect to the image display unit 20 and the distance from the image display unit 20 to the object are calculated. In this case, the center position of the right optical image display unit 26 and the left optical image display unit 28 of the image display unit 20 may be used as a reference when calculating the position and distance.

  As a method for calculating the position by the position detection unit 162, a method can be employed in which the coordinates in the target outside scene image recognized from the outside scene image in step S13 are obtained, and the relative position of the object with respect to the image display unit 20 is obtained from the obtained coordinates. In this case, the position detection unit 162 can calculate an estimated value of the distance from the image display unit 20 to the target from the size of the target image in the outside scene image. Here, data indicating the positional relationship between the outside scene image and the position of the image display unit 20 may be added to the data of the outside scene image.

  In addition, the position detection unit 162 images the target with the right camera 61 and the left camera 62, detects the target image in the captured image, and based on the coordinates and size of the detected image in the captured image, the position and distance of the target And may be calculated. When the image acquisition unit 161 acquires an outside scene image by the operation of FIG. 6, the above processing can be executed based on the outside scene image. In addition, when the image acquisition unit 161 acquires the outside scene image by the operations of FIGS. 7 and 8 or the operation of FIG. 9, the position detection unit 162 captures the image by the right camera 61 and the left camera 62 after detecting the target from the outside scene image. To execute the above process.

  Further, the position detection unit 162 may obtain the distance to the target using the distance sensor 64. When the position detection unit 162 determines that the target detected from the outside scene image is located in the detection direction of the distance sensor 64, the position detection unit 162 operates the distance sensor 64 to detect the distance, and the detected distance is set as the distance to the target. do it.

  Based on the position and distance of the target calculated by the position detection unit 162 in step S14, the display position control unit 165 calculates the position of the target in the display area of the right optical image display unit 26 and the left optical image display unit 28 (step S14). S15). Specifically, the display areas of the right optical image display unit 26 and the left optical image display unit 28 correspond to half mirrors 261A and 262A, as shown in FIG. The display position control unit 165 calculates the target position of the AR content with reference to the range of the outside scene visually recognized by the user through the half mirrors 261A and 262A.

  The display position control unit 165 determines whether or not the target position calculated in step S15 is within the display area of the image display unit 20 (step S16). Here, the display area refers to a range of an outside scene that is visually recognized by the user through the half mirrors 261A and 262A. This target exists in the outside scene, and the display position control unit 165 determines whether or not the target appears to overlap with the half mirrors 261A and 262A to the user.

In step S16, the display position control unit 165 may determine each of the half mirror 261A and the half mirror 262A. In this case, the display position control unit 165 determines whether or not the target is included in the range that the user visually recognizes through the half mirror 261A, and the target is included in the range that the user visually recognizes through the half mirror 262A. Whether or not is determined.
For example, when the image selection unit 163 selects 3D AR content, the target is preferably within the display range in both the half mirrors 261A and 262A. In this case, the display position control unit 165 determines that the target is in the display area when the target is in the display range in both the half mirrors 261A and 262A.
Further, for example, when the image selection unit 163 selects the AR content for displaying the explanatory text or the annotation text or the image regarding the target, the AR content may be displayed on one of the half mirrors 261A and 262A. In this case, the display position control unit 165 determines that the target is within the display area if the target is within the display range of one of the half mirrors 261A and 262A.
As described above, the display position control unit 165 may integrate the determination results regarding the half mirrors 261A and 262A according to the type of AR content selected by the image selection unit 163, and perform the determination in step S16.

  When the display position control unit 165 determines that it is not within the target position display region (step S16; NO), the operation processing unit 164 controls the display control unit 190 to display a guidance image for guiding the line of sight as a right optical image. The information is displayed on the display unit 26 and the left optical image display unit 28 (step S17).

FIG. 10 is a diagram illustrating a display example in the present embodiment, in which (A) shows a user's visual field VR, (B) shows an example of an outside scene image, and (C) shows a display example of a guidance image. .
FIG. 10A shows an example of the visual field VR of the user's right eye. The visual field of the user's left eye is the same as or symmetrical with that of FIG. An area corresponding to the half mirror 261A in the visual field VR is a display area D1, and an image drawn on the right LCD 241 (FIG. 4) is displayed in the display area D1.
In the example of FIG. 10A, a bicycle can be seen in the field of view VR, but this bicycle is a real bicycle placed outside the image display unit 20. That is, the user views the bicycle in the outside scene through the right optical image display unit 26.

FIG. 10B shows an example of the outside scene image P acquired by the image acquisition unit 161. In this example, the image acquisition unit 161 acquires an outside scene image P in a range wider than the range visually recognized by the user, and the outside scene image P includes a range larger than the range visible in the user's visual field VR. For reference, in the outside scene image P, a range that can be seen in the visual field VR is indicated by a symbol P1, and a range corresponding to the display region D1 is indicated by a symbol P2.
In the example of FIG. 10B, the target OB corresponding to the AR content is outside the range P1 that is visible in the visual field VR, and is also outside the range P2 corresponding to the display area D1. For this reason, the display position control unit 165 determines in step S16 that the position of the target OB is not within the display area. In this case, the guidance image displayed in step S17 by the operation processing unit 164 is an image that moves the user's line of sight toward the target OB, as shown in FIG. 10C, for example. A guide image D2 in FIG. 10C is an arrow indicating the direction of the target OB. When the user changes the direction of the head according to this arrow, the target OB may enter the display area D1.

  The motion processing unit 164 determines whether or not the movement of the head-mounted display device 100 is detected (step S18). The motion processing unit 164 detects a change in the orientation or position of the image display unit 20 using the 9-axis sensor 66, and determines that the movement is detected when there is a change larger than a preset threshold value. Further, the operation processing unit 164 may detect the movement of the head mounted display device 100 using the GPS 115. When it determines with not detecting the movement of the head mounted display apparatus 100 (step S18; NO), the operation | movement process part 164 continues the display of a guidance image, and performs determination of step S18 for every fixed time. When the movement of the head-mounted display device 100 is detected (step S18; YES), the operation processing unit 164 returns to step S14.

  When the display position control unit 165 determines that the target is within the display area (step S16; YES), the operation processing unit 164 controls the display control unit 190 to display the AR content on the right optical image display unit 26. Then, it is displayed on the left optical image display unit 28 (step S19). Here, the operation processing unit 164 may thin the AR content data and cause the display control unit 190 to display the data. Specifically, the data thinning includes processing such as reducing a frame rate by extracting a part of a frame of moving image data, reducing the resolution of the frame, and the like. Thereby, there is an advantage that the amount of data processed by the display control unit 190 can be reduced and a delay in display timing can be prevented.

  Subsequently, the display position control unit 165 performs display position fitting (step S20). Fitting is a function in which the user operates the control device 10 to manually adjust the display position of the AR content. This is executed if necessary, such as when the user desires to change the position where the AR content is displayed in step S19. Therefore, step S20 can be omitted, and it may be configured to set whether or not to perform the fitting of step S20. In step S20, the display position control unit 165 moves the display position of the AR content according to the operation of the operation unit 135 or the like. Here, the display position control unit 165 may change the display size of the AR content in accordance with the operation of the operation unit 135 or the like. Further, the adjustment amount of the position adjustment and the size adjustment executed in step S <b> 20 may be stored as setting data 121 in the storage unit 120.

Through the operations in steps S19 to S20, the user can visually recognize the AR content at a position overlapping the target or a position corresponding to the target.
Thereafter, the operation processing unit 164 determines whether or not there is an operation for instructing zoom display on the operation unit 135 or the like (step S21). When there is a corresponding operation (step S21; YES), the motion processing unit 164 detects the user's line of sight with the line-of-sight sensor 68 (step S22). Based on the detected line of sight, the motion processing unit 164 identifies the direction in which the user gazes in the display area, executes zoom display centered on the gaze direction (step S23), and proceeds to step S24. When it is determined that there is no operation for instructing zoom display (step S21; NO), the operation processing unit 164 proceeds to step S24.

FIG. 11 is a diagram illustrating a display example of the head-mounted display device 100, where (A) illustrates an example of reduced display, (B) illustrates an example of detecting a gaze direction, and (C) illustrates zoom display. An example of
The reduced display shown in FIG. 11A is an example of the operation performed by the operation processing unit 164 in response to the operation of the operation unit 135, and the outside scene image P (FIG. 10B) is reduced and displayed in the display area D1. To display. If the outside scene image P is an image in a wider range than the outside scene that the user sees through the image display unit 20, when the reduced display is performed, the user sees the outside scene in a wider range than the visual field VR. Can do.

In step S22, the display position control unit 165 detects the line of sight using the line-of-sight sensor 68, obtains the direction in which the user is gazing based on the detected line of sight, and the user views the outside scene that overlaps the display area D1. Calculate the gaze point you are gazing at. In FIG. 11B, the gazing point VPO is inside the display area D1, but if the line-of-sight sensor 68 is used, even when the gazing point VPO is outside the display area D1, it can be detected.
In the example of FIG. 11C, a predetermined range centered on the gazing point VPO is enlarged and displayed in the display area D1. Although part of the outside scene image P is enlarged and displayed in FIG. 11C, the AR image being displayed in the display area D1 may be enlarged and displayed.

  In step S24, the motion processing unit 164 determines whether or not the movement of the head-mounted display device 100 is detected (step S24). The motion processing unit 164 detects the movement of the head-mounted display device 100 using the 9-axis sensor 66 and / or the GPS 115. When it determines with not detecting the movement of the head mounted display apparatus 100 (step S24; NO), the operation | movement process part 164 continues the display of a guidance image, and performs determination of step S21 for every fixed time.

  When the movement of the head-mounted display device 100 is detected (step S24; YES), the operation processing unit 164 calculates the position of the target in the display area (step S25). This operation is the same as step S15. The display position control unit 165 determines whether or not the target position calculated in step S25 is within the display area of the image display unit 20 (step S26). The determination in step S26 can be performed in the same manner as in step S16.

When the display position control unit 165 determines that the target position is not within the display region (step S26; NO), the control unit 140 proceeds to step S17 and displays the guidance image for guiding the line of sight to the right optical image display unit 26. And it displays on the left optical image display part 28 (step S17).
When it is determined that the target position of the AR display is within the display area (step S26; YES), the display position control unit 165 calculates and updates the display position of the AR display, and operates based on the updated position. The processing unit 164 controls the display control unit 190 to display the AR content (step S27).

  Thereafter, the operation processing unit 164 determines whether or not a display end condition is satisfied (step S28). If the display end condition is not satisfied (step S28; NO), the process returns to step S21. If the display end condition is satisfied (step S28; YES), the display of the AR content is stopped. The display end condition is, for example, that the operation unit 135 has been instructed to end, or that the AR content display has been completed.

  In addition, when a movement is detected by step S24, the control part 140 is good also as a structure which updates AR display during a movement. In this case, the control unit 140 detects the target from the captured images of the right camera 61 and the left camera 62, specifies the target position, and obtains the display position of the AR display so as to overlap the target. Furthermore, the control unit 140 can detect the movement speed or movement amount of the target from the captured image, and can move the display position of the AR display in accordance with the movement of the target. Specifically, a plurality of captured images of the right camera 61 and the left camera 62 are acquired, a change in the relative position between the target and the background, that is, an object other than the target in the plurality of captured images is detected, and the moving speed or movement of the target Find the amount. Thereby, AR display can be performed so as to follow the moving object.

  Here, the control unit 140 controls the nine-axis sensor 66 so that the moving speed and the moving amount of the target position can be accurately obtained from the captured image even when the background object in the captured image moves. The detected value may be used. When the object moves, the user can think of moving the head so that the object is within the field of view. The control unit 140 detects the movement of the image display unit 20 from the detection value of the nine-axis sensor 66 when detecting the relative movement between the background and the target in the captured image. For example, when the moving direction of the target with respect to the background detected from the captured image matches or approximates the moving direction of the image display unit 20, the control unit 140 determines that the target is moving and displays the AR display. Adjust the position. For example, when the moving direction of the target with respect to the background detected from the captured image is different from the moving direction of the image display unit 20, the control unit 140 determines that the background is moving. In this case, since there is a possibility that the user feels uncomfortable by changing the display position of the AR display, the control unit 140 does not change the display position of the AR display.

  Further, the image or the like for AR display may be generated based on the detection result or the like by which the control unit 140 detects the object from the right camera 61 and the left camera 62, but the image data stored in the storage unit 120 or the like in advance. You may display based on. In addition, the control unit 140 may cut out a part of the captured image of the right camera 61 and / or the left camera 62 and display the AR. In this case, it is possible to reduce the load of processing for generating an image or the like for AR display, so that AR display can be performed at high speed.

  As described above, the head-mounted display device 100 according to the first embodiment to which the present invention is applied displays an image so that the head-mounted display device 100 transmits the outside scene and can be visually recognized together with the outside scene. An image display unit 20 is provided. In addition, an outside scene image including an outside scene that is visible through the image display unit 20 is acquired, and an object that is viewed through the image display unit 20 is recognized based on the acquired outside scene image. The control part 140 displayed on the image display part 20 is provided. Thereby, the information regarding the object can be displayed in correspondence with the position and size of the object in the outside scene visually recognized by the user. For this reason, inconsistency between the outside scene visually recognized by the user and the display image can be prevented, and effective display can be performed. Moreover, even if the area (display area D1) in which the virtual image can be displayed in the image display unit 20 is small, it is possible to perform display related to the object existing in the real space, and to reduce the restriction due to the size of the area where the virtual image can be displayed. It is possible to perform AR display over a wide real space.

In addition, when the target is outside the range where information is displayed on the image display unit 20, the control unit 140 performs guidance display for guiding the line of sight to the target by the image display unit 20. Thereby, it is possible to guide the line of sight so that the target position falls within the range in which information is displayed.
Further, the control unit 140 displays, on the image display unit 20, an image of the outside scene in a range that is visible through the image display unit 20 based on the acquired outside scene image. Thereby, the appearance of the image of the outside scene visually recognized by the user can be changed.

  The image display unit 20 is a head-mounted display unit that is worn on the user's head, and includes a line-of-sight sensor 68 that detects the user's line of sight. The control unit 140 changes the display of the image display unit 20 in accordance with the direction of the user's line of sight detected by the line-of-sight sensor 68. Thereby, the display image can be made to correspond to the user's line of sight, and inconsistency between the outside scene visually recognized by the user and the display image can be prevented more effectively.

  In addition, the control unit 140 displays a partial region of the outside scene image on the image display unit 20 and displays it on the image display unit 20 in the outside scene image corresponding to the direction of the user's line of sight detected by the line of sight detection unit. The position of the partial area is changed. Accordingly, since the outside scene image is displayed in correspondence with the user's line of sight, inconsistency between the outside scene visually recognized by the user and the display image can be prevented, and the outside scene image can be visually recognized by the user.

The head-mounted display device 100 includes a right camera 61 and a left camera 62 that capture a range including an outside scene that is visible through the image display unit 20. As described in the first aspect, the control unit 140 may acquire the captured images of the right camera 61 and the left camera 62 as an outside scene image. In this case, an outside scene image corresponding to the outside scene visually recognized by the user can be acquired and displayed without preparing an outside scene image in advance.
The control unit 140 may generate an outside scene image by acquiring and synthesizing a plurality of captured images captured by the right camera 61 and the left camera 62 that are a plurality of imaging units. In this case, an outside scene image having a preferred size and angle of view can be obtained using a plurality of captured images.

  Further, as described in the second aspect, the control unit 140 may generate an outside scene image by acquiring and combining a plurality of captured images captured by the right camera 61 and the left camera 62. In this case, it is not limited to the angle of view of the right camera 61 and the left camera 62, and an outside scene image having a preferable size and angle of view can be obtained.

The head-mounted display device 100 includes a communication unit 117 connected to an external device, and the control unit 140 acquires an outside scene image from the external device connected to the communication unit 117 as described in the third aspect. May be. In this case, the outside scene image can be acquired without storing the outside scene image in advance or providing a means for imaging.
The head-mounted display device 100 includes a GPS 115 that detects the current position, and the control unit 140 acquires an external scene image corresponding to the current position detected by the GPS 115 from an external device via the communication unit 117. May be. In this case, an outside scene image corresponding to the position of the head-mounted display device 100 can be acquired.

The head-mounted display device 100 is a display device that is worn on the user's head, and is half mirrors 261A and 262A serving as display areas that display images so that they can be seen through the outside scene and visible together with the outside scene. It has the image display part 20 which has. The control unit 140 included in the head-mounted display device 100 recognizes a target located outside the range that can be seen through the half mirrors 261A and 262A, and outputs information related to the target.
For example, when the target is viewed through the outside of the half mirrors 261A and 262A in the image display unit 20, the control unit 140 outputs information that guides the user to a state in which the target is visible. The information output by the control unit 140 is, for example, a guidance image D2 indicating the direction in which the AR display target is located. In this case, the method in which the control unit 140 outputs information is a method of displaying the guidance image D2 on the image display unit 20, but the user may be guided by indicating the target position by text. Further, a sound for guiding the user may be output from the right earphone 32 and / or the left earphone 34.

[Second Embodiment]
FIG. 12 is a flowchart illustrating the operation of the head-mounted display device 100 according to the second embodiment.
Since the configuration of the head-mounted display device 100 according to the second embodiment is the same as the configuration described with reference to FIGS. 1 to 4 in the first embodiment, illustration and description thereof are omitted.
In the second embodiment, the head-mounted display device 100 allows the user to view the target when the target of the AR content is outside the range that can be viewed through the image display unit 20. The guidance for is output.

  When the user wears the head-mounted display device 100 and the display of the AR content is instructed by the operation of the operation unit 135, the control unit 140 starts an operation related to the display (step S11).

  The control unit 140 acquires target information regarding the target position of the AR content display (step S31). The target information includes data for obtaining a relative position between the head-mounted display device 100 and the target. Specifically, data indicating the relative position between the head-mounted display device 100 and the target may be included, or data indicating the relative position between a predetermined target and the target may be included.

For example, the target information may include data that can identify the position of the target with a position detection system using GPS or WiFi. In this case, the control unit 140 detects the position of the image display unit 20 using the GPS 115 and the communication unit 117, and based on the detected position of the image display unit 20 and target information, the control unit 140 detects the position of the image display unit 20 and the target. The relative position of can be specified.
Further, the target of AR display is not limited to the entire object, and a part or surface specified in the object may be set as the target. In this case, data for specifying the position of the target with respect to the position of the object including the target can be included in the target information. For example, the target information can include data for specifying the entire object, such as CAD data of the object including the target, an external view of the entire object, and data for specifying the position of the target in the entire object.

  The controller 140 causes the image acquisition unit 161 to execute an outside scene image acquisition process (step S12). The outside scene image acquisition process can be executed in the first to third modes described above.

  The control unit 140 recognizes the target of AR display from the outside scene image acquired in step S12 and the target information acquired in step S31 by the position detection unit 162 (step S32). The target of AR display is an object in real space that is visually recognized by the user when displaying AR content, as described above. In step S32, the position detection unit 162 recognizes the target in the outside scene image acquired in step S12, as described with reference to step S13 (FIG. 5). When the target is not included in the outside scene image, the position detection unit 162 specifies the position of the target (the relative position between the head-mounted display device 100 and the target) using the target information acquired in step S31.

  When the target information includes data indicating the relative position between the target and the target, the position detection unit 162 detects the target from the outside scene image, and determines the relative position between the detected target and the head-mounted display device 100. Ask. Based on the calculated relative position and the data included in the target information, the position detection unit 162 determines the position of the target with respect to the head-mounted display device 100. In the process of detecting the target from the outside scene image, data used for the process of detecting the target image from the captured image is used. For example, the data includes the color, shape, and other information of the captured image when the target is captured. Includes a feature value indicating the feature. This data is included in the target information, for example. The position detection unit 162 performs processing for extracting an object image from the image data of the outside scene image, calculates the feature amount of the extracted object image, and compares the calculated feature amount with the feature amount included in the target information Collate. When the feature values are close or the same, the position detection unit 162 recognizes the image extracted from the outside scene image as the target image.

  When the target information acquired in step S31 includes data indicating the relative position of the target and the head-mounted display device 100, the position detection unit 162 specifies the target position without using the outside scene image. Also good. This process may be executed when the position detection unit 162 cannot detect both the target and the target from the outside scene image.

The position detection unit 162 determines whether or not the target position recognized in step S32 is a position that can be visually recognized by the user (step S33). Since the user wearing the image display unit 20 sees the outside scene (real space) through the image display unit 20, the position detection unit 162 is within a range that the user can see through the image display unit 20. Determine whether there is a target.
Here, when there is an object in the range which can be visually recognized by the user (Step S33; YES), the position detection unit 162 proceeds to Step S14. The subsequent operation is as described in the first embodiment.

On the other hand, when there is no object in the range which can be visually recognized by the user (step S33; NO), the operation processing unit 164 performs a guiding operation for guiding the user (step S34).
When the control unit 140 proceeds to step S34, there is no target in the range that the user can see through the image display unit 20, that is, the user's field of view. For this reason, in order for the user to see the object, the user himself / herself needs to move (forward or backward) or change the direction of the head. The control unit 140 guides or notifies the operation to be performed by the user in order to make the target enter the user's field of view through the guidance operation in step S34. That is, the guidance operation is an operation of the head-mounted display device 100 for guiding the user to a state where the user can visually recognize the target. Specifically, the guidance operation includes image display, text display, and audio output, and these may be executed in combination.

FIG. 13 is a diagram illustrating a specific example of the guiding operation, where (A) illustrates an example of the guiding operation using an image, and (B) illustrates an example of the guiding operation using an image and text.
In the example of FIG. 13A, the guidance display D3 is displayed in the display area D1 in the visual field VR of the user. As described above, the display area D1 is an area in which the virtual image can be visually recognized by the user's eyes by the half mirrors 261A and 262A.

The guidance display D3 illustrated in FIG. 13A indicates a specific direction with an arrow. The direction indicated by the guidance display D3 is, for example, (1) the direction in which the object is located, (2) the direction in which the user should move to make the object visible, or (3) the state in which the object can be visually recognized. This is the direction in which the user should turn the face (head). The meaning or purpose of the direction indicated by the guidance display D3 (for example, (1) to (3) above) is referred to as an attribute of the guidance image.
The attribute of the guidance display D3 may be set in advance as to which of the above (1) to (3), or may be distinguishable depending on the display mode of the guidance display D3.

For example, the control unit 140 may be configured to be able to switch the display mode of the guidance display D3. Specifically, the display that blinks the guidance display D3 and the display that does not blink may be switched, or the display color of the guidance display D3 may be selected and switched from a plurality of colors, and the display size of the guidance display D3 may be changed. The display size may be selected from a plurality of stages.
In this case, the control unit 140 can select the display mode of the guidance display D3 corresponding to the attribute of the guidance display D3. For example, a display mode may be set in association with each attribute of the guidance display D3, and the control unit 140 may determine the display mode of the guidance display D3 according to this setting.

  FIG. 13B shows an example in which the control unit 140 displays a guidance display D4 including text in addition to the guidance display D3. The guidance display D4 is displayed in the display area D1 together with the guidance display D3, and provides information by text. The content of the guidance display D4 is arbitrary, but may be content related to information provided by the guidance display D3. For example, additional information about the direction indicated by the guidance display D3 may be provided by the guidance display D4. Further, the information content of the guidance display D4 may be the same information as the guidance display D3, or may be an explanation of the information of the guidance display D3. The contents of the guidance display D4 may be information that can be understood without looking at the guidance display D3.

  The guidance display D4 may be content that prompts the user to move. For example, the text may be “Please go forward one step” or “Go forward 50 cm and approach the target bicycle”. Moreover, the text which promotes a movement and the text explaining the objective of movement and the operation | movement after a movement may be included. For example, “Go forward 50 cm and approach the target bicycle. The AR display tool overlaps the bicycle screw”. Also, it may be content that prompts the user's action using the arrow of the guidance display D3, for example, “There is a tool box in the direction of the yellow arrow on the left. Please take out the wrench from the tool box.” It can be.

The control unit 140 can guide the user so that the target of the AR display overlaps the display area D1 with the guidance displays D3, D4, and the like illustrated in FIGS. Further, the control unit 140 may guide the AR display target so as to overlap the center of the display area D1.
In addition, the control unit 140 may change the display mode of the display for guidance, such as the guidance displays D3 and D4, according to the relative position between the display area D1 and the AR display target. That is, the display mode may be changed according to the relative position between the target and the real space that the user sees through the display area D1. Specifically, the distance between the real space that the user sees through the display area D1 and the target is classified into a plurality of stages, and the display size of the guidance display D3 and the guidance display D4 is increased as the distance is longer. The shorter the is, the smaller the display size may be. Further, when the distance is long, the display position of the guidance display D3 or the guidance display D4 is set to a position closer to the center of the display area D1, and the shorter the distance, the more the display position is moved to the side where the AR display target is located. Good.

Moreover, the control part 140 may combine not only guidance display D3 and D4 but the guidance by an audio | voice. It is possible to use a language preset by the user with respect to the control device 10 so as to convey information related to guidance and guidance to the user.
The specific content provided by voice can be the same as the content of the guidance display D4, for example. In addition, instructions such as a standing position, a line of sight, a head orientation, a hand position, and a holding method may be output to the user by guidance display including voice and voice. For example, when the work procedure is guided according to the work scenario by AR display, information for guiding the work state may be provided by text display or voice. The scenario includes data including at least one of an image, text, and sound, a display order is set for the image and / or text, and a sound output timing is set in association with the display order of the image and / or text. . When there are a plurality of objects (images, texts, sounds, etc.) to be output, the objects may be switched and output. The output timing and switching timing of the object may be set in association with the passage of time from the start of scenario playback, or may be triggered by an input to the control device 10. Further, the control unit 140 may detect the trigger by analyzing the captured images of the right camera 61 and the left camera 62. In this case, the feature amount of the image detected as the trigger is included in the scenario. It is also possible to include guidance images such as guidance displays D3 and D4, text, or voice output by the control unit 140 as part of the scenario. Further, the scenario data can include data such as the feature amount of the AR display target.
The control unit 140 may output guidance text and voice such as guidance displays D3 and D4 corresponding to the target position recognized in step S32 together with the text and voice of the scenario.

  The control unit 140 may generate the text to be output as in the guidance display D4 or the content of the audio to be output based on the target information acquired in step S31 and the recognition result in step S32 each time. In addition, the control unit 140 selects a text from data stored in the storage unit 120 or the like or data included in data acquired from the external device OA, displays the selected text, or converts it into speech and outputs it. May be.

  The guidance display D3, D4, AR display, or the like displayed by the control unit 140 in the guidance display may be displayed using image data stored in advance in the storage unit 120 or the like. In this case, for example, when the relative position between the target and the user greatly changes in a short time, the display can be performed promptly. Further, the control unit 140 may cut out a part of the captured images of the right camera 61 and the left camera 62 and perform AR display.

  Moreover, the structure which can switch the display mode of the guidance display D4 may be sufficient as the control part 140. FIG. For example, the display that blinks the guidance display D4 and the display that does not blink may be switched, or the display color of the guidance display D4 may be selected and switched from a plurality of colors, and a plurality of fonts and character sizes of the guidance display D4 may be selected. You may select from the font and character size.

  The control unit 140 may switch the display mode of the guidance displays D3 and D4 according to the attribute of the guidance display D3 and the content of the guidance display D4 as described above. When the attribute of the guidance display D3 is clear from the content of the guidance display D4, the control unit 140 may not adjust the display mode corresponding to the attribute of the guidance display D3.

  The contents of the guidance operation executed by the control unit 140 are not limited to the examples shown in FIGS. For example, a notification marker may be displayed on the periphery of the display area D1. The marker may be an arrow as in the guidance display D3, a geometric figure such as a circle or a polygon, or an illustration. The marker may be accompanied by text, but it is preferable that the marker be displayed in a form without text so that the user can intuitively recognize it.

FIG. 14 shows an example of the guide operation by the marker as another specific example of the guide operation of the head-mounted display device 100.
In the example of FIG. 14, markers D6 and D10 are displayed on the periphery of the display area D1 in the visual field VR of the user. In this example, the control unit 140 displays caution information that calls attention to the user, caution information that particularly requires attention or warning about matters related to safety, and work information related to work according to the scenario. .
For example, the marker D10 is a marker indicating warning information regarding the left direction in the figure, and is displayed in a color (for example, red) set as a display color of the warning information. In synchronism with the timing of starting the display of the marker D10, the control unit 140 converts the text such as “Do not move further to the left. Do not hit the wall” to sound and convert the right earphone 32 and the left earphone. 34 may be output. The marker D6 is displayed in a color (for example, yellow) set as the display color of the caution information. In synchronization with the start timing of the display of the marker D6, the control unit 140 converts the text such as “You can see a progress guide arrow to the left front. You may output from the earphone 34.

  The marker D7 indicates the position of an AR display target, and is displayed in a color (for example, green) set as a display color of work information. In synchronization with the start of displaying the marker D6, the control unit 140 utters a text such as “Go ahead 30 meters to the left. You can see the target package. The target is a green marker.” You may convert and output from the right earphone 32 and the left earphone 34. FIG. Here, when the object itself is visible through the display area D1, the control unit 140 performs AR display on the display area D1. When the distance from the head-mounted display device 100 to the target is longer than the set distance, the control unit 140 cannot directly recognize the target due to a wall or an obstacle and cannot detect the target from the captured image. Is displayed. Thereby, it is possible to notify the user of an object that is in a position overlapping with the display area D1 and cannot be directly visually recognized or difficult to visually recognize directly.

  Markers having directions, such as the markers D6 and D10, are stored in advance as an image or AR object in which the pointing direction is specified. That is, when displaying the marker in which the pointing direction is specified, the control unit 140 reads and displays the marker image corresponding to the display direction. These images can be referred to as AR display objects having vectors. The direction may be associated with a character (text) as well as the arrow image illustrated in the figure. In this case, compared with the case where the marker image or character is generated, the control unit 140 has an advantage that the AR display can be performed quickly.

Furthermore, in the head-mounted display device 100, the marker displayed in the display area D1 may be configured to be usable as an operation target that receives user input.
FIG. 15 is a diagram illustrating another specific example of the guiding operation of the head-mounted display device 100, (A) illustrates an example in which a marker is an operation target, (B) illustrates an operation example with respect to the marker, (C) shows another example of the operation for the marker.

In FIG. 15A, a marker D11 is displayed in the display area D1. When the control unit 140 detects an operation on the marker D11, for example, the control unit 140 displays and outputs text information D12. The text information D12 is a weather forecast, and is acquired by the control unit 140 using the communication unit 117 or the like.
In FIG. 15B, a marker D6 and a marker D13 are displayed in the display area D1. The marker D6 is displayed, for example, as caution information as described with reference to FIG. On the other hand, the marker D13 is an operation target, and when the marker D13 is operated, for example, a pull-down menu D14 is displayed.

The operation of selecting the marker D13 is performed, for example, when the user speaks an instruction voice and the control unit 140 detects the voice from the voice collected by the microphone 63. Moreover, it is good also as a structure which a user can select a marker by moving a head. The control unit 140 detects the movement of the head of the user wearing the image display unit 20 by the 9-axis sensor 66 and selects a marker displayed at a position corresponding to the detected movement. In the example of FIG. 15B, the marker D13 displayed on the upper right of the display area D1 is selected by the user moving the head to the upper right. Further, an operation for selecting one item from the items displayed on the pull-down menu D14 (four items in the example of FIG. 15B) can also be performed by voice or head movement. When the head-mounted display device 100 includes a sensor that detects the movement of the user's body, not only the head but also the movement of the user's body may be detected as an operation.
The marker to be operated such as the marker D13 can be displayed simultaneously with the marker indicating the warning information, the caution information, or the work information. For example, in the example of FIG. 15B, the marker is displayed simultaneously with the marker D6. In this case, the display color of the marker to be operated is preferably set to a display color different from the marker indicating the warning information, the attention information, or the work information. In this case, the user can easily identify the marker of the warning information, the attention information, or the work information and the marker to be operated by the color of the marker.

  In addition, the control unit 140 may be configured to display a plurality of markers to be investigated in the display area D1. In this case, the user can select one marker by voice from a plurality of displayed markers. The selection method by voice only needs to utter words that specify the marker attributes such as the color, shape, and position of the markers. For this reason, it is preferable that the plurality of markers displayed simultaneously in the display area D1 have different attributes. . For example, if the user speaks a word that specifies the color of the marker in a state where markers of different colors such as red, blue, and yellow are displayed, the marker corresponding to the spoken word is selected.

Further, FIG. 15C shows an example in which a direction instruction operation is performed using a plurality of markers D21, D22, D23, and D24. In the example of FIG. 15C, a key image D20 corresponding to the up / down / left / right direction instruction is displayed, and markers D21, D22, D23, and D24 are arranged at positions corresponding to the instruction direction. Each of the markers D21, D22, D23, and D24 has an attribute that can be visually discriminated. For example, the display colors are different colors such as red, blue, and yellow.
When the user utters a word that specifies a marker attribute by voice (for example, a word such as red, blue, or yellow that specifies a display color), the control unit 140 selects the corresponding marker from the markers D21, D22, D23, and D24. Select. The control unit 140 determines that an operation for instructing the direction has been performed in the direction corresponding to the selected marker. That is, by selecting any of the markers D21, D22, D23, and D24, it is possible to instruct the direction corresponding to the selected marker. Thereby, it becomes possible to perform a voice-free operation in a hands-free manner so that the hardware of the cross key such as the key image D20 is substantially operated.

  For example, when a specific direction (for example, right direction) is instructed by the key image D20, the control unit 140 switches the display mode of the AR display to the display for near-position work. The image display unit 20 is configured to form a virtual image on the user's eyes by half mirrors 261A and 262A that can transmit the outside scene. For this reason, the size of the object displayed in the display area D1 depends on the distance to the outside scene that is visible through the image display unit 20 for the user wearing the image display unit 20 due to human visual characteristics. Looks like size. In the head-mounted display device 100, the default (initial) setting is displayed in accordance with a case where the user pays attention to a position 4 m away from the image display unit 20 in the real space.

  Specifically, the control unit 140 adjusts the display position of the AR image (including an object such as a marker) in the display area D1 to the convergence angle of both eyes when the user pays attention to the position 4 m ahead. And The convergence angle refers to an angle formed by a straight line connecting the right eye RE and the target OB and a straight line connecting the left eye LE and the target OB when paying attention to the target OB shown in FIG. The magnitude of the convergence angle changes corresponding to the distance (distance to the target OB) that the user gazes at. When the distance that the user gazes is shorter than 4 m, the convergence angle of both eyes is larger than the convergence angle assumed by the default setting. In this case, the control unit 140 moves the image display position on the half mirrors 261A and 262A toward the center of the image display unit 20, that is, the nose side. Further, when the distance that the user gazes is longer than 4 m, the convergence angle of both eyes is smaller than the convergence angle assumed by the default setting. In this case, the control unit 140 moves the image display position on the half mirrors 261A and 262A to a position closer to the end of the image display unit 20, that is, the ear side. The operation of the key image D20 in FIG. 15C can be made to correspond to the adjustment of the convergence angle. When the specific direction of the key image D20 is instructed, the control unit 140 moves the display position in the display area D1 to the nose side so that the convergence angle is increased. Thereby, a display suitable for the case where the user performs work while paying attention to a close position such as a hand can be performed. Further, when the opposite side of the specific direction is instructed in the key image D20, the control unit 140 moves the display position in the display area D1 to the ear side so that the convergence angle becomes small. Thereby, a display suitable for the case where the user performs work while gazing at a distant place can be performed. In this example, the user can adjust the display position suitable for the distance to the work target by a hands-free operation.

[Third Embodiment]
FIG. 16 is a flowchart showing the operation of the head-mounted display device 100 according to the third embodiment.
Since the configuration of the head-mounted display device 100 of the third embodiment is the same as the configuration described with reference to FIGS. 1 to 4 in the first embodiment, illustration and description are omitted.
In the third embodiment, the head-mounted display device 100 detects that the user's line of sight has moved while the AR content is being displayed, and changes the display position in accordance with the movement of the line of sight.

When the user wears the head-mounted display device 100 and the display of the AR content is instructed by the operation of the operation unit 135, the control unit 140 starts an operation related to the display (step S11).
The controller 140 causes the image acquisition unit 161 to execute an outside scene image acquisition process (step S12). The outside scene image acquisition process can be executed in the first to third modes described above.

  After the image acquisition unit 161 acquires the outside scene image by the outside scene image acquisition process, the operation processing unit 164 detects the user's line of sight using the line-of-sight sensor 68 (step S41). Based on the detected line of sight, the motion processing unit 164 identifies a direction in which the user is gazing in the display area, and performs a process of cutting out a portion corresponding to the gazing direction from the outside scene image (step S42).

In step S <b> 41, the motion processing unit 164 detects the user's line of sight using the line-of-sight sensor 68, and specifies the direction in which the user is gazing with the image display unit 20 as a reference. The reference in this case can be the center position of the right optical image display unit 26 and the left optical image display unit 28 of the image display unit 20, the center of the half mirrors 261A, 262A, or the like.
In step S <b> 42, the motion processing unit 164 identifies the correspondence between the outside scene image acquired by the image acquisition unit 161 in step S <b> 12 and the actual outside scene that the user views through the image display unit 20. The outside scene image acquired by the image acquisition unit 161 includes an actual outside scene visually recognized by the user, and additionally includes an outside scene image in a range that cannot be visually recognized by the user. When the outside scene image acquisition processing in step S12 is executed in the first mode (FIG. 6) or the second mode (FIGS. 7 and 8), the motion processing unit 164 displays the images of the right camera 61 and the left camera 62. Based on the corner, the relative position between the outside scene visually recognized by the user and the outside scene image is obtained. When the outside scene image acquisition process is executed in the third mode (FIG. 9), the operation processing unit 164 is based on the position and direction of the image display unit 20 detected when receiving the outside scene image from the external device. The relative position between the outside scene visually recognized by the user and the outside scene image is obtained. Based on this relative position, the motion processing unit 164 identifies a position that overlaps the user's line-of-sight direction in the outside scene image, that is, a gaze point.

FIG. 17 is a diagram showing a display example in the third embodiment, where (A) shows a user's visual field VR, (B) shows an example of an outside scene image, and (C) shows a table corresponding to the line-of-sight direction. An example is shown.
FIGS. 17A and 17C show examples of the visual field VR of the user's right eye, and the visual field of the user's left eye is the same as or symmetrical with FIGS. 17A and 17C. Illustration is omitted. A display area D1 is shown as an area corresponding to the half mirror 261A in the visual field VR.
In the example of FIG. 17A, a bicycle is visible in the field of view VR, but this bicycle is a real bicycle placed outside the image display unit 20. That is, the user views the bicycle in the outside scene through the right optical image display unit 26.

  As illustrated in FIG. 17B, the motion processing unit 164 identifies a position in the outside scene image corresponding to the user's line-of-sight direction, and sets this position as the gazing point VPO. Further, the operation processing unit 164 performs processing for cutting out a rectangular portion P3 centered on the gazing point VPO from the outside scene image. The size of the portion P3 is set in advance according to the display resolution of the right LCD 241 and the left LCD 242.

The display position control unit 165 determines the display position and size of the image cut out by the operation processing unit 164 in step S42 (step S43). Subsequently, the operation processing unit 164 adjusts the display brightness, and displays the outside scene image cut out in step S42 according to the position and size determined in step S43 (step S44).
A display example of the outside scene image is shown in FIG. In the visual field VR, an image centered on the gazing point VPO is displayed at the position of the half mirror 261A. The display image of the half mirror 261A is a display in which the display position of a part of the outside scene is shifted, and the user can visually recognize the outside scene at the position he / she wants to see in a state where the outside scene is shifted to an easily viewable position.
When the display brightness is adjusted, the visibility between the image cut out from the outside scene image and the actual outside scene can be adjusted. As the amount of image light incident on the user's eyes by the half mirrors 261A and 262A increases, the amount of external light transmitted through the half mirrors 261A and 262A becomes relatively small, so that the actual outside scene becomes difficult to see. In other words, when the brightness of the display image of the image display unit 20 is high, the visibility of the display image is high and the visibility of the actual outside scene is low. On the contrary, if the brightness of the display image of the image display unit 20 is lowered, the visibility of the display image is lowered and the visibility of the actual outside scene is enhanced. Since the image displayed in the third embodiment is an outside scene image, it is similar to an actual outside scene. For this reason, if the brightness is increased, there is an advantage that it is difficult to confuse the displayed outside scene image with the actual outside scene. On the other hand, if the brightness of the display image is lowered, there is an advantage that the image can be displayed while ensuring the visibility of the outside scene.

After starting the display of the outside scene image at step S44, the motion processing unit 164 detects the line of sight by the line-of-sight sensor 68 and determines whether or not the line of sight has moved (step S45). That is, the operation processing unit 164 determines whether or not the user's gaze direction has changed from a previously detected gaze direction exceeding a preset threshold value.
When it is determined that there is no movement of the line of sight (step S45; NO), the operation processing unit 164 repeatedly performs the line-of-sight detection by the line-of-sight sensor 68 and the determination of step S45 at predetermined time intervals. If it is determined that the line of sight has moved (step S45; YES), the motion processing unit 164 identifies the position of the point of gaze based on the line of sight after the movement, and the portion corresponding to the identified point of gaze is determined in step S12. It cuts out from the outside scene image acquired by (step S46).

  The display position control unit 165 determines the display position and size of the outside scene image cut out by the motion processing unit 164 in step S46 (step S47). The motion processing unit 164 updates the displayed image to the outside scene image cut out in step S46 according to the position and size determined in step S47 (step S48).

  Thereafter, the operation processing unit 164 determines whether or not the display end condition is satisfied (step S49). If the display end condition is not satisfied (step S49; NO), the process returns to step S45. If the display end condition is satisfied (step S49; YES), the display of the AR content is stopped. The display end condition is, for example, that the operation unit 135 has been instructed to end, or that the AR content display has been completed.

As described above, in the third embodiment, the control unit 140 of the head-mounted display device 100 displays a partial area of the outside scene image acquired by the image acquisition unit 161 on the image display unit 20 and is detected by the line-of-sight sensor 68. The position of the partial area displayed on the image display unit 20 in the outside scene image is changed corresponding to the direction of the user's line of sight. For this reason, since the outside scene image is displayed in correspondence with the line of sight of the user, inconsistency between the outside scene visually recognized by the user and the display image can be prevented, and the outside scene image can be visually recognized by the user. In addition, since the display of the outside scene image changes so as to follow the change in the user's line of sight, it appears to the user that the outside scene moves according to the line of sight. Further, when the outside scene image includes a range larger than the user's field of view, it seems to the user that the visible range is enlarged. In this way, by displaying the pseudo outside scene in accordance with the user's line of sight and changing the position of the pseudo outside scene, it is possible to provide a new operational feeling related to the visibility of the outside scene.
In addition, the operation processing unit 164 can adjust the display brightness of the outside scene image displayed on the image display unit 20 so that the display image can be visually recognized more clearly than the actual outside scene. Accordingly, the actual outside scene and the outside scene image are not confused, the user can be focused on the outside scene image, and the effect of displaying the pseudo outside scene can be made more remarkable.

[Fourth Embodiment]
FIG. 18 is a functional block diagram of each part constituting the head-mounted display device 100A according to the fourth embodiment.
The head-mounted display device 100A has the same configuration as the head-mounted display device 100 of the first embodiment, except that the head-mounted display device 100A includes a right transparent LCD 271 and a left transparent LCD 272. About this common structural part, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  Specifically, the head-mounted display device 100A includes a right display driving unit 22A having a right transparent LCD 271 and a left display driving unit 24A having a left transparent LCD 272 in the image display unit 20B. The right display driving unit 22A is configured such that the right display driving unit 22 is provided with a right transparent LCD 271 configured by a transmissive liquid crystal display panel, and the right transparent LCD 271 is driven by the right LCD control unit 211. Further, the left display driving unit 24A is configured such that the left display driving unit 24 is provided with a left transparent LCD 272 configured by a transmissive liquid crystal display panel, and the left transparent LCD 272 is driven by the left LCD control unit 212.

  The image data displayed by the right LCD control unit 211 driving the right transparent LCD 271 and the image data displayed by the left LCD control unit 212 driving the left transparent LCD 272 are transmitted from the image processing unit 160 to the transmission unit 51. And transmitted via the receiving unit 53.

FIG. 19 is a principal plan view showing the configuration of the optical system provided in the image display unit 20B, and illustrates the left eye LE and the right eye RE of the user for explanation.
As shown in FIG. 19, the right transparent LCD 271 is disposed between the right light guide plate 261 and the light control plate 20A. The right transparent LCD 271 is disposed so as to cover the front side of the image display unit 20B, which is the side of the right light guide plate 261 opposite to the user's eye side. The right transparent LCD 271 has light transmittance, and further transmits the external light OL transmitted through the light control plate 20A to the right light guide plate 261 side.
The left transparent LCD 272 is disposed between the left light guide plate 262 and the light control plate 20A. That is, the left transparent LCD 272 is disposed so as to cover the front side of the image display unit 20B that is the opposite side of the left light guide plate 262 from the user's eye side. The left transparent LCD 272 is light transmissive, and further transmits external light OL transmitted through the light control plate 20A to the left light guide plate 262 side.

  The light transparency of the right transparent LCD 271 and the left transparent LCD 272 is not limited, but it is preferable to transmit the external light OL with a light amount that allows the right transparent LCD 271 and the left transparent LCD 272 to be visually recognized. Further, it is not required that the outside scene can be seen through the right transparent LCD 271 and the left transparent LCD 272.

FIG. 20 is a flowchart showing the operation of the head-mounted display device 100A. FIG. 21 is a diagram showing a display example of the head-mounted display device 100A, where (A) shows an example of display on the right transparent LCD 271 and the left transparent LCD 272, and (B) is shown by the half mirrors 261A and 262A. An example of display is shown. (C) shows a state in which the display of the right light guide plate 261, the left light guide plate 262, the right transparent LCD 271 and the left transparent LCD 272 are overlapped.
21A, 21B, and 21C show examples of the right transparent LCD 271 and the visual field VR corresponding to the right eye of the user. Since the display example corresponding to the left eye of the user is the same display as that shown in FIGS. 21A to 21C, or the display content is symmetrical and the display content is the same, the illustration is omitted.

  When the user wears the head-mounted display device 100A (step S61) and the display of the AR content is instructed by the operation of the operation unit 135, the operation processing unit 164 executes a process of acquiring an outside scene image ( Step S62). The operation in step S62 is the same as the operation shown in step S12 in FIG. 5 of the first embodiment described above.

  The motion processing unit 164 detects the user's line of sight with the line-of-sight sensor 68 (step S63). Based on the detected line of sight, the operation processing unit 164 identifies the direction in which the user is gazing in the display area, and performs a process of cutting out a portion corresponding to the gazing direction from the outside scene image (step S64).

  In step S <b> 64, the motion processing unit 164 detects the user's line of sight using the line-of-sight sensor 68, and specifies the direction in which the user is gazing with the image display unit 20 as a reference. The reference in this case can be the center position of the right optical image display unit 26 and the left optical image display unit 28 of the image display unit 20, the center of the half mirrors 261A, 262A, or the like. Then, the motion processing unit 164 associates the outside scene image with the user's line-of-sight direction, and identifies a position that overlaps the user's line-of-sight direction in the outside scene image, that is, a gaze point.

  The motion processing unit 164 displays an image cut out from the outside scene image with the gazing point as a reference by the right display driving unit 22A and the left display driving unit 24A (step S65), and then the outside scene image is displayed on the right transparent LCD 271 and the left transparent image. The image is displayed on the LCD 272 (step S66).

  In FIG. 21A, the display area of the right transparent LCD 271 is denoted by reference numeral D5. An outside scene image is displayed in the display area D5. The outside scene image includes an outside scene that can be visually recognized by a user wearing the image display unit 20 when the user does not wear the image display unit 20 at the same position and orientation. Since the head-mounted display device 100A includes the right transparent LCD 271 and the left transparent LCD 272, there is a possibility that the external light OL incident on the user's eyes does not have an amount of light that can clearly see the outside scene. Therefore, an outside scene image is displayed on the right transparent LCDs 271 and 272 so that the user can recognize the outside scene. However, in the display area D5, it is desirable not to display at a position overlapping the display area D1 shown in FIG. Since the half mirrors 261A and 262A transmit the image light of the right transparent LCD 271 and the left transparent LCD 272, in order to ensure the visibility of the display images of the half mirrors 261A and 262A, the display images of the right transparent LCD 271 and the left transparent LCD 272 are displayed. A range that avoids the half mirrors 261A and 262A is preferable. The display area D5 in FIG. 21A is configured not to be displayed by the half mirror 261A.

As shown in FIG. 21B, an image cut out from the outside scene image according to the user's line of sight is displayed in the display area D1. The entire outside scene image shown in FIG. 21A is displayed in the display area D5 corresponding to the right transparent LCD 271. The image displayed in the display area D5 and the image displayed in the display area D1 overlap with the user's eyes and are visually recognized as shown in FIG.
The right transparent LCD 271 and the left transparent LCD 272 may have a configuration that does not have optimized optical characteristics like the right light guide plate 261 and the left light guide plate 262, and an image displayed in the display area D5 is displayed in the display area D1. Compared to the above, there is a possibility that an unclear image appears. However, as shown in FIG. 21C, the user sees a clear outside scene image corresponding to the line-of-sight direction in the display area D1, and the entire outside scene image is displayed in the display area D5 as the background. . For this reason, since the user can visually recognize the entire outside scene image and a part in the line-of-sight direction, the user who cannot directly see the outside scene can be shown a pseudo outside scene.

The motion processing unit 164 detects the line of sight by the line-of-sight sensor 68 and determines whether or not the line of sight has moved (step S67). That is, the operation processing unit 164 determines whether or not the user's gaze direction has changed from a previously detected gaze direction exceeding a preset threshold value. When it is determined that there is no movement of the line of sight (step S67; NO), the operation processing unit 164 repeatedly executes the line-of-sight detection by the line-of-sight sensor 68 and the determination in step S67 every predetermined time. If it is determined that the line of sight has moved (step S67; YES), the operation processing unit 164 identifies the position of the gazing point based on the line of sight after movement, and the portion corresponding to the identified gazing point is determined in step S62. It cuts out from the outside scene image acquired by (step S68).
The display position control unit 165 updates the image being displayed by the half mirrors 261A and 262A to the outside scene image cut out in step S68 (step S69).

  Thereafter, the operation processing unit 164 determines whether or not a display end condition is satisfied (step S70). If the display end condition is not satisfied (step S70; NO), the process returns to step S67. If the display end condition is satisfied (step S70; YES), the display of the AR content is stopped. The display end condition is, for example, that the operation unit 135 has been instructed to end, or that the AR content display has been completed.

  As described above, the head-mounted display device 100A according to the fourth embodiment performs a combination of the display on the right transparent LCD 271 and the left transparent LCD 272 and the display on the right light guide plate 261 and the left light guide plate 262. The entire AR content (for example, an outside scene image) is displayed on the right transparent LCD 271 and the left transparent LCD 272, and an image obtained by cutting out a part of the AR content is displayed on the right light guide plate 261 and the left light guide plate 262. Thereby, effective AR display can be performed by combining two displays. Further, the images displayed on the right light guide plate 261 and the left light guide plate 262 may be images cut out from the AR content corresponding to the user's line-of-sight direction. In addition, an image cut out from the AR content may be enlarged or reduced. Furthermore, when the AR content is a captured image of the right camera 61 and the left camera 62 or an outside scene image generated using the captured image, the user can visually recognize the outside scene.

  In addition, this invention is not restricted to the structure of said each embodiment, In the range which does not deviate from the summary, it can be implemented in a various aspect.

  In each of the above embodiments, the right optical image display unit 26 and the left optical image display unit 28 transmit external light in a range in which the user can see the outside scene through the display unit, and the right optical image display unit 26 and The configuration in which the user visually recognizes the outside scene by allowing the outside light transmitted through the left optical image display unit 28 to enter the user's eyes is illustrated. In this configuration, the range in which the user can see through the display unit has been described as the range occupied by the right optical image display unit 26 and the left optical image display unit 28 in the user's field of view (field of view). It is not limited to this. For example, even in a region that overlaps or is included in the right optical image display unit 26 and the left optical image display unit 28 in the user's field of view, a range that cannot be visually recognized by components or accessories such as a frame provided in the image display unit 20 Alternatively, if there is a region, it may be excluded from the “visible range”. In addition, data indicating the relative positional relationship between the portion that transmits the outside scene in the right optical image display unit 26 and the left optical image display unit 28 and the visual field of the user may be stored in the storage unit 120 or the like in advance. Good. In this case, the range visible to the user can be a range defined by the data stored in the storage unit 120. Further, the right optical image display unit 26 and the left optical image display unit 28 transmit the outside scene, the user's field of view, and the right optical image display unit 26 and the left optical image display unit 28 display images. The data indicating the relative positional relationship may be stored in advance in the storage unit 120 or the like. This data may include data specifying the size of the display area and the portion that transmits the outside scene in the right optical image display unit 26 and the left optical image display unit 28. In addition, the data may be generated by the control unit 140 and stored in the storage unit 120 by performing calibration in a state where the user wears the image display unit 20. In this calibration, the control unit 140 displays a calibration image on the right optical image display unit 26 and the left optical image display unit 28, and the positional relationship between the position where the calibration image can be visually recognized and the user's visual field. The user may perform input or operation, and the control unit 140 may detect the operation and generate the data. In this case, for example, an image in which a plurality of geometric figures or the like are arranged can be used as the calibration image.

  Further, the configuration in which the user visually recognizes the outside scene through the display unit is not limited to the configuration in which the right optical image display unit 26 and the left optical image display unit 28 transmit external light. For example, the present invention can be applied to a display device that displays an image in a state where an outside scene cannot be visually recognized. Specifically, a captured image of the right camera 61 and / or the left camera 62, an image or CG generated based on the captured image, a video based on video data stored in advance or video data input from the outside, and the like are displayed. The present invention can be applied to a display device. This type of display device can include a so-called closed display device in which an outside scene cannot be visually recognized. Of course, a display device that displays video data or an analog video signal input from the outside without performing processing such as AR display, MR display, or VR display is also included as an application target of the present invention.

  Further, for example, instead of the image display units 20 and 20B, an image display unit of another method such as an image display unit worn like a hat may be adopted, and an image corresponding to the left eye of the user is used. And a display unit that displays an image corresponding to the right eye of the user. The display device of the present invention may be configured as a head mounted display mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet. In this case, the part for positioning the position with respect to the user's body and the part positioned with respect to the part can be used as the mounting portion.

Furthermore, in the above embodiment, the image display units 20 and 20B and the control device 10 are separated and connected by way of the connection unit 40. However, the control device 10 and the image display unit 20 It is also possible to adopt a configuration in which is integrally formed and is mounted on the user's head.
As the control device 10, a notebook computer, a tablet computer, or a desktop computer may be used. The control device 10 may be a game machine, a portable phone, a portable electronic device including a smartphone or a portable media player, other dedicated devices, or the like. The control device 10 may be configured separately from the image display units 20 and 20B, and various signals may be transmitted and received between the control device 10 and the image display units 20 and 20B by wireless communication.

  For example, as a configuration for generating image light in the image display units 20 and 20B, a configuration including an organic EL (Organic Electro-Luminescence) display and an organic EL control unit may be used. Further, as a configuration for generating image light, LCOS (Liquid crystal on silicon, LCoS is a registered trademark), a digital micromirror device, or the like can be used.

The “display unit” in the present invention corresponds to a configuration that emits image light. In the following description, the fact that the head-mounted display device 100 emits image light is referred to as “display”.
In each of the above embodiments, image light is generated by the left and right image light generation units described with reference to FIG. 4, and the right optical image display unit 26 and the left optical image display unit 28 shown in FIG. A configuration in which image light is irradiated toward each of the eyes and the left eye and the image light is incident on each of the right eye and the left eye of the user is illustrated. The configuration of the “display unit” is not limited to this. That is, as long as it emits image light, it is not limited to the structure of FIG.2 and FIG.4. For example, in the configuration of the present embodiment, image light is emitted toward the user's eyes by the “right light guide” and “left light guide” having the half mirrors 261A and 262A. Further, as a configuration for generating image light, a right backlight 221 and a left backlight 222, and a right LCD 241 and a left LCD 242 are provided. The “display unit” does not require these configurations.

  For example, image light generated by a mechanism built in either or both of the right display driving unit 22 and the left display driving unit 24 of the image display unit 20 is transmitted to the user side of the image display unit 20, that is, the user's eyes. The light may be reflected by a reflection mechanism provided on the facing side and emitted to the user's eyes. Here, as the reflection mechanism, for example, a scanning optical system using a MEMS (Micro Electro Mechanical Systems) mirror can be adopted. In other words, a scanning optical system having a MEMS mirror that scans the light emitted from the image light generation unit may be provided, and the light scanned by the scanning optical system may be directly incident on the user's eyes. Further, an optical member on which a virtual image is formed by light scanned by the scanning optical system may be provided in the image display unit 20. This optical member forms a virtual image by scanning light scanned by the MEMS mirror. In this case, when the MEMS mirror scans light, a virtual image is formed on the virtual image forming surface, and the image is visually recognized (recognized) when the user catches the virtual image with eyes. The optical component in this case may be one that guides light through a plurality of reflections, such as the right light guide plate 261 and the left light guide plate 262 of the above embodiment, and may use a half mirror surface. .

Further, the scanning optical system is not limited to the configuration including the MEMS mirror. The mechanism that generates the image light may also be a laser light source that emits laser light. For example, the present invention can also be applied to a laser retinal projection type head mounted display. That is, the light emitting unit includes a laser light source and an optical system that guides the laser light source to the user's eyes, and the laser light is incident on the user's eyes to scan the retina and form an image on the retina. A configuration that allows the user to visually recognize an image may be employed.
Moreover, it may replace with the virtual image formation surface which receives the light scanned, and the structure which guides image light to a user's eyes using a diffraction grating may be sufficient. That is, the optical member is not limited to guiding the image light inside the optical member, and may have only a function of guiding the image light by refracting and / or reflecting it toward the user's eyes.

  In the configuration including the scanning optical system having MEMS or the like, the position where the user visually recognizes the image, that is, the display position of the image can be changed by changing the mounting angle of the scanning optical system in the image display unit 20. Therefore, in the process of changing the display position in each of the above embodiments, an operation of changing the angle of the scanning optical system may be performed instead of the operation of changing the display position of the image on the right LCD 241 and the left LCD 242.

  The optical system that guides the image light to the user's eyes may include an optical member that transmits external light that is incident on the apparatus from the outside and that is incident on the user's eyes together with the image light. Moreover, you may use the optical member which is located ahead of a user's eyes and overlaps a part or all of a user's visual field.

  Further, in each of the above embodiments, a configuration in which virtual images are formed by the half mirrors 261A and 262A on a part of the right optical image display unit 26 and the left optical image display unit 28 positioned in front of the user's eyes is illustrated. The present invention is not limited to this, and an image may be displayed on a display area having an area that occupies the entire surface or most of the right optical image display unit 26 and the left optical image display unit 28. In this case, a process of reducing the image may be included in the operation of changing the display position of the image.

  Furthermore, the optical element of the present invention is not limited to the right light guide plate 261 and the left light guide plate 262 having the half mirrors 261A and 262A, and may be any optical component that allows image light to enter the user's eyes. A diffraction grating, a prism, or a holographic display unit may be used.

  In addition, at least a part of the functional blocks shown in FIGS. 4 and 18 may be realized by hardware, or may be realized by cooperation of hardware and software. As described above, the present invention is not limited to a configuration in which independent hardware resources are arranged. Further, the program executed by the control unit 140 may be stored in the storage unit 120 or the storage device in the control device 10, or the program stored in the external device is acquired via the communication unit 117 or the interface 125. It is good also as a structure to execute. Of the components formed in the control device 10, only the operation unit 135 may be formed as a single user interface (UI). Further, the configuration formed in the control device 10 may be formed in the image display unit 20 in an overlapping manner. For example, the control unit 140 shown in FIGS. 4 and 18 may be formed in both the control device 10 and the image display unit 20, or the control unit 140 and the image display units 20 and 20B formed in the control device 10. The functions performed by the CPU formed in the above may be configured separately.

  DESCRIPTION OF SYMBOLS 10 ... Control apparatus 20, 20B ... Image display part (display part), 21 ... Right holding part, 22, 22A ... Right display drive part, 23 ... Left holding part, 24, 24A ... Left display drive part, 25 ... Interface , 26 ... right optical image display unit, 28 ... left optical image display unit, 61 ... right camera (imaging unit), 62 ... left camera (imaging unit), 64 ... distance sensor, 68 ... gaze sensor (gaze detection unit), DESCRIPTION OF SYMBOLS 100, 100A ... Head-mounted display apparatus (display apparatus), 117 ... Communication part (connection part), 120 ... Memory | storage part, 125 ... Interface, 140 ... Control part, 150 ... Operating system, 160 ... Image processing part, 161 ... image acquisition unit (outside scene image acquisition unit), 162 ... position detection unit, 163 ... image selection unit, 164 ... operation processing unit, 165 ... display position control unit, 187 ... audio processing unit, 190 ... display control unit, 2 DESCRIPTION OF SYMBOLS 1 ... Right backlight control part, 202 ... Left backlight control part, 211 ... Right LCD control part, 212 ... Left LCD control part, 221 ... Right backlight, 222 ... Left backlight, 241 ... Right LCD, 242 ... Left LCD, 251 ... right projection optical system, 252 ... left projection optical system, 261 ... right light guide plate, 262 ... left light guide plate, 261A, 262A ... half mirror, 271 ... right transparent LCD, 272 ... left transparent LCD.

Claims (22)

A display unit that displays an image so that it can be seen through the outside scene together with the outside scene;
An outside scene image acquisition unit that acquires an outside scene image including an outside scene that is visible through the display unit;
A control unit that recognizes an object viewed through the display unit based on an outside scene image acquired by the outside scene image acquisition unit, and displays information on the object on the display unit;
A display device comprising: The control unit, when the target is outside the range in which the information is displayed on the display unit, by the display unit to perform a guidance display that guides the line of sight to the target;
The display device according to claim 1. The control unit displays, on the display unit, an image of an outside scene in a range that is visible through the display unit, based on the outside scene image acquired by the outside scene image acquisition unit;
The display device according to claim 1 or 2. The display unit is a head-mounted display unit that is worn on a user's head, and includes a first display unit that displays the image, and the first display unit that is closer to the outside scene than the first display unit. A second display unit that displays an image that is visually recognized in a wider range,
The control unit performs guidance display for guiding a line of sight to the target by the second display unit,
The display device according to claim 1, wherein: The display unit is a head-mounted display unit mounted on the user's head,
A line-of-sight detector that detects the line of sight of the user;
The control unit changes the display of the display unit corresponding to the direction of the line of sight of the user detected by the line-of-sight detection unit;
The display device according to claim 1, wherein: The control unit displays a partial region of the outside scene image acquired by the outside scene image acquisition unit on the display unit, and corresponds to the direction of the user's line of sight detected by the line of sight detection unit. Changing the position of a partial area to be displayed on the display unit;
The display device according to claim 5. An imaging unit that images a range including an outside scene that is visible through the display unit;
The outside scene image acquisition unit acquires a captured image of the imaging unit as the outside scene image;
The display device according to claim 1, wherein: The outside scene image acquisition unit generates the outside scene image by acquiring and combining a plurality of captured images captured by the imaging unit;
The display device according to claim 7. A plurality of the imaging units;
The outside scene image acquisition unit generates the outside scene image by acquiring and combining a plurality of captured images respectively captured by the plurality of imaging units;
The display device according to claim 7 or 8. It has a connection to connect to external equipment,
The outside scene image acquisition unit acquires the outside scene image from the external device connected to the connection unit;
The display device according to claim 1, wherein: It has a position detector that detects the current position,
The outside scene image acquisition unit acquires the outside scene image corresponding to the current position detected by the position detection unit from the external device connected to the connection unit;
The display device according to claim 10. The control unit recognizes the target by extracting the target image from the outside scene image acquired by the outside scene image acquisition unit;
The display device according to claim 1, wherein: A display device mounted on a user's head,
A display unit having a display area for transmitting an outside scene and displaying an image so as to be visible together with the outside scene;
A control unit for recognizing an object located outside a range that can be seen through the display area and outputting information on the object;
A display device comprising: The control unit outputs the information that guides the user to a state in which the target can be visually recognized when the target is outside a range that can be viewed through the display unit.
The display device according to claim 13. The control unit guides the user to a state in which the target is visible through the display region when the target is viewed through the outside of the display region in the display unit. Outputting information,
The display device according to claim 13. The control unit displays characters and / or images constituting the information in a display area of the display unit;
The display device according to any one of claims 13 to 15. It has a sound output unit that outputs sound,
The control unit outputs sound constituting the information by the sound output unit;
The display device according to claim 13, wherein the display device is a display device. The control unit obtains position information indicating the position of the object, and obtains a relative position between the object and a range that the user sees through the display area based on the obtained position information. Outputting the information;
The display device according to claim 13, wherein: The control unit obtains an outside scene image obtained by imaging the outside scene, and obtains a relative position between the target and a range that the user sees through the display area based on the obtained outside scene image. Outputting information,
The display device according to claim 13, wherein the display device is a display device. Control a display device that includes a display unit that displays an image so that it can be seen with the outside scene through the outside scene,
An outside scene image including an outside scene that is visible through the display unit is acquired, an object that is viewed through the display unit is recognized based on the acquired outside scene image, and information about the object is displayed. Display in the department,
A control method of a display device characterized by the above. A computer that controls a display device that includes a display unit that transmits an outside scene and displays an image so as to be visible together with the outside scene.
An outside scene image acquisition unit that acquires an outside scene image including an outside scene that is visible through the display unit;
A control unit that recognizes an object viewed through the display unit based on an outside scene image acquired by the outside scene image acquisition unit, and displays information on the object on the display unit;
Program to make it work. A computer that controls a display device that includes a display unit that has a display area that displays an image so that the image can be visually recognized through the outside scene.
A control unit for recognizing an object located outside a range that can be seen through the display area and outputting information on the object;
Program to make it work.

Images