JP2016034133A - Head-mounted display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a head-mounted display device capable of improving use feeling.SOLUTION: A head-mounted display device 1 includes: a display element 8 for emitting image light to display an image; sight line direction detecting means 2 for detecting a sight line direction of an observer; imaging means 3 for imaging an area in the sight line direction; a sight line direction determining portion 11 for determining whether the sight line direction turns downward or not, on the basis of a detection result of the sight line direction detecting means 2; an operation determining portion 12 for determining whether a shape of a hand of the observer indicates a predetermined operation pattern or not, on the basis of an input signal from the imaging means 3; and a display control portion 13 for controlling display by the display element 8, on the basis of determination results of the sight line direction determining portion 11 and the operation determining portion 12. The display control portion 13 makes the display element 8 display an operation object when it is determined that the sight line direction turns downward and the shape of the hand indicates the predetermined operation pattern.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a head-mounted display device mounted on an observer's head.

Conventionally, a head-mounted display device that has an appearance like glasses and allows an observer to observe an image formed by a display element such as an LCD (Liquid Crystal Display) using an optical system has been proposed (for example, Patent Document 1).
A head-mounted display device described in Patent Literature 1 includes a display having a display element (LCD or the like), an optical system, and the like, and a head position detection device that is attached to the display and detects the tilt of the observer's head. Is provided. This head-mounted display device is configured to display a virtual keyboard (operation object) at the position of the observer's hand when the head position detection device detects that the observer's head is tilted downward. ing.

Japanese Patent Laid-Open No. 9-179062

  However, in the head-mounted display device described in Patent Literature 1, when the observer's head is facing downward, the virtual keyboard is always displayed. For this reason, when the observer does not intend to operate the keyboard, there is a problem that the display on the keyboard feels bothersome and uncomfortable for the observer.

  An object of the present invention is to provide a head-mounted display device that can improve the feeling of use.

The head-mounted display device of the present invention includes a display element that emits image light, a gaze direction detection unit that detects a gaze direction of an observer, an imaging unit that images a region in the gaze direction, and the gaze direction detection A line-of-sight direction determination unit that determines whether the line-of-sight direction is directed downward based on a detection result of the means, and a shape of the observer's hand based on an input signal from the imaging means And a display control unit that controls display by the display element based on determination results of the line-of-sight direction determination unit and the operation determination unit. A position operation corresponding to the position of the hand in the display area of the image light when it is determined that the line-of-sight direction is directed downward and the shape of the hand indicates the first pattern By object And executes the process to be defined.
Here, the operation object refers to an object displayed as an operation means of the head-mounted display device.

In the present invention, the display control unit displays the operation object on the display element when it is determined that the observer's line-of-sight direction is directed downward and the shape of the observer's hand shows a predetermined pattern. Let
Thus, even when the observer's line-of-sight direction is facing downward, the operation object is not displayed unless the observer's hand shows a predetermined pattern. For this reason, it is possible to prevent the operation object from being displayed against the intention of the observer, and to improve the feeling of use.
Further, in the present invention, when it is determined that the shape of the hand indicates the first pattern, the operation object at the position corresponding to the position of the hand in the display area is executed, which is contrary to the intention of the observer. The treatment can be prevented and the usability can be further improved.

  In the head-mounted display device of the present invention, it is preferable that the line-of-sight direction detecting means is an acceleration sensor provided in the head-mounted display device.

By the way, when detecting the observer's line of sight itself, that is, the direction in which the observer's eyes are facing, the configuration of the detection means is generally complicated.
On the other hand, in the present invention, the line-of-sight direction detection means is an acceleration sensor provided in the head-mounted display device, and the observation is performed by detecting the inclination of the observer's head from the acceleration acting on the acceleration sensor. The direction of the person's gaze is detected. For this reason, since the gaze direction can be detected using a general-purpose and simple configuration called an acceleration sensor, the head-mounted display device can be manufactured at low cost and can be downsized.

The head-mounted display device of the present invention preferably includes a light guide unit that guides the image light emitted from the display element to a predetermined position.
In the head-mounted display device of the present invention, it is preferable that the light guide unit is configured so that an external image can be observed.

The head-mounted display device of the present invention includes a projection lens that projects the image light of the display element, and the light guide unit is fixed to the projection lens and is projected from the projection lens. It is preferable to guide light to the predetermined position.
In the present invention, since the head-mounted display device is a so-called see-through type in which an external image can be observed through a light guide plate, the observer can also observe an external image as well as an image formed on the display element. be able to. As a result, the observer can observe the image formed on the display element and the external field image at the same time, so it is not necessary to remove the head-mounted display device when attempting to observe the external field image. A feeling can be improved more.

  In the head-mounted display device of the present invention, it is preferable that the display control unit displays the operation object at a lower position in the display area.

  In the present invention, since the display control unit displays the operation object at a lower position in the display area, even when the head-mounted display device is a see-through type, the observer's hand blocks the display area when operating the operation object. This can be prevented and the usability can be further improved.

  In the head-mounted display device of the present invention, the display control unit displays the display when it is determined that the line-of-sight direction is directed downward and the shape of the hand indicates the first operation pattern. It is preferable to display a cursor at a position corresponding to the position of the hand in the region.

  In the present invention, when it is determined that the line-of-sight direction is directed downward and the shape of the hand indicates the first operation pattern, the cursor is displayed at a position corresponding to the position of the hand in the display area. The pointing position in the display area can be made to be recognized well by the observer while preventing the display contrary to the intention of the observer.

  In the head-mounted display device of the present invention, when the display control unit determines that the line-of-sight direction is directed downward and the shape of the hand indicates a second pattern, the display region It is preferable to execute processing defined by an operation object at a position corresponding to the position of the hand.

  In the head-mounted display device of the present invention, the operation determination unit determines whether or not the time during which the hand shape indicates the first pattern continues for a predetermined time or more, and the display control unit , When it is determined that the line-of-sight direction is directed downward and the time during which the shape of the hand indicates the first pattern continues for a predetermined time or longer, the position of the hand in the display area is It is preferable to execute a process of moving the operation object at the corresponding position according to the position of the hand.

  In the present invention, when it is determined that the time during which the shape of the hand indicates the first pattern continues for a predetermined time or longer, the operation object at the position corresponding to the position of the hand in the display area is aligned with the position of the hand. Therefore, the operation object can be easily moved only by the observer moving the position of the hand.

  In the head-mounted display device according to the aspect of the invention, when the display control unit determines that the line-of-sight direction is directed downward and the shape of the hand indicates a third pattern, the display area It is preferable to execute a process of displaying a list of operations related to operation objects at positions corresponding to the positions of the hands.

  In the present invention, when it is determined that the shape of the hand indicates the third pattern, a list of operations related to the operation objects at positions corresponding to the position of the hand in the display area is displayed. It is possible to display an operation related to the operation object simply by making it. For this reason, the operation menu regarding the operation object can be easily displayed, and the usability can be further improved.

In the head-mounted display device according to the aspect of the invention, the display control unit may display an image based on the input image signal on the display area when it is determined that the line-of-sight direction does not face downward. preferable.
In the head-mounted display device of the present invention, the display control unit is input when it is determined that the line-of-sight direction is directed downward and the shape of the hand does not correspond to the predetermined pattern. It is preferable to display an image based on the image signal in the display area.

The perspective view which shows the external appearance of the head mounted display apparatus in one Embodiment of this invention. The block diagram which shows the structure of a head mounting | wearing type display apparatus. The figure which shows the mounting state of a head mounting | wearing type display apparatus. The figure which shows the shape of the hand showing an operation pattern. The flowchart explaining the display control of a head mounting type display apparatus. The figure which shows the use condition of a head mounting type display apparatus. The figure which shows the use condition of a head mounting type display apparatus. The figure which shows the use condition of a head mounting type display apparatus.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[1. Configuration of head mounted display device]
FIG. 1 is a perspective view showing the appearance of the head-mounted display device 1.
In FIG. 1, for convenience of explanation, an axis parallel to an optical axis Ax (FIG. 3) of a projection lens 91 described later is taken as a Z axis, a horizontal axis perpendicular to the Z axis is taken as an X axis, and a vertical perpendicular to the Z axis is taken. The axis is the Y axis. The same applies to the subsequent figures. Further, in the Z axis, the observer side is defined as the + Z axis side, and the side away from the observer is defined as the −Z axis side. FIG. 3 shows only the side corresponding to the left eye of the observer for convenience of explanation.
As shown in FIG. 1, the head-mounted display device 1 is a head-mounted display having an appearance like glasses, and allows an observer wearing the head-mounted display device 1 to recognize image light. . Further, the head-mounted display device 1 of the present embodiment is configured by a see-through type virtual image display device capable of observing an external image with see-through.
The head-mounted display device 1 includes a line-of-sight direction detection unit 2, an imaging unit 3, an image forming device 4 (FIG. 2), a light guide plate 5 as a light guide unit, and a rim 6 that holds the light guide plate 5. And a temple 7.

The line-of-sight direction detection means 2 is provided on the rim 6 and detects the line-of-sight direction of the observer.
In the present embodiment, the line-of-sight direction detection means 2 includes an acceleration sensor (gravity sensor) and a gyro sensor, and detects the tilt state (tilt state with respect to the horizontal direction) of the head-mounted display device 1 to detect the observer's direction. The line-of-sight direction is detected, and a signal corresponding to the detected inclination is output to the control device 10.

The imaging means 3 is provided on the rim 6 and images an area in the viewing direction of the observer.
In the present embodiment, the imaging means 3 is provided in the center of the front surface of the rim 6, and images the front of the head-mounted display device 1, that is, the direction in which the viewer's head is facing, An area in the line of sight direction is imaged. The imaging means 3 is constituted by, for example, a CCD camera including an area sensor having a CCD (Charge Coupled Device) as an imaging element, and outputs a signal corresponding to the captured image to the control device 10.

As shown in FIG. 1, the image forming apparatus 4, the light guide plate 5, and the temple 7 are provided as a pair corresponding to the left eye and the right eye of the observer wearing the head-mounted display device 1, respectively. Are symmetrical (left-right symmetric) with respect to.
Since the pair of image forming apparatuses 4, the pair of light guide plates 5, and the pair of temples 7 have the same configuration on the left and right, only the side corresponding to the left eye of the observer will be described below.

[2. Configuration of image forming apparatus]
FIG. 2 is a block diagram illustrating a configuration of the head-mounted display device 1 including the image forming device 4.
FIG. 3 is a diagram showing a wearing state of the head-mounted display device 1.
The image forming device 4 is a device that forms and projects image light, and includes a display element 8, a projection optical device 9, and a control device 10 (FIG. 2), as shown in FIGS.

  The display element 8 emits image light and displays an image under display control of the control device 10. In the present embodiment, the display element 8 is attached to a transmissive liquid crystal display device 81 that modulates incident light to form image light, and a light incident side of the liquid crystal display device 81. A backlight 82 (FIG. 3) such as an LED (Light Emitting Diode) that emits light is provided.

  The projection optical device 9 includes a projection lens 91 that projects image light emitted from the display element 8 as parallel light, and a lens barrel 92 (FIG. 3) that houses the projection lens 91 therein. The projection optical device 9 is fixed to the rim 6, and the projection lens 91 is fixed to the light guide plate 5 via the lens barrel 92 and the rim 6. Thereby, the projection lens 91 is configured not to be displaced with respect to the light guide plate 5 (FIGS. 1 and 3).

The control device 10 is configured by mounting a semiconductor device such as a CPU (Central Processing Unit) or an IC (Integrated Circuit) chip on a circuit board, and drives the liquid crystal display device 81 based on an image signal input from the outside. Image light based on the image signal is formed on the liquid crystal display device 81.
As illustrated in FIG. 2, the control device 10 includes a line-of-sight direction determination unit 11, an operation determination unit 12, and a display control unit 13.

The line-of-sight direction determination unit 11 determines whether the line-of-sight direction of the observer is directed downward based on the input signal from the line-of-sight direction detection unit 2.
In the present embodiment, the line-of-sight direction determination unit 11 determines that the line-of-sight direction of the observer is facing downward when the inclination angle from the line-of-sight direction detection unit 2 exceeds a predetermined threshold.

The operation determination unit 12 determines whether or not the shape of the observer's hand indicates a predetermined operation pattern based on the input signal from the imaging unit 3.
In the present embodiment, the operation determination unit 12 recognizes that the observer is performing a predetermined operation when the shape of the observer's hand is the shape shown in FIG.
FIG. 4A shows a hand shape of a pointing pattern as a first operation pattern in which the observer points to a location on the screen.
FIG. 4B shows a hand shape of a click pattern as a second operation pattern for performing a so-called left click operation of the mouse.
FIG. 4C shows a hand shape of a menu display pattern as a third operation pattern for performing a so-called mouse right click operation.

  Here, the image forming apparatus 4 includes a storage device (not shown) that stores each operation pattern, and the operation determination unit 12 performs a predetermined operation by the observer based on the operation pattern stored in the storage device. It is configured to recognize what you are doing. The operation pattern stored in the storage device can be edited by an operation pattern registration unit (not shown), and the observer can set an arbitrary operation pattern.

  The display control unit 13 generates image data from the input image signal, and outputs a drive signal to the display element 8 based on the image data, thereby causing the display element 8 to display an image based on the image signal. Further, the display control unit 13 displays an operation object T (FIG. 8) such as a keyboard or an icon on the display element 8 based on the determination results of the line-of-sight direction determination unit 11 and the operation determination unit 12, or the operation object T Execute the defined process.

[3. Configuration of light guide plate]
The light guide plate 5 is made of a light-transmissive resin material or the like, takes in the image light emitted from the image forming apparatus 4, and then guides it to a predetermined external position (the left or right eye of the observer). As shown in FIG. 3, the light guide plate 5 is formed in a substantially isosceles trapezoid shape when viewed from the direction along the Y axis. The light guide plate 5 includes a light incident surface 51, a first reflection surface 52, a first total reflection surface 53, a second total reflection surface 54, a second reflection surface 55, and a light emission surface 56. Is formed.

  The first reflecting surface 52 is formed by forming a film such as aluminum vapor deposition on a flat inclined surface facing the light incident surface 51 and inclined with respect to the XY plane, and the light guide plate 5 through the light incident surface 51. The image light captured inside is reflected toward the first total reflection surface 53.

The first total reflection surface 53 is configured by a plane obtained by extending the light incident surface 51 (parallel to the XY plane (perpendicular to the optical axis Ax)), and is located on the + Z axis side.
The second total reflection surface 54 is formed in a flat shape parallel to the first total reflection surface 53 and is located on the −Z axis side.
Then, the image light reflected by the first reflecting surface 52 is separated from the light incident surface 51 and the first reflecting surface 52 by the total reflection on the first and second total reflecting surfaces 53 and 54 (the other side). To the light guide plate 5 side).

  The first and second total reflection surfaces 53 and 54 are not limited to those in which image light is totally reflected and guided by the interface with the air without providing a mirror or a half mirror on the surface. (2) It also includes reflection formed by forming a mirror coat or a half mirror film on the whole or a part of the total reflection surfaces 53 and 54. For example, after the incident angle of the image light satisfies the total reflection condition, the whole or a part of the first and second total reflection surfaces 53 and 54 is subjected to mirror coating or the like to reflect substantially all the image light. The case of making it include is also included. Moreover, as long as image light with sufficient brightness can be obtained, the whole or part of the first and second total reflection surfaces 53 and 54 may be coated with a somewhat transmissive mirror.

The second reflection surface 55 is formed in a flat shape that faces the light emission surface 56 and is inclined with respect to the XY plane, and reflects the image light guided by the first and second total reflection surfaces 53 and 54 by reflection. The light is guided from the exit surface 56 to a predetermined external position (the left eye or the right eye of the observer). The second reflecting surface 55 is configured by a half mirror or the like, and is configured to reflect image light and to transmit an external image.
In addition, as long as it has the function to guide the image light guided by the first and second total reflection surfaces 53 and 54 to a predetermined external position, the polarization beam is not limited to the second reflection surface 55 such as a half mirror. A splitter, a hologram diffraction grating, or the like may be employed.

[4. Temple structure)
As shown in FIG. 3, the temple 7 has a hollow interior, and the image forming apparatus 4 is accommodated in the hollow portion. In the present embodiment, the projection optical device 9 and the display element 8 are arranged in order from the light guide plate 5 side toward the distal end side of the temple 7.
The temple 7 is locked to the observer's ear when the head-mounted display device 1 is attached to the observer.

[5. Display control〕
Next, display control of the above-described head-mounted display device 1 will be described with reference to the drawings.
FIG. 5 is a flowchart for explaining display control of the head-mounted display device 1.
6 to 8 are diagrams illustrating a usage state of the head-mounted display device.
In FIG. 5, first, the line-of-sight direction determination unit 11 of the control device 10 performs a determination process as to whether or not the observer's head is inclined downward, so that the line-of-sight direction of the observer is directed downward. It is determined whether or not there is (step S1). When it is determined in step S1 that the observer's line-of-sight direction is directed downward, the operation determination unit 12 executes an observer operation determination process based on the shape of the observer's hand (step S2).

  Here, when it is determined in step S1 that the observer's line-of-sight direction is not directed downward, the display control unit 13 displays an image based on the input image signal on the display area A (from FIG. 6). (Step S3). In this case, as shown in FIG. 6, the observer observes the image with the head kept horizontal, and the operation object T such as a keyboard is not displayed.

  Although it is determined in step S1 that the observer's line of sight is directed downward, in step S2, the shape of the observer's hand is a predetermined operation pattern (in this embodiment, FIG. 4 (A) to FIG. When it is recognized that the operation pattern is not 4 (C), the display control unit 13 causes the display element 8 to display an image based on the input image signal in the display area A (step S3). In this case, as shown in FIG. 7, the observer observes the image with the line-of-sight direction facing downward, but the operation object T is not displayed. Since the head-mounted display device 1 of the present embodiment is a see-through type, the observer can take notes on a sheet, for example, while observing an image.

  On the other hand, when it is recognized in step S2 that the shape of the observer's hand is a predetermined operation pattern, the display control unit 13 displays an image based on the input image signal as shown in FIG. As a result, the operation object T is displayed at a lower position in the display area A (step S4). In addition, the operation determination unit 12 determines the position of the observer's hand (in this embodiment, the pointing position of the observer) based on the input signal from the imaging unit 3, and the display control unit 13 is in the display area A. A cursor having a shape such as an arrow is displayed at a position corresponding to the position (step S5).

  Here, as the operation object T displayed in the display area A, a character key and other operation keys (keys for performing operations such as return (enter), movement, return, input, erase, copy, character conversion) Examples include keyboards having various icons, various icons, and the like. When the keyboard is displayed, an input display field for displaying characters input by the keyboard is displayed at the same time as the keyboard, and when the return key is clicked after the characters are input, various application documents and Internet search sites in the display area A are displayed. The character displayed in the input display field may be input to the character input area or the like.

Subsequently, the operation determination unit 12 determines whether or not the shape of the observer's hand is the click pattern shown in FIG. 4B (step S6).
When it is determined in step S6 that the shape of the observer's hand is a click pattern, the operation determination unit 12 determines whether or not the time during which the shape of the observer's hand indicates the click pattern has continued for a predetermined time or more. Is determined (step S7).

  When it is determined in step S7 that the time indicating the click pattern is less than the predetermined time, the display control unit 13 is defined by the operation object T (FIG. 8) at a position corresponding to the finger pointing position of the observer. The process is executed (step S8). That is, the display control unit 13 performs a process corresponding to a so-called left click operation in which a click button on the left side of the mouse is pressed in a state where the cursor is positioned on the icon in the display area A. On the other hand, when it is determined in step S7 that the time indicating the click pattern has continued for a predetermined time or longer, the display control unit 13 selects an operation object at a position corresponding to the position of the hand in the display area A. A so-called mouse drag operation process is performed to move according to the hand position (step S9).

  On the other hand, when it is determined in step S6 that the hand shape is not a click pattern, the operation determination unit 12 determines whether or not the hand shape is a menu display pattern (step S10). If it is determined in step S10 that the shape of the hand is not a menu display pattern, the process returns to step S1, and if not, the display control unit 13 lists operations related to the operation object at a position corresponding to the position of the hand in the display area. It is displayed (step S11). That is, the display control unit 13 performs a process corresponding to a so-called right click operation in which the click button on the right side of the mouse is pressed in a state where the cursor is positioned on the icon in the display area A.

The embodiment described above has the following effects.
In the present embodiment, the display control unit 13 displays the operation object T when it is determined that the viewer's line-of-sight direction is directed downward and the shape of the observer's hand indicates a predetermined operation pattern. Displayed on the display element 8.
Thus, even when the observer's line-of-sight direction is pointing downward, the operation object T is not displayed unless the observer's hand shows a predetermined operation pattern. For this reason, it can prevent that the operation object T is displayed against an observer's intention, and can improve a usability | use_condition.

By the way, when detecting the observer's line of sight itself, that is, the direction in which the observer's eyes are facing, the configuration of the detection means is generally complicated.
On the other hand, in this embodiment, the line-of-sight direction detection means 2 is an acceleration sensor provided in the head-mounted display device 1, and detects the tilt of the observer's head from the acceleration acting on the acceleration sensor. Thus, the visual line direction of the observer is detected. For this reason, since a gaze direction can be detected using a general-purpose and simple configuration called an acceleration sensor, the head-mounted display device 1 can be manufactured at low cost and can be downsized.

  In the present embodiment, the head-mounted display device 1 is a so-called see-through type in which an external image can be observed through the light guide plate 5, so that an external image as well as an image formed on the display element 8 is observed. Can be observed. As a result, the observer can observe the image formed by the display element 8 and the external field image at the same time, so that it is not necessary to remove the head-mounted display device 1 when observing the external field image. , The feeling of use can be further improved.

  In the present embodiment, the display control unit 13 displays the operation object T at a lower position in the display area A. Therefore, even when the head-mounted display device 1 is a see-through display, It is possible to prevent the hand from blocking the display area A, and the usability can be further improved.

  In the present embodiment, when it is determined that the line-of-sight direction is directed downward and the shape of the observer's hand indicates the first operation pattern, the position corresponds to the position of the hand in the display area A. Display the cursor. For this reason, it is possible to make the observer recognize the pointing position in the display area A well while preventing display contrary to the intention of the observer.

  In the present embodiment, when it is determined that the shape of the observer's hand indicates the second operation pattern, the processing of the operation object T at a position corresponding to the position of the hand in the display area A is executed. Processing contrary to the intention of the observer can be prevented, and the usability can be further improved.

It should be noted that the present invention is not limited to the above-described embodiments, and modifications, improvements, and the like within the scope that can achieve the object of the present invention are included in the present invention.
In the above embodiment, the head-mounted display device employs the display element 8 including the transmissive liquid crystal display device 81. However, the present invention is not limited to this, and other configurations, for example, a reflective liquid crystal display device. Alternatively, a display element including a digital micromirror device or the like may be employed. A self-luminous display element such as an organic EL device may be employed.

  In the above embodiment, the head-mounted display device 1 is a virtual image display type that allows an observer to recognize the image light emitted from the display element 8 via the light guide plate 5. A direct display type in which the image light emitted from 8 is directly recognized by an observer may be used. Moreover, as a light guide part, you may employ | adopt not only the light guide plate 5 but other structures, such as a mirror.

  In the embodiment, the line-of-sight direction detection unit 2 is configured to detect the observer's line-of-sight direction by detecting the tilt state of the head-mounted display device 1, but is not limited thereto. As the line-of-sight direction detecting means 2, any structure can be adopted as long as the line-of-sight direction of the observer can be detected. For example, the structure of detecting the line of sight of the observer itself may be adopted.

  In the embodiment, the display control unit 13 displays the operation object T on the display element 8 together with the image based on the image signal. However, only the operation object T may be displayed. Examples of such a case include menu display in a state where an image based on an image signal is not displayed.

In the above embodiment, the pointing pattern, the click pattern, and the menu display pattern are used as the first to third operation patterns. However, the present invention is not limited to this, and the operation pattern can be set arbitrarily.
In the above embodiment, when it is determined that the time indicating the click pattern has continued for a predetermined time or more, the operation object is moved in accordance with the position of the observer's hand, but this is not a limitation. For example, other objects and screens displayed in the display area A such as various application documents and Internet search sites may be moved.

  The present invention can be used for a head-mounted display device such as a head-mounted display.

  DESCRIPTION OF SYMBOLS 1 ... Head-mounted display apparatus, 2 ... Gaze direction detection means, 3 ... Imaging means, 5 ... Light guide plate (light guide part), 8 ... Display element, 11 ... Line-of-sight direction determination unit, 12 ... operation determination unit, 13 ... display control unit, 91 ... projection lens, T ... operation object.

The head-mounted display device of the present invention includes a display element that emits image light, a gaze direction detection unit that detects a gaze direction of an observer, an imaging unit that images a region in the gaze direction, and the gaze direction detection based on the detection result of means for determining, and determines the line of sight direction determining section for determining whether or not the line-of-sight direction is directed downward, based on the input signal from the imaging means, the shape of the hand of the observer An operation determination unit, and a display control unit that controls display by the display element based on determination results of the line-of-sight direction determination unit and the operation determination unit, and the display control unit has the line-of-sight direction downward The cursor is displayed at a position corresponding to the position of the hand in the display area when the operation determining unit determines that the pattern is set and the shape of the hand is indicated. Features and That.
Here, the operation object refers to an object displayed as an operation means of the head-mounted display device.

In the present invention, the cursor is displayed at a position corresponding to the position of the hand in the display area when it is determined that the line-of-sight direction is directed downward and the shape of the hand is set . It is possible to make the observer recognize the pointing position in the display area well while preventing display contrary to the intention of the observer.

  The head-mounted display device of the present invention includes a display element that emits image light, a gaze direction detection unit that detects a gaze direction of an observer, an imaging unit that images a region in the gaze direction, and the gaze direction detection A line-of-sight direction determining unit that determines whether or not the line-of-sight direction is directed downward based on a detection result of the means, and an operation of determining the shape of the observer's hand based on an input signal from the imaging means A display control unit configured to control display by the display element based on determination results of the determination unit and the determination results of the visual line direction determination unit and the operation determination unit, and the display control unit has the visual line direction facing downward. And when the operation determination unit determines that the shape of the hand is a set pattern, the operation related to the operation object at a position corresponding to the position of the hand in the display area is determined. Wherein the display.

In the present invention, when the hand shape is determined to indicate a set pattern, since displaying a list of operations related to the operation object at a position corresponding to the position of the hand in the display area is set to the hand shape The operation related to the operation object can be displayed only by making the pattern. For this reason, the operation menu regarding the operation object can be easily displayed, and the usability can be further improved.

  A head-mounted display device as a related technique of the present invention includes a display element that emits image light, a gaze direction detection unit that detects a gaze direction of an observer, an imaging unit that captures an area in the gaze direction, Based on the detection result of the line-of-sight direction detection means, a line-of-sight direction determination unit that determines whether or not the line-of-sight direction is directed downward, and the shape of the observer's hand based on an input signal from the imaging means An operation determination unit that determines whether or not indicates a predetermined pattern, and a display control unit that controls display by the display element based on determination results of the line-of-sight direction determination unit and the operation determination unit. In the display region of the image formed by the image light, the display control unit determines that the line-of-sight direction is directed downward and the shape of the hand indicates a set pattern. Above Performing the operations defined by the operation object at a position corresponding to the position.

In the related technology , the display control unit places an operation object on the display element when it is determined that the viewer's line-of-sight direction is directed downward and the shape of the observer's hand indicates a predetermined pattern. Display.
Thus, even when the observer's line-of-sight direction is facing downward, the operation object is not displayed unless the observer's hand shows a predetermined pattern. For this reason, it is possible to prevent the operation object from being displayed against the intention of the observer, and to improve the feeling of use.
Further, in the related technology, when it is determined that the hand shape indicates the set pattern, the operation object at the position corresponding to the position of the hand in the display area is executed. Therefore, it is possible to prevent the processing contrary to the above, and to further improve the feeling of use.

In the related technology, the operation determination unit determines whether or not the time during which the shape of the hand indicates the set pattern has continued for a predetermined time or more, and the display control unit determines that the line-of-sight direction is The position corresponding to the position of the hand in the display region when it is determined that the time when the shape of the hand is facing downward and the shape of the hand indicating the set pattern has continued for a predetermined time or longer. It is preferable to execute a process of moving the operation object in accordance with the position of the hand.

In the related technology , when it is determined that the time indicating the pattern in which the hand shape is set continues for a predetermined time or longer, the operation object at the position corresponding to the position of the hand in the display area is moved to the position of the hand. Therefore, the operation object can be easily moved only by the observer moving the hand position.

  In the head-mounted display device of the present invention, it is preferable that the line-of-sight direction detecting means is an acceleration sensor provided in the head-mounted display device.

By the way, when detecting the observer's line of sight itself, that is, the direction in which the observer's eyes are facing, the configuration of the detection means is generally complicated.
On the other hand, in the present invention, the line-of-sight direction detection means is an acceleration sensor provided in the head-mounted display device, and the observation is performed by detecting the inclination of the observer's head from the acceleration acting on the acceleration sensor. The direction of the person's gaze is detected. For this reason, since the gaze direction can be detected using a general-purpose and simple configuration called an acceleration sensor, the head-mounted display device can be manufactured at low cost and can be downsized.

  The head-mounted display device of the present invention preferably includes a light guide unit that guides the image light emitted from the display element to a predetermined position.

  In the head-mounted display device of the present invention, it is preferable that the light guide unit is configured so that an external image can be observed.

The head-mounted display device of the present invention includes a projection lens that projects the image light of the display element, and the light guide unit is fixed to the projection lens and is projected from the projection lens. It is preferable to guide light to the predetermined position.
In the present invention, since the head-mounted display device is a so-called see-through type in which an external image can be observed through a light guide plate, the observer can also observe an external image as well as an image formed on the display element. be able to. As a result, the observer can observe the image formed on the display element and the external field image at the same time, so it is not necessary to remove the head-mounted display device when attempting to observe the external field image. A feeling can be improved more.

  In the head-mounted display device of the present invention, it is preferable that the display control unit displays the operation object at a lower position in the display area.

  In the present invention, since the display control unit displays the operation object at a lower position in the display area, even when the head-mounted display device is a see-through type, the observer's hand blocks the display area when operating the operation object. This can be prevented and the usability can be further improved.

In the head-mounted display device according to the aspect of the invention, the display control unit may display an image based on the input image signal on the display area when it is determined that the line-of-sight direction does not face downward. preferable.
In the head-mounted display device of the present invention, the display control unit determines that the line-of-sight direction is directed downward and the shape of the hand does not correspond to a predetermined pattern determined by the operation determination unit. It is preferable that an image based on the input image signal is displayed in the display area.

Claims (11)

A display element that emits image light;
Gaze direction detection means for detecting the gaze direction of the observer;
Imaging means for imaging the region in the line-of-sight direction;
A line-of-sight direction determining unit that determines whether the line-of-sight direction is directed downward based on a detection result of the line-of-sight direction detection unit;
An operation determination unit that determines whether or not the shape of the observer's hand shows a predetermined pattern based on an input signal from the imaging means;
A display control unit that controls display by the display element based on determination results of the line-of-sight direction determination unit and the operation determination unit,
The display control unit, when it is determined that the line-of-sight direction is directed downward and the shape of the hand indicates the first pattern, the hand in the display area of the image formed by the image light. A head-mounted display device that executes processing defined by an operation object at a position corresponding to the position of the head. The head-mounted display device according to claim 1,
The head-mounted display device, wherein the line-of-sight direction detection means is an acceleration sensor provided in the head-mounted display device. The head-mounted display device according to claim 1 or 2,
A head-mounted display device comprising: a light guide unit that guides the image light emitted from the display element to a predetermined position. The head-mounted display device according to claim 3,
The head-mounted display device, wherein the light guide unit is configured to be able to observe an external image. The head-mounted display device according to claim 3 or 4,
A projection lens for projecting the image light of the display element;
The light guide unit is fixed to the projection lens, and guides the image light projected from the projection lens to the predetermined position. The head-mounted display device according to any one of claims 1 to 5,
The display control unit displays the operation object at a lower position in the display area. The head-mounted display device according to any one of claims 1 to 6,
The display control unit moves the cursor to a position corresponding to the position of the hand in the display area when it is determined that the line-of-sight direction is directed downward and the shape of the hand indicates a second pattern. A head-mounted display device characterized by displaying. The head-mounted display device according to any one of claims 1 to 7,
The operation determination unit determines whether or not the time during which the shape of the hand indicates the first pattern continues for a predetermined time or more,
When the display control unit determines that the line-of-sight direction is directed downward and the time during which the shape of the hand indicates the first pattern continues for a predetermined time or longer, the display control unit A head-mounted display device that executes a process of moving an operation object at a position corresponding to the position of the hand according to the position of the hand. The head-mounted display device according to any one of claims 1 to 8,
The display control unit operates the position corresponding to the position of the hand in the display area when it is determined that the line-of-sight direction is directed downward and the shape of the hand indicates a third pattern. A head-mounted display device that executes a process of displaying a list of operations related to an object. The head-mounted display device according to any one of claims 1 to 9,
The head mounted display device, wherein the display control unit displays an image based on the input image signal in the display area when it is determined that the line-of-sight direction is not directed downward. The head-mounted display device according to any one of claims 1 to 10,
The display control unit displays an image based on the input image signal in the display area when it is determined that the line-of-sight direction is directed downward and the shape of the hand does not correspond to the predetermined pattern. A head-mounted display device characterized in that

Images